Claude 2: Complete Benchmark Performance Guide

anthropic 2 2023-07-11 Proprietary

In-depth analysis of Claude 2 performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendoranthropic
Version2
Release Date2023-07-11
Context Window100000 tokens
Input Modalitiestext
Output Modalitiestext
LicenseProprietary
Documentationhttps://docs.anthropic.com/claude/docs

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU66.3%2023-07-115-shotview
HUMANEVAL31.7pass@12023-07-11—view
GSM8K32.5%2023-07-110-shot CoTview
MATH21.9%2023-07-110-shot CoTview
BBH60.2%2023-07-113-shot CoTview
GPQA31.4%2023-07-110-shotview
IFEVAL55%2023-07-11prompt_strictview
ARC80.4%2023-07-11challengeview
MUSR35.2%2023-07-110-shotview
WINOGRANDE66.4%2023-07-110-shotview

Pricing

TierPriceCurrency
Input$11 / MtokUSD
Output$32 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://www.anthropic.com/pricing · as of 2023-07-11

Compliance

  • Data Residency: US
  • SOC2: ✓
  • HIPAA: ✗
  • GDPR: ✓
  • ISO 27001: ✗

Claude 2

모델 개요

Anthropic Claude 2, 100K 上下文, 在编码/数学/推理上超越 Claude 1.3, 首次开放 API。

핵심 사양

공급업체버전출시일컨텍스트 창입력 모달리티출력 모달리티라이선스
Anthropic22023-07-11100KtexttextProprietary

벤치마크 성능

벤치마크점수단위비고
MMLU (Massive Multitask Language Understanding)66.3%5-shot
HumanEval31.7pass@1—
GSM8K (Grade School Math 8K)32.5%0-shot CoT
MATH21.9%0-shot CoT
BBH (BIG-Bench Hard)60.2%3-shot CoT
GPQA31.4%0-shot
IFEval55.0%prompt_strict
ARC80.4%challenge
MUSR35.2%0-shot
WinoGrande66.4%0-shot

가격

입력출력캐시 읽기캐시 쓰기
————

백만 토큰당

강점

  • 上下文窗口 100K，支持长文本。

약점

  • HumanEval 31.7，代码能力较弱。
  • 闭源专有模型，不支持自托管。

사용 사례

  • 长文档摘要

참고문헌