Grok-2: Complete Benchmark Performance Guide

xai 2 2024-08-13 Proprietary

In-depth analysis of Grok-2 performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendorxai
Version2
Release Date2024-08-13
Context Window131072 tokens
Input Modalitiestext, image
Output Modalitiestext
LicenseProprietary
Documentationhttps://docs.x.ai/

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU87.5%2024-08-135-shotview
HUMANEVAL88.4pass@12024-08-13—view
GSM8K93.2%2024-08-130-shot CoTview
MATH76.8%2024-08-130-shot CoTview
BBH84%2024-08-133-shot CoTview

Pricing

TierPriceCurrency
Input$2 / MtokUSD
Output$10 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://x.ai/api · as of 2024-08-13

Compliance

  • Data Residency: US
  • SOC2: ✗
  • HIPAA: ✗
  • GDPR: ✗
  • ISO 27001: ✗

Grok-2

모델 개요

xAI Grok-2 是 xAI 旗舰模型，131K 上下文窗口，支持文本与图像输入，在 MMLU、HumanEval、MATH 等基准上接近头部闭源模型，集成于 X 平台提供实时信息。

핵심 사양

공급업체버전출시일컨텍스트 창입력 모달리티출력 모달리티라이선스
Xai22024-08-13131Ktext, imagetextProprietary

벤치마크 성능

벤치마크점수단위비고
MMLU (Massive Multitask Language Understanding)87.5%5-shot
HumanEval88.4pass@1—
GSM8K (Grade School Math 8K)93.2%0-shot CoT
MATH76.8%0-shot CoT
BBH (BIG-Bench Hard)84.0%3-shot CoT

가격

입력출력캐시 읽기캐시 쓰기
————

백만 토큰당

강점

  • MMLU score 87.5, strong knowledge reasoning.
  • HumanEval 88.4, excellent code generation.
  • GSM8K 93.2, robust math reasoning.
  • 支持文本、图像、音频多模态输入。

약점

  • 闭源专有模型，不支持自托管。

사용 사례

  • 代码生成与调试
  • 视觉与图像理解
  • Agent 工作流与工具调用

참고문헌