Grok Vision Beta: Complete Benchmark Performance Guide

xai vision-beta 2023-11-04 Proprietary

In-depth analysis of Grok Vision Beta performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendorxai
Versionvision-beta
Release Date2023-11-04
Context Window8192 tokens
Input Modalitiestext, image
Output Modalitiestext
LicenseProprietary
Documentationhttps://docs.x.ai/

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU63.3%2023-11-045-shotview
HUMANEVAL44pass@12023-11-04—view
GSM8K35.2%2023-11-040-shot CoTview
MATH24%2023-11-040-shot CoTview
BBH56.8%2023-11-043-shot CoTview
GPQA32.4%2023-11-040-shotview
IFEVAL46.8%2023-11-04prompt_strictview
ARC88.2%2023-11-04challengeview
MUSR44.5%2023-11-040-shotview
WINOGRANDE67.5%2023-11-040-shotview

Pricing

TierPriceCurrency
Input$1 / MtokUSD
Output$5 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://x.ai/api · as of 2023-11-04

Compliance

  • Data Residency: US
  • SOC2: ✗
  • HIPAA: ✗
  • GDPR: ✗
  • ISO 27001: ✗

Grok Vision Beta

모델 개요

xAI Grok Vision Beta 早期多模态预览版, 8K 上下文, 支持图像理解, 集成 X 平台实时信息。

핵심 사양

공급업체버전출시일컨텍스트 창입력 모달리티출력 모달리티라이선스
Xaivision-beta2023-11-048Ktext, imagetextProprietary

벤치마크 성능

벤치마크점수단위비고
MMLU (Massive Multitask Language Understanding)63.3%5-shot
HumanEval44.0pass@1—
GSM8K (Grade School Math 8K)35.2%0-shot CoT
MATH24.0%0-shot CoT
BBH (BIG-Bench Hard)56.8%3-shot CoT
GPQA32.4%0-shot
IFEval46.8%prompt_strict
ARC88.2%challenge
MUSR44.5%0-shot
WinoGrande67.5%0-shot

가격

입력출력캐시 읽기캐시 쓰기
————

백만 토큰당

강점

  • 支持文本、图像、音频多模态输入。

약점

  • HumanEval 44.0，代码能力较弱。
  • 闭源专有模型，不支持自托管。
  • 上下文窗口 8K 偏小。

사용 사례

  • 视觉与图像理解

참고문헌