Mistral Nemo: Complete Benchmark Performance Guide

mistral nemo 2024-07-18 Apache 2.0

In-depth analysis of Mistral Nemo performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendormistral
Versionnemo
Release Date2024-07-18
Context Window128000 tokens
Input Modalitiestext
Output Modalitiestext
LicenseApache 2.0
Documentationhttps://docs.mistral.ai/

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU69.2%2024-07-185-shotview
HUMANEVAL69.4pass@12024-07-18—view
GSM8K70%2024-07-180-shot CoTview
MATH43.6%2024-07-180-shot CoTview
BBH71.2%2024-07-183-shot CoTview
GPQA36.1%2024-07-180-shotview
IFEVAL56%2024-07-18prompt_strictview
ARC90.5%2024-07-18challengeview
MUSR38.5%2024-07-180-shotview
WINOGRANDE73.2%2024-07-180-shotview

Pricing

TierPriceCurrency
Input$0.18 / MtokUSD
Output$0.18 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://mistral.ai/technology/ · as of 2024-07-18

Compliance

  • Data Residency: EU
  • SOC2: ✓
  • HIPAA: ✗
  • GDPR: ✓
  • ISO 27001: ✓

Mistral Nemo

모델 개요

Mistral Nemo 12B 开源模型, 128K 上下文, 与 NVIDIA 合作开发, 适合本地部署与多语言任务。

핵심 사양

공급업체버전출시일컨텍스트 창입력 모달리티출력 모달리티라이선스
Mistralnemo2024-07-18128KtexttextApache 2.0

벤치마크 성능

벤치마크점수단위비고
MMLU (Massive Multitask Language Understanding)69.2%5-shot
HumanEval69.4pass@1—
GSM8K (Grade School Math 8K)70.0%0-shot CoT
MATH43.6%0-shot CoT
BBH (BIG-Bench Hard)71.2%3-shot CoT
GPQA36.1%0-shot
IFEval56.0%prompt_strict
ARC90.5%challenge
MUSR38.5%0-shot
WinoGrande73.2%0-shot

가격

입력출력캐시 읽기캐시 쓰기
————

백만 토큰당

강점

  • 上下文窗口 128K，支持长文本。

약점

  • 闭源专有模型，不支持自托管。

사용 사례

  • 长文档摘要

참고문헌