Overview

由生物学、物理学、化学领域的博士专家撰写的难题集，确保问题难以通过Google搜索直接找到答案

Metrics

Metric	Unit	Direction
accuracy	%	↑ Higher is better

Sources

Model Score Ranking

#	Model	Vendor	Score
1	GPT-4o (2024-08-06)	openai	62
2	Gemini 2.0 Flash Thinking	google	61
3	GPT-4o (2024-05-13)	openai	58.8
4	o1 Preview	openai	57.7
5	o1	openai	57.5
6	Gemini 1.5 Pro 002	google	56.5
7	Gemini 2.0 Flash	google	55
8	Qwen2 72B	alibaba	53.6
9	Llama 3.3 70B	meta	52.8
10	Claude 3.5 Sonnet (2024-10-22)	anthropic	50.9
11	Jamba 1.5 Large	other	50.8
12	Yi Vision	other	50.3
13	Sonar Reasoning	other	50.1
14	GPT-4 Turbo	openai	49.2
15	Mistral Medium	mistral	48.7
16	Gemini 1.0 Pro	google	47.7
17	Gemini 1.0 Ultra	google	46.9
18	GPT-4 0125 Preview	openai	46.8
19	Sonar Huge	other	46.7
20	Claude 3 Opus	anthropic	46
21	Qwen1.5 110B	alibaba	45.9
22	Command Nightly	cohere	45.5
23	Command R (08-2024)	cohere	44.6
24	Qwen2.5 14B	alibaba	44.5
25	Claude 3 Haiku	anthropic	44.4
26	Command R	cohere	44
27	StableLM 2 12B	other	44
28	Grok-2 Vision	xai	43.8
29	Mistral Small	mistral	43.7
30	Falcon 180B	other	43.2
31	Mixtral 8x7B	mistral	43
32	GPT-4o mini	openai	42.6
33	Llama 3.1 Nemotron 70B	meta	42.6
34	Orca 2 13B	other	42.5
35	Llama 3.2 90B Vision	meta	42.2
36	GPT-4 Vision	openai	41.8
37	Claude 3 Sonnet (2024-02-29)	anthropic	41.7
38	Sonar Large	other	41.6
39	WizardLM Team WizardLM 2 8x22B	other	41.4
40	GPT-4 32K	openai	40.9
41	o1 mini	openai	40.8
42	DeepSeek Math 7B	deepseek	40.5
43	Hermes 3 Llama 3.1 405B	other	40.5
44	Phi-3.5 MoE	other	40.2
45	Llama 3.1 70B	meta	40
46	Gemini 1.5 Flash-8B 002	google	39.8
47	Qwen1.5 72B	alibaba	39.8
48	Claude 3 Haiku (2024-03-07)	anthropic	39.6
49	Qwen2.5 32B	alibaba	39.4
50	Qwen2 57B	alibaba	39.4
51	DBRX Base	other	39.3
52	Microsoft WizardLM 2 8x22B	other	39.3
53	GPT-4	openai	39.2
54	Nous Hermes 2 Yi 34B	other	39
55	Command R+ (08-2024)	cohere	38.7
56	Claude 3 Opus (2024-02-29)	anthropic	38.6
57	Gemini 1.5 Flash	google	38.5
58	DeepSeek V2 Chat	deepseek	38.3
59	Llama 3 70B	meta	38.2
60	Yi Large Turbo	other	38.2
61	Mistral Large	mistral	38.1
62	Yi 1.5 9B	other	38
63	Gemini 1.0 Flash	google	37.7
64	NVIDIA Llama 3.1 Nemotron 70B	other	37.6
65	Mathstral 7B	mistral	37.5
66	Jamba Instruct	other	37.4
67	Llemma 7B	other	37.4
68	OLMo 7B	other	37.4
69	DeepSeek LLM 67B	deepseek	37.3
70	GLM-4 9B Chat	other	37.2
71	Gemma 7B	google	37.1
72	Zephyr ORPO 141B Alpha	other	36.8
73	Microsoft WizardCoder Python 34B	other	36.6
74	Code Llama 13B	meta	36.4
75	Codestral Mamba	mistral	36.3
76	GPT-4 Vision Preview	openai	36.3
77	GPT-4 1106 Preview	openai	36.2
78	StarCoder2 3B	other	36.2
79	Mistral Nemo	mistral	36.1
80	Claude 3 Sonnet	anthropic	35.9
81	Nous Hermes 2 Solar 10.7B	other	35.9
82	Llama 3.2 11B Vision	meta	35.5
83	Gemma 2 27B	google	35.4
84	GLM-4 Plus	other	35.4
85	StarCoder2 15B	other	35.3
86	Yi Large	other	35.2
87	Ministral 8B	mistral	35.1
88	Nous Hermes 2 Mixtral 8x7B	other	35.1
89	Gemini 1.5 Flash-8B	google	35
90	Open-Platypus	other	34.9
91	Llama 3.1 8B	meta	34.8
92	Baichuan2 13B Chat	other	34.7
93	DBRX Instruct	other	34.7
94	Yi 1.5 34B	other	34.7
95	Sonar Small	other	34.5
96	Zephyr 7B Alpha	other	34.3
97	Jamba 1.5 Mini	other	34.2
98	MPT 7B	other	34.2
99	Mistral 7B v0.2	mistral	33.9
100	Qwen1.5 14B	alibaba	33.9
101	StarCoder2 7B	other	33.6
102	Baichuan2 7B Chat	other	33.5
103	Argilla Notus 7B v1	other	32.7
104	Codestral	mistral	32.4
105	Grok Vision Beta	xai	32.4
106	Gemini 1.5 Flash 002	google	32
107	OLMo 7B SFT	other	31.8
108	Grok-2 Mini	xai	31.7
109	Qwen2 7B	alibaba	31.6
110	DeepSeek V2	deepseek	31.5
111	Hermes 3 Llama 3.1 8B	other	31.5
112	Claude 2	anthropic	31.4
113	GLM-4 Air	other	31.3
114	GLM-4 Flash	other	31.3
115	Mistral 7B v0.3	mistral	31.2
116	TigerBot 70B Chat	other	31.2
117	Claude 3.5 Haiku	anthropic	31.1
118	Nous Capybara 7B	other	31
119	Command R7B	cohere	30.9
120	Phi-3 Small	other	30.9
121	Qwen1.5 32B	alibaba	30.6
122	OLMo 7B Instruct	other	30.5
123	Gemma 2 9B	google	30.3
124	OpenChat 3.6 8B	other	30.3
125	Jamba 1.5	other	30.2
126	Microsoft WizardMath 7B v1	other	30.2
127	Command Light	cohere	30.1
128	Hermes 3 Llama 3.1 70B	other	30.1
129	Mistral Small 3	mistral	30.1
130	Phi-4	other	30.1
131	Nous Capybara 34B	other	29.7
132	Code Llama 7B	meta	29.5
133	Yi 1.5 6B	other	29.2
134	Code Bison	google	29.1
135	Jurassic-2 Ultra	other	28.7
136	Mistral Tiny	mistral	28.7
137	Flan-T5 XXL	other	28.6
138	Zephyr 7B Beta	other	28.6
139	Qwen2.5 7B	alibaba	28.5
140	GLM-4V 9B	other	28.3
141	Llama 3 8B	meta	27.9
142	Llama Guard 3 8B	meta	27.9
143	Text Bison	google	27.7
144	Flan-UL2	other	27.2
145	Llama 2 70B	meta	27.2
146	Code Llama 34B	meta	27
147	GPT-3.5	openai	26.9
148	OLMo 1.7 7B	other	26.9
149	DeepSeek Coder 33B	deepseek	26.8
150	StableLM 3 4B	other	26.8
151	Claude 2.1	anthropic	26.4
152	OLMo 2 1124 7B	other	26.3
153	Yi 6B	other	26.3
154	Microsoft WizardLM 2 7B	other	26.1
155	StarChat2 15B v0.1	other	25.9
156	Jurassic-2 Mid	other	25.7
157	Phi-3 Medium	other	25.6
158	DeepSeek Coder V2	deepseek	25.4
159	Code Llama 70B	meta	25.3
160	Mistral 7B v0.1	mistral	25.3
161	Chat Bison	google	25
162	StableCode 3B	other	24.5
163	Phi-3.5 Mini	other	24.4
164	MPT 30B	other	24.2
165	DeepSeek Coder 7B	deepseek	23.9
166	Qwen2.5 0.5B	alibaba	23.2
167	Llama 2 7B	meta	23.1
168	Flan-T5 XL	other	22.6
169	Llama 3.2 1B	meta	22.6
170	GPT-3.5 Turbo	openai	22.4
171	Llama 2 13B	meta	22.3
172	Phi-1	other	22.1
173	Qwen2 1.5B	alibaba	22.1
174	Claude Instant 1	anthropic	21.8
175	Phi-2	other	21.8
176	GPT-3.5 Turbo 16K	openai	21.6
177	Falcon 40B	other	21.1
178	StableLM Zephyr 3B	other	21
179	OpenChat 3.5 1210	other	20.9
180	Phi-3 Mini	other	20.7
181	Grok Beta	xai	20.6
182	Yi 34B	other	20.6
183	PaLM 2	google	20.3
184	Qwen2.5 1.5B	alibaba	20.2
185	ChatGLM3 6B	other	20.1
186	Ministral 3B	mistral	19
187	Llama 3.2 3B	meta	18.8
188	Capybara 1.5B	other	18.1
189	Gemma 2B	google	18
190	Llama Guard 2 8B	meta	18
191	Qwen2.5 3B	alibaba	18
192	Phi-3 Vision	other	17.4
193	Phi-1.5	other	16.9
194	StableLM 2 1.6B	other	15
195	Embed English v3	cohere	0

GPQA

説明

由生物学、物理学、化学领域的博士专家撰写的难题集，确保问题难以通过Google搜索直接找到答案

コア仕様

カテゴリ	ライセンス	最終更新
knowledge	CC-BY-4.0	2024-01-01

ベンチマーク

単位
%

ソース

公式URL

https://github.com/idavidrein/gpqa