Overview

小学级别科学考试问题集，分为Easy和Challenge两个子集，挑战子集需要推理能力

Metrics

Metric	Unit	Direction
challenge_acc	%	↑ Higher is better
easy_acc	%	↑ Higher is better

Sources

Model Score Ranking

#	Model	Vendor	Score
1	Claude 3.5 Sonnet (2024-10-22)	anthropic	96.8
2	GPT-4o (2024-05-13)	openai	96.8
3	o1	openai	96.8
4	Sonar Reasoning	other	96.2
5	o1 Preview	openai	96
6	Command Nightly	cohere	95.9
7	Gemini 1.5 Pro 002	google	95.9
8	GPT-4o (2024-08-06)	openai	95.9
9	Gemini 1.0 Ultra	google	95.7
10	Gemini 2.0 Flash	google	95.7
11	Qwen1.5 110B	alibaba	95.7
12	Yi Large Turbo	other	95.6
13	GPT-4 0125 Preview	openai	95.5
14	Yi Vision	other	95.1
15	Gemini 2.0 Flash Thinking	google	95
16	GPT-4 32K	openai	95
17	StableLM 2 12B	other	94.9
18	Claude 3 Sonnet	anthropic	94.8
19	Claude 3 Sonnet (2024-02-29)	anthropic	94.8
20	Sonar Huge	other	94.8
21	Yi Large	other	94.8
22	DeepSeek V2 Chat	deepseek	94.6
23	GPT-4 Vision Preview	openai	94.6
24	Llama 3.2 90B Vision	meta	94.6
25	Mistral Medium	mistral	94.6
26	Qwen2 57B	alibaba	94.6
27	Claude 3 Opus	anthropic	94.5
28	Gemini 1.5 Flash-8B	google	94.5
29	GLM-4 Plus	other	94.5
30	Hermes 3 Llama 3.1 405B	other	94.5
31	Nous Hermes 2 Mixtral 8x7B	other	94.5
32	Command R (08-2024)	cohere	94.4
33	Qwen2 72B	alibaba	94.4
34	Claude 3 Haiku	anthropic	94.3
35	GPT-4 1106 Preview	openai	94.3
36	GLM-4 Flash	other	94.2
37	GPT-4 Turbo	openai	94.2
38	Jamba 1.5	other	94.2
39	Qwen2.5 14B	alibaba	94.2
40	Gemini 1.0 Pro	google	94.1
41	GPT-4	openai	94.1
42	Sonar Large	other	94.1
43	DeepSeek LLM 67B	deepseek	94
44	Command R	cohere	93.9
45	DBRX Instruct	other	93.9
46	GPT-4o mini	openai	93.9
47	Llama 3.3 70B	meta	93.9
48	Qwen2.5 32B	alibaba	93.9
49	Jamba 1.5 Large	other	93.8
50	Llama 3.1 Nemotron 70B	meta	93.7
51	o1 mini	openai	93.7
52	Yi 1.5 34B	other	93.7
53	Claude 3 Opus (2024-02-29)	anthropic	93.4
54	Command R+ (08-2024)	cohere	93.4
55	Gemini 1.5 Flash	google	93.4
56	Llama 3 70B	meta	93.4
57	Nous Hermes 2 Yi 34B	other	93.4
58	DBRX Base	other	93.3
59	GPT-4 Vision	openai	93.3
60	Grok-2 Vision	xai	93.3
61	Mistral Large	mistral	93.3
62	Gemini 1.5 Flash-8B 002	google	93.2
63	NVIDIA Llama 3.1 Nemotron 70B	other	93.2
64	Gemma 2 27B	google	93.1
65	Qwen1.5 14B	alibaba	92.7
66	Gemini 1.0 Flash	google	92.6
67	Qwen1.5 72B	alibaba	92.6
68	GLM-4 Air	other	92.3
69	Llama 3.1 70B	meta	92.3
70	WizardLM Team WizardLM 2 8x22B	other	92.3
71	DeepSeek Coder 33B	deepseek	92.2
72	Grok-2 Mini	xai	92.2
73	DeepSeek V2	deepseek	92.1
74	Microsoft WizardLM 2 8x22B	other	92.1
75	Phi-3.5 MoE	other	91.9
76	Claude 3.5 Haiku	anthropic	91.8
77	Gemini 1.5 Flash 002	google	91.8
78	Jamba 1.5 Mini	other	91.8
79	Mistral Small 3	mistral	91.8
80	OLMo 1.7 7B	other	91.8
81	Sonar Small	other	91.7
82	StarChat2 15B v0.1	other	91.7
83	Claude 3 Haiku (2024-03-07)	anthropic	91.6
84	Falcon 180B	other	91.6
85	Mixtral 8x7B	mistral	91.6
86	Mathstral 7B	mistral	91.5
87	Hermes 3 Llama 3.1 70B	other	91.4
88	Microsoft WizardMath 7B v1	other	91.4
89	Orca 2 13B	other	91.4
90	Qwen1.5 32B	alibaba	91.4
91	Zephyr ORPO 141B Alpha	other	91.4
92	Jamba Instruct	other	91.2
93	Llama 3 8B	meta	91.2
94	Microsoft WizardCoder Python 34B	other	91.2
95	Mistral Small	mistral	91.2
96	Command R7B	cohere	90.7
97	DeepSeek Math 7B	deepseek	90.5
98	Mistral Nemo	mistral	90.5
99	StarCoder2 7B	other	90.5
100	Gemma 2 9B	google	90.3
101	GLM-4V 9B	other	90.2
102	Code Llama 7B	meta	90.1
103	Phi-3 Medium	other	90.1
104	Argilla Notus 7B v1	other	89.8
105	Code Llama 70B	meta	89.8
106	Phi-3 Small	other	89.6
107	Llama 2 13B	meta	89.5
108	OLMo 7B SFT	other	89.3
109	OpenChat 3.6 8B	other	89.2
110	PaLM 2	google	89.2
111	ChatGLM3 6B	other	88.9
112	Mistral 7B v0.3	mistral	88.9
113	Flan-UL2	other	88.8
114	Code Llama 34B	meta	88.7
115	Hermes 3 Llama 3.1 8B	other	88.7
116	Mistral 7B v0.1	mistral	88.7
117	GLM-4 9B Chat	other	88.6
118	DeepSeek Coder V2	deepseek	88.5
119	Gemma 7B	google	88.4
120	Phi-1	other	88.4
121	Qwen2.5 7B	alibaba	88.4
122	Yi 6B	other	88.4
123	Grok Vision Beta	xai	88.2
124	OpenChat 3.5 1210	other	88.1
125	Flan-T5 XL	other	88
126	OLMo 7B Instruct	other	88
127	TigerBot 70B Chat	other	87.9
128	Code Llama 13B	meta	87.8
129	Yi 1.5 9B	other	87.8
130	StarCoder2 15B	other	87.6
131	MPT 7B	other	87.5
132	StarCoder2 3B	other	87.5
133	Codestral	mistral	87.4
134	Codestral Mamba	mistral	87.4
135	Llama 3.1 8B	meta	87.4
136	Llemma 7B	other	87.4
137	Yi 34B	other	87.2
138	Nous Capybara 7B	other	87
139	GPT-3.5 Turbo	openai	86.8
140	Qwen2 7B	alibaba	86.8
141	StableLM 3 4B	other	86.8
142	Text Bison	google	86.7
143	Llama 2 7B	meta	86.5
144	OLMo 7B	other	86.4
145	StableCode 3B	other	86.4
146	DeepSeek Coder 7B	deepseek	86.3
147	OLMo 2 1124 7B	other	86.2
148	Yi 1.5 6B	other	86.2
149	Code Bison	google	86.1
150	Baichuan2 7B Chat	other	86
151	Jurassic-2 Mid	other	86
152	Llama 3.2 11B Vision	meta	86
153	Command Light	cohere	85.9
154	Nous Hermes 2 Solar 10.7B	other	85.9
155	Microsoft WizardLM 2 7B	other	85.6
156	Ministral 8B	mistral	85.5
157	Mistral 7B v0.2	mistral	85.5
158	Qwen2 1.5B	alibaba	85.4
159	Qwen2.5 0.5B	alibaba	85.4
160	Phi-4	other	85.1
161	Claude 2.1	anthropic	84.8
162	Zephyr 7B Alpha	other	84.7
163	Phi-3 Mini	other	84.4
164	Baichuan2 13B Chat	other	83.2
165	GPT-3.5 Turbo 16K	openai	83.1
166	Falcon 40B	other	82.9
167	Chat Bison	google	82.6
168	Mistral Tiny	mistral	82.4
169	Jurassic-2 Ultra	other	82.1
170	StableLM Zephyr 3B	other	82.1
171	GPT-3.5	openai	81.8
172	Zephyr 7B Beta	other	81.8
173	Flan-T5 XXL	other	81.7
174	Grok Beta	xai	81.7
175	Ministral 3B	mistral	81.5
176	Phi-3.5 Mini	other	81.5
177	Claude Instant 1	anthropic	80.9
178	MPT 30B	other	80.7
179	Nous Capybara 34B	other	80.7
180	Claude 2	anthropic	80.4
181	Llama 2 70B	meta	80.3
182	Capybara 1.5B	other	80.1
183	Open-Platypus	other	80.1
184	Gemma 2B	google	77.7
185	StableLM 2 1.6B	other	77.7
186	Phi-3 Vision	other	77.4
187	Qwen2.5 1.5B	alibaba	76.7
188	Qwen2.5 3B	alibaba	76.5
189	Llama 3.2 3B	meta	76.3
190	Llama Guard 2 8B	meta	76.3
191	Llama Guard 3 8B	meta	76
192	Llama 3.2 1B	meta	75.9
193	Phi-1.5	other	75.9
194	Phi-2	other	75.1
195	Embed English v3	cohere	0

ARC

설명

小学级别科学考试问题集，分为Easy和Challenge两个子集，挑战子集需要推理能力

핵심 사양

카테고리	라이선스	최종 업데이트
reasoning	CC-BY-SA-4.0	2018-02-01

벤치마크

단위
%
%

소스

공식 URL

https://allenai.org/data/arc