Overview

大规模Winograd Schema挑战集，需要常识推理来消除代词指代歧义

Metrics

Metric	Unit	Direction
accuracy	%	↑ Higher is better

Sources

Model Score Ranking

#	Model	Vendor	Score
1	Gemini 2.0 Flash	google	89.5
2	GPT-4o (2024-05-13)	openai	89.3
3	Gemini 2.0 Flash Thinking	google	88.1
4	Claude 3 Opus (2024-02-29)	anthropic	87.8
5	GPT-4o (2024-08-06)	openai	87.7
6	Sonar Huge	other	87.7
7	Grok-2 Vision	xai	87.5
8	Sonar Reasoning	other	87.2
9	Claude 3.5 Sonnet (2024-10-22)	anthropic	86.9
10	GPT-4 Vision	openai	86.9
11	Llama 3.3 70B	meta	86.8
12	o1	openai	86.7
13	GPT-4 0125 Preview	openai	86.6
14	Hermes 3 Llama 3.1 405B	other	86.5
15	Gemini 1.5 Pro 002	google	85.8
16	GLM-4 Plus	other	85.8
17	o1 Preview	openai	85.2
18	Claude 3 Haiku	anthropic	84.8
19	Command R	cohere	84.8
20	DeepSeek V2	deepseek	84.7
21	Grok-2 Mini	xai	84.7
22	Llama 3.1 Nemotron 70B	meta	84.7
23	NVIDIA Llama 3.1 Nemotron 70B	other	84.7
24	Qwen2.5 14B	alibaba	84.7
25	Gemini 1.5 Flash-8B 002	google	84.6
26	Gemini 1.0 Pro	google	84.5
27	Jamba 1.5	other	84.5
28	Jamba 1.5 Large	other	84.3
29	StableLM 2 12B	other	84.1
30	Yi Large	other	84.1
31	Qwen1.5 110B	alibaba	84
32	Hermes 3 Llama 3.1 70B	other	83.9
33	Claude 3.5 Haiku	anthropic	83.8
34	Claude 3 Sonnet	anthropic	83.8
35	WizardLM Team WizardLM 2 8x22B	other	83.5
36	Command R+ (08-2024)	cohere	83.4
37	Nous Hermes 2 Yi 34B	other	83.4
38	Mistral Large	mistral	83.3
39	Qwen1.5 72B	alibaba	83.1
40	Qwen2 72B	alibaba	82.5
41	Mistral Medium	mistral	82.4
42	Yi Large Turbo	other	82.3
43	Yi Vision	other	82.3
44	GPT-4 32K	openai	82.2
45	Nous Hermes 2 Mixtral 8x7B	other	82.1
46	GPT-4 1106 Preview	openai	82
47	Mixtral 8x7B	mistral	82
48	Claude 3 Opus	anthropic	81.9
49	Gemini 1.5 Flash 002	google	81.9
50	Qwen2.5 32B	alibaba	81.9
51	DBRX Instruct	other	81.8
52	Microsoft WizardLM 2 8x22B	other	81.6
53	Sonar Small	other	81.4
54	GLM-4 Flash	other	81.1
55	Sonar Large	other	81.1
56	Jamba 1.5 Mini	other	81
57	Llama 3.1 70B	meta	81
58	Command Nightly	cohere	80.8
59	GPT-4 Turbo	openai	80.8
60	GPT-4 Vision Preview	openai	80.6
61	Gemma 2 27B	google	80.5
62	Gemini 1.0 Ultra	google	80.4
63	Claude 3 Sonnet (2024-02-29)	anthropic	80.3
64	GPT-4	openai	80.3
65	DeepSeek V2 Chat	deepseek	80.2
66	Orca 2 13B	other	80.2
67	Gemini 1.0 Flash	google	80.1
68	Code Llama 34B	meta	80
69	Falcon 180B	other	80
70	DBRX Base	other	79.8
71	DeepSeek Coder 33B	deepseek	79.8
72	DeepSeek Coder 7B	deepseek	79.8
73	Gemini 1.5 Flash-8B	google	79.7
74	Zephyr ORPO 141B Alpha	other	79.7
75	GPT-4o mini	openai	79.6
76	Mistral 7B v0.1	mistral	79.6
77	Qwen1.5 32B	alibaba	79.6
78	Claude 3 Haiku (2024-03-07)	anthropic	79.5
79	GLM-4 9B Chat	other	79.4
80	Command R (08-2024)	cohere	79.3
81	Llama 3.2 90B Vision	meta	79.3
82	DeepSeek Math 7B	deepseek	79.2
83	Llama 3 70B	meta	79.2
84	Yi 1.5 34B	other	79.2
85	Gemini 1.5 Flash	google	78.9
86	OLMo 7B Instruct	other	78.9
87	DeepSeek Coder V2	deepseek	78.8
88	Mistral Small 3	mistral	78.8
89	OLMo 7B SFT	other	78.8
90	OLMo 7B	other	78.7
91	Qwen1.5 14B	alibaba	78.6
92	Jamba Instruct	other	78.3
93	o1 mini	openai	78.3
94	Phi-3.5 MoE	other	78.3
95	Phi-4	other	78.3
96	Mistral Small	mistral	78.2
97	DeepSeek LLM 67B	deepseek	78.1
98	GLM-4 Air	other	78.1
99	Hermes 3 Llama 3.1 8B	other	78.1
100	Qwen2 57B	alibaba	78.1
101	Code Bison	google	77.9
102	MPT 30B	other	77.9
103	Llama 3.1 8B	meta	77.6
104	Qwen2 7B	alibaba	77.3
105	Claude Instant 1	anthropic	77.2
106	Llama 2 7B	meta	77.2
107	Nous Capybara 7B	other	77.1
108	StarCoder2 3B	other	76.9
109	Microsoft WizardLM 2 7B	other	76.8
110	GPT-3.5 Turbo	openai	76.7
111	OLMo 1.7 7B	other	76.6
112	Yi 1.5 9B	other	76.6
113	Llama 2 13B	meta	76.4
114	Code Llama 7B	meta	76.3
115	Flan-UL2	other	76.2
116	Codestral	mistral	75.8
117	Llama 3 8B	meta	75.6
118	Code Llama 70B	meta	75.5
119	Flan-T5 XL	other	75.5
120	Ministral 8B	mistral	75.3
121	OLMo 2 1124 7B	other	75.3
122	StarChat2 15B v0.1	other	75
123	Zephyr 7B Alpha	other	75
124	Phi-3 Small	other	74.9
125	Command Light	cohere	74.7
126	Phi-3 Vision	other	74.7
127	OpenChat 3.6 8B	other	74.5
128	Phi-1	other	74.5
129	Yi 1.5 6B	other	74.5
130	Jurassic-2 Ultra	other	74.4
131	Nous Hermes 2 Solar 10.7B	other	74.4
132	Flan-T5 XXL	other	74.2
133	Codestral Mamba	mistral	74.1
134	Command R7B	cohere	74.1
135	Gemma 7B	google	74.1
136	Phi-1.5	other	74.1
137	StableLM 3 4B	other	74.1
138	Text Bison	google	74.1
139	GPT-3.5 Turbo 16K	openai	74
140	Zephyr 7B Beta	other	74
141	Llama 2 70B	meta	73.4
142	Mistral Nemo	mistral	73.2
143	GLM-4V 9B	other	73.1
144	StarCoder2 7B	other	73.1
145	Phi-2	other	72.8
146	GPT-3.5	openai	72.7
147	Llama 3.2 11B Vision	meta	72.7
148	Mistral 7B v0.3	mistral	72.7
149	Gemma 2 9B	google	72.5
150	Yi 6B	other	72.2
151	Phi-3 Medium	other	72.1
152	Gemma 2B	google	72
153	Mistral 7B v0.2	mistral	71.7
154	StableLM Zephyr 3B	other	71.6
155	Yi 34B	other	71.5
156	Qwen2.5 7B	alibaba	71.4
157	Mathstral 7B	mistral	71
158	StableCode 3B	other	70.9
159	Llemma 7B	other	70.8
160	Microsoft WizardMath 7B v1	other	70.5
161	Code Llama 13B	meta	70.4
162	Microsoft WizardCoder Python 34B	other	70.4
163	StableLM 2 1.6B	other	70.3
164	StarCoder2 15B	other	70.2
165	Argilla Notus 7B v1	other	69.8
166	Baichuan2 13B Chat	other	69.5
167	OpenChat 3.5 1210	other	69.3
168	Grok Beta	xai	69.2
169	Capybara 1.5B	other	68.9
170	PaLM 2	google	68.6
171	Phi-3.5 Mini	other	68.6
172	Baichuan2 7B Chat	other	68.3
173	Llama Guard 3 8B	meta	68
174	Llama 3.2 3B	meta	67.8
175	ChatGLM3 6B	other	67.7
176	Grok Vision Beta	xai	67.5
177	Claude 2.1	anthropic	67.4
178	Nous Capybara 34B	other	67.3
179	Jurassic-2 Mid	other	67.1
180	Phi-3 Mini	other	67.1
181	Open-Platypus	other	66.7
182	Mistral Tiny	mistral	66.6
183	Llama Guard 2 8B	meta	66.5
184	Claude 2	anthropic	66.4
185	TigerBot 70B Chat	other	66.3
186	Chat Bison	google	66.2
187	Falcon 40B	other	65.6
188	MPT 7B	other	65.5
189	Ministral 3B	mistral	64.8
190	Qwen2.5 1.5B	alibaba	64.4
191	Qwen2.5 0.5B	alibaba	64.3
192	Llama 3.2 1B	meta	64
193	Qwen2 1.5B	alibaba	62.9
194	Qwen2.5 3B	alibaba	61.4
195	Embed English v3	cohere	0

WinoGrande

説明

大规模Winograd Schema挑战集，需要常识推理来消除代词指代歧义

コア仕様

カテゴリ	ライセンス	最終更新
reasoning	CC-BY-4.0	2024-02-01

ベンチマーク

単位
%

ソース

公式URL

https://winogrande.allenai.org