New: connect Claude & other AIs to GenAIList over MCP — research the catalog and contribute to the shared knowledge base. Learn how →
BIG-Bench Hard benchmark
AI model leaderboard for the BIG-Bench Hard benchmark. Compare how large language models score on BIG-Bench Hard, see the full ranking, and understand what this AI benchmark measures. DeepSeek-V3 currently leads with 89.5%. 23 challenging tasks from BIG-Bench where prior LLMs fell below the human baseline.
Leaderboard
| # | Model | Organization | Score | Variant | Source |
|---|---|---|---|---|---|
| #1 | DeepSeek-V3 | DeepSeek | 89.5% | cited-tulu3 | official ↗ |
| #2 | Telechat2-115B | China Telecom | 89.04% | from-hf-readme | official ↗ |
| #3 | TeleChat2-35B | China Telecom | 88.6% | from-hf-readme | official ↗ |
| #4 | Tulu 3 405B | Allen Institute for AI | 88.6% | instruct-0-shot-cot | official ↗ |
| #5 | Hermes 3 405B | Nous Research | 87.7% | cited-tulu3 | official ↗ |
| #6 | Llama 3.1-405B | Meta AI | 87.1% | instruct-cited-tulu3 | official ↗ |
| #7 | GPT-4o (Nov 2024) | OpenAI | 83.3% | cited-tulu3 | official ↗ |
| #8 | Hermes 3 70B | Nous Research | 82.1% | cited-tulu3 | official ↗ |
| #9 | Granite-4.0-H-Small | IBM | 81.62% | 3-shot-cot | official ↗ |
| #10 | GPT-4o mini | OpenAI | 80.4% | cited-phi35moe | official ↗ |
| #11 | DeepSeek-V2 (MoE-236B) | DeepSeek | 79.7% | cited-telechat2 | official ↗ |
| #12 | Phi-3.5-MoE | Microsoft | 79.1% | instruct-0-shot-cot | official ↗ |
| #13 | TeleChat2-7B | China Telecom | 77.3% | from-hf-readme | official ↗ |
| #14 | Yi-1.5-34B | 01.AI | 76.4% | cited-qwen2-57b | official ↗ |
| #15 | Qwen1.5-110B | Qwen | 74.8% | cited-telechat2 | official ↗ |
| #16 | Llama 3.1-70B | Meta AI | 73.8% | instruct-cited-tulu3 | official ↗ |
| #17 | Llama 3.1-8B | Meta AI | 72.66% | instruct-cited-granite-3-2 | official ↗ |
| #18 | Qwen2.5-7B | Qwen | 72.4% | cited-phi4-mini | official ↗ |
| #19 | Phi-4 Mini | Microsoft | 70.4% | instruct-0-shot-cot | official ↗ |
| #20 | Qwen2.5-7B | Qwen | 70.4% | instruct-cited-granite-3-2 | official ↗ |
| #21 | Kimi Linear | Moonshot | 69.4% | sft | official ↗ |
| #22 | Granite-4.0-H-Micro | IBM | 69.36% | 3-shot-cot | official ↗ |
| #23 | Granite 3.1 8B | IBM | 68.55% | instruct-cited-granite-3-2 | official ↗ |
| #24 | Qwen2.5 Instruct (72B) | Qwen | 67.2% | instruct-cited-tulu3 | official ↗ |
| #25 | Qwen2-57B-A14B | Qwen | 67% | base-from-hf-readme | official ↗ |
| #26 | Qwen1.5-32B | Qwen | 66.8% | cited-qwen2-57b | official ↗ |
| #27 | Gemini 1.5 Flash (Sep 2024) | Google DeepMind | 66.7% | cited-phi35moe | official ↗ |
| #28 | Granite-4.0-H-Tiny | IBM | 66.34% | 3-shot-cot | official ↗ |
| #29 | Tulu 3 8B | Allen Institute for AI | 66% | instruct-3-shot-cot | official ↗ |
| #30 | TeleChat2-3B | China Telecom | 65.99% | from-hf-readme | official ↗ |
| #31 | DeepSeek-R1-Distill-Llama-8B | DeepSeek | 65.71% | cited-granite-3-2 | official ↗ |
| #32 | DeepSeek-R1-Distill-Qwen-7B | Qwen | 65.04% | cited-granite-3-2 | official ↗ |
| #33 | DeepSeek-R1-Distill-Qwen-7B | DeepSeek | 65.04% | cited-granite-3-2 | official ↗ |
| #34 | Granite 3.2 8B | IBM | 64.77% | instruct-from-hf-readme | official ↗ |
| #35 | Gemma 2 9B | Google DeepMind | 63.5% | instruct-cited-phi35moe | official ↗ |
| #36 | Llama 3.1-8B | Meta AI | 63.4% | instruct-cited-phi35moe | official ↗ |
| #37 | phi-3.5-mini | Microsoft | 63.1% | cited-phi4-mini | official ↗ |
| #38 | Llama 3.1-8B | Meta AI | 62.8% | instruct-cited-tulu3 | official ↗ |
| #39 | Qwen2-7B | Qwen | 62.6% | base-from-hf-readme | official ↗ |
| #40 | Mistral NeMo | Mistral AI | 60.2% | instruct-cited-phi35moe | official ↗ |
| #41 | Llama 3-8B | Meta AI | 57.7% | base-cited-qwen2-7b | official ↗ |
| #42 | Qwen2.5-3B | Qwen | 56.2% | cited-phi4-mini | official ↗ |
| #43 | Mistral 7B | Mistral AI | 56.1% | base-cited-qwen2-7b | official ↗ |
| #44 | Llama 3.2 3B | Meta AI | 55.4% | cited-phi4-mini | official ↗ |
| #45 | Gemma 7B | Google DeepMind | 55.1% | base-cited-qwen2-7b | official ↗ |
| #46 | Granite 3.1 2B | IBM | 54.46% | instruct-cited-granite-3-2 | official ↗ |
| #47 | Qwen2.5-7B | Qwen | 54.1% | instruct-3-shot-cited-falcon3 | official ↗ |
| #48 | Falcon3-7B | Technology Innovation Institute | 52.4% | instruct-3-shot | official ↗ |
| #49 | Granite 3.2 2B | IBM | 52.27% | instruct-from-hf-readme | official ↗ |
| #50 | Ministral 8B | Mistral AI | 51.2% | cited-phi4-mini | official ↗ |
| #51 | Marin 8B | Marin | 50.6% | base-from-hf-readme | official ↗ |
| #52 | Mixtral 8x7B | Mistral AI | 50.3% | cited-qwen2-57b | official ↗ |
| #53 | Llama 3.1-8B | Meta AI | 48.6% | instruct-3-shot-cited-falcon3 | official ↗ |
| #54 | Llama 3.1-8B | Meta AI | 46.4% | base-cited-marin | official ↗ |
| #55 | Jamba | AI21 Labs | 45.4% | cited-qwen2-57b | official ↗ |
| #56 | Phi-2 | Microsoft | 43.4% | cited-qwen2 | official ↗ |
| #57 | Qwen1.5-7B | Qwen | 40.2% | base-cited-qwen2-7b | official ↗ |
| #58 | Qwen2-1.5B | Qwen | 37.2% | base-from-hf-readme | official ↗ |
| #59 | Gemma 2B | Google DeepMind | 35.2% | cited-qwen2 | official ↗ |
| #60 | Qwen2-0.5B | Qwen | 28.4% | base-from-hf-readme | official ↗ |
| #61 | Llama 3.1-8B | Meta AI | 25.29% | base-cited-falcon-mamba | official ↗ |
| #62 | Llama 3-8B | Meta AI | 24.5% | cited-falcon-mamba | official ↗ |
| #63 | Mistral 7B | Mistral AI | 22.02% | v0-1-cited-falcon-mamba | official ↗ |
| #64 | Gemma 7B | Google DeepMind | 21.12% | cited-falcon-mamba | official ↗ |
| #65 | Falcon Mamba | Technology Innovation Institute | 19.88% | 3-shot-from-hf-readme | official ↗ |
Frequently asked questions about BIG-Bench Hard
What is the BIG-Bench Hard benchmark?
23 challenging tasks from BIG-Bench where prior LLMs fell below the human baseline.
How is the BIG-Bench Hard benchmark scored?
BIG-Bench Hard is scored using the accuracy (%) metric, where a higher score is better. The maximum achievable score is 100.000. GenAIList aggregates reported scores from model providers and papers into a single ranked leaderboard.
Which AI model scores highest on BIG-Bench Hard?
As of the latest reported scores on GenAIList, DeepSeek-V3 achieves the highest result on BIG-Bench Hard with a score of 89.5%.
Is a higher BIG-Bench Hard score better?
Yes. On BIG-Bench Hard a higher score indicates better performance, so models near the top of the leaderboard are the strongest.