// BENCHMARK

HumanEval benchmark

AI model leaderboard for the HumanEval benchmark. Compare how large language models score on HumanEval, see the full ranking, and understand what this AI benchmark measures. GPT-4o (Nov 2024) currently leads with 97%. Python function-completion benchmark introduced by OpenAI to measure functional correctness of generated code.

Leaderboard

# Model Organization Score Variant Source
#1 GPT-4o (Nov 2024) OpenAI 97% pass@10-cited-tulu3 official ↗
#2 Llama 3.1-405B Meta AI 95.9% instruct-pass@10-cited-tulu3 official ↗
#3 Tulu 3 405B Allen Institute for AI 95.9% instruct-pass@10 official ↗
#4 DeepSeek-V3 DeepSeek 94.6% pass@10-cited-tulu3 official ↗
#5 DeepSeek V4-Pro DeepSeek 94.5% pass@1 official ↗
#6 Qwen2.5 Instruct (72B) Qwen 94% instruct-pass@10-cited-tulu3 official ↗
#7 Kimi K2.6 Moonshot 93.8% pass@1 official ↗
#8 Claude 3.5 Sonnet Anthropic 93.7% 1022-cited-minimax-text official ↗
#9 Llama 3.1-70B Meta AI 93.6% instruct-pass@10-cited-tulu3 official ↗
#10 Qwen2.5-7B Qwen 93.35% instruct-cited-granite-3-2 official ↗
#11 GLM-5.1 Zhipu AI 93.2% pass@1 official ↗
#12 Hermes 3 405B Nous Research 92.3% pass@10-cited-tulu3 official ↗
#13 DeepSeek-V3 DeepSeek 92.1% cited-minimax-text official ↗
#14 Claude 3.5 Sonnet Anthropic 92% cited-nvlm official ↗
#15 DeepSeek-V3 DeepSeek 91.5% pass@1 official ↗
#16 DeepSeek V4-Flash DeepSeek 91.3% pass@1 official ↗
#17 Ministral 8B Mistral AI 91% instruct-pass@10-cited-tulu3 official ↗
#18 GPT-4 (Jun 2023) OpenAI 90.2% cited-nvlm official ↗
#19 GPT-4o (Nov 2024) OpenAI 90.2% cited-minimax-text official ↗
#20 Granite 3.1 8B IBM 89.63% instruct-cited-granite-3-2 official ↗
#21 Hermes 3 70B Nous Research 89.6% pass@10-cited-tulu3 official ↗
#22 Gemini 2.0 Flash Google DeepMind 89.6% exp-cited-minimax-text official ↗
#23 Granite 3.2 8B IBM 89.35% instruct-from-hf-readme official ↗
#24 Llama 3.1-405B Meta AI 89% instruct-cited-minimax-text official ↗
#25 Llama 3.1-405B Meta AI 89% instruct-cited-nvlm official ↗
#26 DeepSeek-V2.5 DeepSeek 89% python-from-hf-readme official ↗
#27 dots.llm1 Rednote 88.4% pass@1 official ↗
#28 NVLM-D 72B NVIDIA 88.4% megatron-from-hf-readme official ↗
#29 Granite-4.0-H-Small IBM 88% pass@1 official ↗
#30 Kimi K2 Moonshot 87.1% pass@1 official ↗
#31 MiniMax-Text-01 MiniMax 86.9% from-hf-readme official ↗
#32 Qwen2.5 Instruct (72B) Qwen 86.6% instruct-cited-minimax-text official ↗
#33 GPT-4o mini OpenAI 86.6% cited-phi35moe official ↗
#34 Gemini 1.5 Pro Google DeepMind 86.6% 002-cited-minimax-text official ↗
#35 Llama 3.1-8B Meta AI 86.3% instruct-pass@10-cited-tulu3 official ↗
#36 Qwen-72B Qwen 86% instruct-cited-nvlm official ↗
#37 Qwen2-72B Qwen 86% instruct-cited-telechat2 official ↗
#38 IBM Granite 4.1 IBM 85.37%
#39 Llama 3.1-8B Meta AI 85.32% instruct-cited-granite-3-2 official ↗
#40 Trinity-Large-Preview Arcee AI 84.8% pass@1 official ↗
#41 Gemini 1.5 Pro Google DeepMind 84.1% aug-2024-cited-nvlm official ↗
#42 Tulu 3 8B Allen Institute for AI 83.9% instruct-pass@10 official ↗
#43 Granite-4.0-H-Tiny IBM 83% pass@1 official ↗
#44 Gemma 4 31B Google DeepMind 82.4% pass@1 official ↗
#45 Llama 3-70B Meta AI 81.7% instruct-cited-nvlm official ↗
#46 DeepSeek-V2 (MoE-236B) DeepSeek 81.1% cited-telechat2 official ↗
#47 Granite-4.0-H-Micro IBM 81% pass@1 official ↗
#48 Llama 3.1-70B Meta AI 80.5% cited-telechat2 official ↗
#49 Llama 3.1-70B Meta AI 80.5% instruct-cited-nvlm official ↗
#50 Granite 3.2 2B IBM 80.13% instruct-from-hf-readme official ↗
#51 DeepSeek-R1-Distill-Qwen-7B DeepSeek 79.89% cited-granite-3-2 official ↗
#52 DeepSeek-R1-Distill-Qwen-7B Qwen 79.89% cited-granite-3-2 official ↗
#53 Gemma 4 26B Google DeepMind 79.8% pass@1 official ↗
#54 Granite 3.1 2B IBM 79.45% instruct-cited-granite-3-2 official ↗
#55 Telechat2-115B China Telecom 75% from-hf-readme official ↗
#56 Gemini 1.5 Flash (Sep 2024) Google DeepMind 74.4% cited-phi35moe official ↗
#57 Nemotron-4 340B NVIDIA 73.2% 0-shot-instruct-from-hf-readme official ↗
#58 TeleChat2-35B China Telecom 73% from-hf-readme official ↗
#59 Gemma 2 9B Google DeepMind 71.7% instruct-pass@10-cited-tulu3 official ↗
#60 Phi-3.5-MoE Microsoft 70.7% instruct-0-shot official ↗
#61 DeepSeek-R1-Distill-Llama-8B DeepSeek 67.54% cited-granite-3-2 official ↗
#62 Llama 3.1-8B Meta AI 66.5% instruct-cited-phi35moe official ↗
#63 Mistral NeMo Mistral AI 63.4% instruct-cited-phi35moe official ↗
#64 Nemotron-H 47B NVIDIA 61% base-0-shot official ↗
#65 Gemma 2 9B Google DeepMind 61% instruct-cited-phi35moe official ↗
#66 Nemotron-H 56B NVIDIA 60.37% base-0-shot official ↗
#67 Nemotron-H 8B NVIDIA 58.54% base-0-shot official ↗
#68 TeleChat2-7B China Telecom 56% from-hf-readme official ↗
#69 Qwen2-57B-A14B Qwen 53% base-from-hf-readme official ↗
#70 Qwen1.5-110B Qwen 52.4% cited-telechat2 official ↗
#71 Qwen2-7B Qwen 51.2% base-from-hf-readme official ↗
#72 Phi-2 Microsoft 47.6% cited-qwen2 official ↗
#73 Yi-1.5-34B 01.AI 46.3% cited-qwen2-57b official ↗
#74 Nous-Hermes-2-Yi-34B Nous Research 43.3% cited-nvlm official ↗
#75 Qwen1.5-32B Qwen 43.3% cited-qwen2-57b official ↗
#76 TeleChat2-3B China Telecom 38% from-hf-readme official ↗
#77 Mixtral 8x7B Mistral AI 37.2% cited-qwen2-57b official ↗
#78 Gemma 7B Google DeepMind 37.2% base-cited-qwen2-7b official ↗
#79 Qwen1.5-7B Qwen 36% base-cited-qwen2-7b official ↗
#80 Llama 3-8B Meta AI 33.5% base-cited-qwen2-7b official ↗
#81 Qwen2-1.5B Qwen 31.1% base-from-hf-readme official ↗
#82 Mistral 7B Mistral AI 29.3% base-cited-qwen2-7b official ↗
#83 Jamba AI21 Labs 29.3% cited-qwen2-57b official ↗
#84 Gemma 2B Google DeepMind 22% cited-qwen2 official ↗
#85 Qwen2-0.5B Qwen 22% base-from-hf-readme official ↗

Frequently asked questions about HumanEval

What is the HumanEval benchmark?

Python function-completion benchmark introduced by OpenAI to measure functional correctness of generated code.

How is the HumanEval benchmark scored?

HumanEval is scored using the pass@1 (%) metric, where a higher score is better. The maximum achievable score is 100.000. GenAIList aggregates reported scores from model providers and papers into a single ranked leaderboard.

Which AI model scores highest on HumanEval?

As of the latest reported scores on GenAIList, GPT-4o (Nov 2024) achieves the highest result on HumanEval with a score of 97%.

Is a higher HumanEval score better?

Yes. On HumanEval a higher score indicates better performance, so models near the top of the leaderboard are the strongest.