Karpathy发出灵魂拷问，评估AI究竟该看哪些指标？答案或许就藏在经典游戏里！最近，加州大学圣迭戈分校Hao AI Lab用超级马里奥等评测AI智能体，Claude 3.7结果令人瞠目结舌。
LLM评估基准的「黄金标准」，正在失效？
一大早，AI大神Karpathy发出质疑，「目前存在一种评估危机，我真的不知道现在该看哪些指标了」。
诸如MMLU、SWE-Bench Verified、Chat...

<div>Karpathy发出灵魂拷问，评估AI究竟该看哪些指标？答案或许就藏在经典游戏里！最近，加州大学圣迭戈分校Hao AI Lab用超级马里奥等评测AI智能体，Claude 3.7结果令人瞠目结舌。LLM评估基准的「黄金标准」，正在失效？一大早，AI大神Karpathy发出质疑，「目前存在一种评估危机，我真的不知道现在该看哪些指标了」。 <img src="https://img.36krcdn.com/hsossms/20250303/v2_38c803e6b3914c03bde7df5176bcabfd@5888275_oswg282793oswg878oswg1307_img_000?x-oss-process=image/format,jpg/interlace,1">诸如MMLU、SWE-Bench Verified、Chatbot Arena等这些基准，各有自己的优劣之处。 如果这些都不够，那么游戏算不算？ 毕竟，曾经红极一时的AlphaGo是围棋界的头号AI；就连OpenAI也早年涉足游戏领域，拿着自研AI在DOTA国际赛中取得亮眼的成绩。 最近，Claude 3.7的出世，让「宝可梦」一时间成为LLM评判的新标杆。 UCSD Hao AI Lab再次出手，开源了一种全新的「游戏智能体」，能够实时让计算机使用智能体（CUA）运行解谜、益智等类型的游戏。 结果显示，Claude 3.7 Sonnet玩超级马里奥足足撑满90s，直接碾压了OpenAI、Gemini和自家前辈；而GPT-4o一上来就直接挂掉了…… 谷歌选手Gemini 1.5 Pro首战即败，而且非常有规律地两步一跳。到了Gemini 2.0虽多走了几步，最终还是栽坑。 <img src="https://img.36krcdn.com/hsossms/20250303/v2_d0b6026fdc92485eafee7863c0d576e9@5888275_oswg230079oswg1080oswg402_img_000?x-oss-process=image/format,jpg/interlace,1">GamingAgent项目代码已开源，下载安装即可观战AI游戏大PK。 <img src="https://img.36krcdn.com/hsossms/20250303/v2_058060cd6a5044179a06209055927623@5888275_oswg127148oswg1080oswg547_img_000?x-oss-process=image/format,jpg/interlace,1"><h2>GPT-4.5反应迟钝，GPT-4o永远被第一个小怪杀死</h2>GPT-4o总是被第一个小怪杀死，像极了操作很烂会被队友喷的游戏菜鸡。短短20s，游戏就结束了。<img src="https://img.36krcdn.com/hsossms/20250303/v2_becee1ccd1e74b6fac255545ed5e7fb7@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">相比之下，GPT-4.5的表现就好多了，起码没卡在第一个小怪。<img src="https://img.36krcdn.com/hsossms/20250303/v2_93084855615c4738982f8891bd7abd1c@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">但它的反应还是很迟缓，几乎是两步一停。跳过一个矮水管之前，也要犹豫片刻，感觉像是刚学会了游戏操作，还在蹒跚学步。<img src="https://img.36krcdn.com/hsossms/20250303/v2_6f714495a44747469c8dfe4da38e1c98@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">一个稍高点的水管，尝试了7次，足足花了10s才跳了过去。<img src="https://img.36krcdn.com/hsossms/20250303/v2_eb7553f27d3149069f121799aa7a6a92@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">好不容易跳了过去，就撞到小怪死掉了。第一回合就这样告终了。<img src="https://img.36krcdn.com/hsossms/20250303/v2_a48f060c82d74c7387034c077512ba86@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">更好笑的是，第二回合的时候，GPT-4.5又栽倒在了第一个小怪那里。毕竟和GPT-4o同属于OpenAI家族，操作都比较菜（bushi）。<img src="https://img.36krcdn.com/hsossms/20250303/v2_63ce158da5b146dab7ee84849a82212f@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">第三回合表现也比较一般，还不如第一回合。第一个矮水管就卡了半天，搁水管底下卡了快10s才想起来跳。<img src="https://img.36krcdn.com/hsossms/20250303/v2_0c863a54bd39473795f39c3fb05b7df6@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">最后虽然丝滑地跳过了第二个水管，但还是被小怪杀死了，还没有第一回合走得远。第一回合起码跳过了第三个水管，虽说刚跳过就被杀了。<img src="https://img.36krcdn.com/hsossms/20250303/v2_115c366cdb5f4ebe8ae3bdecfb58741b@5888275_img_000?x-oss-process=image/format,jpg/interlace,1"><h2>Gemini 1.5两步一跳，2.0栽进坑里</h2>到了谷歌这边，Gemini 1.5 Pro首战也不如意，没能逃过第一个小怪的魔爪。<img src="https://img.36krcdn.com/hsossms/20250303/v2_8c6d40d9bee242c29dd3be062bd7bc9e@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">第二回合Gemini 1.5算是躲过了第一个小怪，甚至还碰到了问号箱，吃到了蘑菇。<img src="https://img.36krcdn.com/hsossms/20250303/v2_f98ddef1c2c043fa8c504ac66786208b@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">有趣的是，和GPT-4.5两步一停不同，Gemini 1.5是「两步一跳」。走了这么一小段路，一共就跳了9回。地板上也跳一跳，水管上也跳一跳。<img src="https://img.36krcdn.com/hsossms/20250303/v2_d277e5dbd24f42278d548fc07c293123@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">最后也是跳过了第三根水管，甚至差点跳过了第四个，算是走得比GPT-4.5要远。 <img src="https://img.36krcdn.com/hsossms/20250303/v2_7555b7c33d804c30969f3695dda50137@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">至于更新的Gemini 2.0 Flash，表现上不出意外地要好得多。 首先，跳得更大胆；其次，跳得也更流畅。 跳到了「前人」未曾涉足的更高的平台上，而且10s就轻松跳过了前面三个水管。 <img src="https://img.36krcdn.com/hsossms/20250303/v2_546d0202e1e940eba9531acca29efc4f@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">虽然第二回合的时候也惨遭第一个小怪的毒手。 <img src="https://img.36krcdn.com/hsossms/20250303/v2_c6b711f040974454b37e3f4ed898952a@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">但最后走得比OpenAI家族和Gemini 1.5都远——跳过了第四根水管，栽倒在了一个没能跳过去的坑中。 <img src="https://img.36krcdn.com/hsossms/20250303/v2_2b13c2983e4e4d36a32bdf06424af490@5888275_img_000?x-oss-process=image/format,jpg/interlace,1"><h2>Claude 3.7 Sonnet发现隐藏奖励</h2>相比之下，Anthropic的Claude，就要惊艳多了。<img src="https://img.36krcdn.com/hsossms/20250303/v2_74f3a95b3b73408b945f0f175d1cf300@5888275_oswg79579oswg864oswg521_img_000?x-oss-process=image/format,jpg/interlace,1">相比于Gemini两步一跳的操作，Claude 3.7的操作更加丝滑，走得也远很多。 尤其是在跳跃的时机上，显得更有章法，碰到水管、碰到坑才会跳。<img src="https://img.36krcdn.com/hsossms/20250303/v2_d92037cfed76469583fc71546e934c5e@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">此外，还会有意识地通过跳跃来躲避小怪。 <img src="https://img.36krcdn.com/hsossms/20250303/v2_2b8b148cb9f9428098faf040b4f76462@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">跳过了Gemini 2.0 Flash两回合都没跳过去的坑，Claude操作下的马里奥终于是吃到了金币；终于是碰到了除了哥布林（形似蘑菇）之外的小怪——库巴（形似乌龟）；甚至还碰出了隐藏奖励——超级星星。 <img src="https://img.36krcdn.com/hsossms/20250303/v2_96c570d6d8d84a1da615c076cc105caa@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">最后是掉到了阶梯平台之间的坑里，结束了游戏。 <img src="https://img.36krcdn.com/hsossms/20250303/v2_3f6f4106edcf48639245779320b00d2c@5888275_img_000?x-oss-process=image/format,jpg/interlace,1"><h2>AI大战2048益智游戏，GPT-4o拿不出手</h2>接下来，再看一个益智类的游戏2048。可能很多人对这款游戏并不熟悉，规则是通过滑动进行拼图，玩家将带有相同数字的方块合并，达到可能最高的数值。 GPT-4o在挑战过程中，因为思考过久，陷入困境。 而Claude 3.7虽多走了几步，比GPT-4o强不少，但最终还是以失败告终。 <img src="https://img.36krcdn.com/hsossms/20250303/v2_fb9126b198494b24af498f0be8ec5816@5888275_img_000?x-oss-process=image/format,jpg/interlace,1"><h2>俄罗斯方块，智商在线</h2>那么Claude 3.7玩俄罗斯方块的表现，又如何呢？<img src="https://img.36krcdn.com/hsossms/20250303/v2_0587d7a43e3441938af678f325791203@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">Anthropic开发者关系负责人Alex Albert称赞道，「非常酷！我们需要把每一款电子游戏都变成一种评估工具」。 <img src="https://img.36krcdn.com/hsossms/20250303/v2_9ef4b120899748c688c5f1d9f0eb8518@5888275_oswg57305oswg1080oswg100_img_000?x-oss-process=image/format,jpg/interlace,1">已经有网友在评论区许愿，让Grok 3加入战场。 看来，LLM评估即将开辟一条全新的路。 参考资料： https://x.com/haoailab/status/1895557913621795076https://x.com/haoailab/status/1895605453461340472https://lmgame.org/#/aboutus本文来自微信公众号<a rel="nofollow" href="https://mp.weixin.qq.com/s/IAcXyI32rfNqqu8lWkShmg">“新智元”</a>，作者：新智元，36氪经授权发布。</div>

Claude 3.7硬控马里奥90秒，GPT-4o开局暴毙，Karpathy直呼基准失效，游戏成LLM新战场

Here is the English translation:

Karpathy raised a soul-searching question, evaluating what indicators should AI really look at? The answer may be hidden in classic games! Recently, UC San Diego's Hao AI Lab used Super Mario and other benchmarks to evaluate AI agents, and the results of Claude 3.7 were astonishing.

Is the "gold standard" of LLM evaluation benchmarks becoming ineffective?

Early in the morning, AI expert Karpathy expressed doubts, "There is currently an evaluation crisis, and I really don't know what indicators I should look at now."

Such as <MMLU>, <SWE-Bench Verified>, <Chat>...

Here is the English translation of the text, with the specified terms preserved:

<div>Karpathy raised a soul-searching question, evaluating what metrics AI should really be looking at? The answer may be hidden in classic games! Recently, UC San Diego's Hao AI Lab used Super Mario and other games to evaluate AI agents, and the results of Claude 3.7 were jaw-dropping.Is the "golden standard" of LLM evaluation benchmarks becoming ineffective?Early in the morning, AI guru Karpathy expressed doubts, "There is currently an evaluation crisis, and I really don't know what metrics to look at now."<img src="https://img.36krcdn.com/hsossms/20250303/v2_38c803e6b3914c03bde7df5176bcabfd@5888275_oswg282793oswg878oswg1307_img_000?x-oss-process=image/format,jpg/interlace,1">Benchmarks like MMLU, SWE-Bench Verified, Chatbot Arena, etc. all have their own pros and cons.If these are not enough, can games be considered?After all, the once-popular AlphaGo was the number one AI in the world of Go; even OpenAI ventured into the gaming field in the early days, and their self-developed AI achieved outstanding results in the DOTA International Tournament.Recently, the emergence of Claude 3.7 has made "Pokémon" a new benchmark for LLM evaluation.UCSD Hao AI Lab has stepped in again, open-sourcing a brand-new "game agent" that can run puzzle and puzzle games in real-time using a computer agent (CUA).The results show that Claude 3.7 Sonnet played Super Mario for a full 90 seconds, directly crushing OpenAI, Gemini, and its own predecessors; while GPT-4o crashed right at the start...Google's Gemini 1.5 Pro was defeated in the first battle, and it jumped in a very regular pattern of two steps per jump. By the time Gemini 2.0 came, it took a few more steps, but still ended up falling into a pit.<img src="https://img.36krcdn.com/hsossms/20250303/v2_d0b6026fdc92485eafee7863c0d576e9@5888275_oswg230079oswg1080oswg402_img_000?x-oss-process=image/format,jpg/interlace,1">The GamingAgent project code has been open-sourced, and you can download and install it to watch the AI game showdown.<img src="https://img.36krcdn.com/hsossms/20250303/v2_058060cd6a5044179a06209055927623@5888275_oswg127148oswg1080oswg547_img_000?x-oss-process=image/format,jpg/interlace,1"><h2>GPT-4.5 is slow to react, and GPT-4o is always killed by the first enemy</h2>GPT-4o is always killed by the first enemy, just like a gaming noob who gets flamed by teammates.The game ends in just 20 seconds.<img src="https://img.36krcdn.com/hsossms/20250303/v2_becee1ccd1e74b6fac255545ed5e7fb7@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">In comparison, GPT-4.5's performance is much better, at least it doesn't get stuck at the first enemy.<img src="https://img.36krcdn.com/hsossms/20250303/v2_93084855615c4738982f8891bd7abd1c@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">But its reactions are still very slow, almost two steps and then a pause.Before jumping over a low pipe, it also hesitates for a moment, feeling like it just learned the game controls and is still learning to walk.<img src="https://img.36krcdn.com/hsossms/20250303/v2_6f714495a44747469c8dfe4da38e1c98@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">For a slightly higher pipe, it tried 7 times and took 10 seconds to jump over it.<img src="https://img.36krcdn.com/hsossms/20250303/v2_eb7553f27d3149069f121799aa7a6a92@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">After finally jumping over, it ran into an enemy and died. The first round ended like this.<img src="https://img.36krcdn.com/hsossms/20250303/v2_a48f060c82d74c7387034c077512ba86@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">Even funnier, in the second round, GPT-4.5 again stumbled at the first enemy. After all, it and GPT-4o belong to the OpenAI family, so their gameplay is rather poor (bushi).<img src="https://img.36krcdn.com/hsossms/20250303/v2_63ce158da5b146dab7ee84849a82212f@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">The third round's performance was also rather average, not as good as the first round. It got stuck at the first low pipe for nearly 10 seconds before remembering to jump.<img src="https://img.36krcdn.com/hsossms/20250303/v2_0c863a54bd39473795f39c3fb05b7df6@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">Finally, it smoothly jumped over the second pipe, but was still killed by an enemy, not going as far as the first round, where it at least jumped over the third pipe before being killed.<img src="https://img.36krcdn.com/hsossms/20250303/v2_115c366cdb5f4ebe8ae3bdecfb58741b@5888275_img_000?x-oss-process=image/format,jpg/interlace,1"><h2>Gemini 1.5 jumps every two steps, 2.0 falls into a pit</h2>On the Google side, Gemini 1.5 Pro didn't fare well in the first battle either, unable to escape the clutches of the first enemy.<img src="https://img.36krcdn.com/hsossms/20250303/v2_8c6d40d9bee242c29dd3be062bd7bc9e@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">In the second round, Gemini 1.5 managed to evade the first enemy, even encountering a question mark box and getting a mushroom.<img src="https://img.36krcdn.com/hsossms/20250303/v2_f98ddef1c2c043fa8c504ac66786208b@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">Interestingly, unlike GPT-4.5's two-step-one-pause, Gemini 1.5 is "two-step-one-jump".After walking such a short distance, it jumped a total of 9 times. Jumping on the floor, jumping on the pipes.<img src="https://img.36krcdn.com/hsossms/20250303/v2_d277e5dbd24f42278d548fc07c293123@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">In the end, it also jumped over the third pipe, and even almost made it over the fourth, going farther than GPT-4.5.<img src="https://img.36krcdn.com/hsossms/20250303/v2_7555b7c33d804c30969f3695dda50137@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">As for the updated Gemini 2.0 Flash, its performance is unsurprisingly much better.First, it jumps more boldly; secondly, the jumps are also more fluid.It reached platforms that its "predecessors" had never set foot on, and easily jumped over the first three pipes in 10 seconds.<img src="https://img.36krcdn.com/hsossms/20250303/v2_546d0202e1e940eba9531acca29efc4f@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">Although in the second round, it also fell victim to the first enemy.<img src="https://img.36krcdn.com/hsossms/20250303/v2_c6b711f040974454b37e3f4ed898952a@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">But in the end, it went farther than the OpenAI family and Gemini 1.5 - it jumped over the fourth pipe, but fell into a pit it couldn't jump over.<img src="https://img.36krcdn.com/hsossms/20250303/v2_2b13c2983e4e4d36a32bdf06424af490@5888275_img_000?x-oss-process=image/format,jpg/interlace,1"><h2>Claude 3.7 Sonnet discovers hidden rewards</h2>In comparison, Anthropic's Claude is much more impressive.<img src="https://img.36krcdn.com/hsossms/20250303/v2_74f3a95b3b73408b945f0f175d1cf300@5888275_oswg79579oswg864oswg521_img_000?x-oss-process=image/format,jpg/interlace,1">Compared to Gemini's two-step-one-jump operation, Claude 3.7's operation is more fluid, and it goes much farther.Especially in the timing of the jumps, it seems more methodical, only jumping when encountering pipes or pits.<img src="https://img.36krcdn.com/hsossms/20250303/v2_d92037cfed76469583fc71546e934c5e@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">In addition, there will also be conscious jumps to avoid small monsters.<img src="https://img.36krcdn.com/hsossms/20250303/v2_2b8b148cb9f9428098faf040b4f76462@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">Skipping the Gemini 2.0 Flash pit twice without jumping over it, Mario under Claude's operation finally ate the gold coins; finally encountered small monsters other than goblins (mushroom-like) - Koopa (turtle-like); even encountered a hidden reward - the Super Star.<img src="https://img.36krcdn.com/hsossms/20250303/v2_96c570d6d8d84a1da615c076cc105caa@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">Finally, fell into the pit between the stair platforms, ending the game.<img src="https://img.36krcdn.com/hsossms/20250303/v2_3f6f4106edcf48639245779320b00d2c@5888275_img_000?x-oss-process=image/format,jpg/interlace,1"><h2>AI Battle 2048 Puzzle Game, GPT-4o Can't Show Off</h2>Next, let's take a look at another puzzle game, 2048.Many people may not be familiar with this game, the rule is to merge blocks with the same number by sliding, and the player tries to reach the highest possible value.During the challenge, GPT-4o got stuck due to prolonged thinking.While CLAUDE 3.7 took a few more steps, it was much stronger than GPT-4o, but still ended in failure.<img src="https://img.36krcdn.com/hsossms/20250303/v2_fb9126b198494b24af498f0be8ec5816@5888275_img_000?x-oss-process=image/format,jpg/interlace,1"><h2>Tetris, Intelligence Online</h2>So how did CLAUDE 3.7 perform in playing TETRIS?<img src="https://img.36krcdn.com/hsossms/20250303/v2_0587d7a43e3441938af678f325791203@5888275_img_000?x-oss-process=image/format,jpg/interlace,1">Anthropic's Head of Developer Relations, Alex Albert, praised, "Very cool! We need to turn every video game into an assessment tool".<img src="https://img.36krcdn.com/hsossms/20250303/v2_9ef4b120899748c688c5f1d9f0eb8518@5888275_oswg57305oswg1080oswg100_img_000?x-oss-process=image/format,jpg/interlace,1">There are already netizens in the comments wishing for Grok 3 to join the battlefield.It seems that the LLM assessment is about to open up a whole new path.References:https://x.com/haoailab/status/1895557913621795076https://x.com/haoailab/status/1895605453461340472https://lmgame.org/#/aboutusThis article is from the WeChat public account <a rel="nofollow" href="https://mp.weixin.qq.com/s/IAcXyI32rfNqqu8lWkShmg">"New Singularity"</a>, author: New Singularity, authorized by 36Kr for release.

Claude 3.7 hard controls Mario for 90 seconds, GPT-4o dies at the beginning, Karpathy calls the benchmark invalid, and games become a new battlefield for LLM

Here is the English translation, with '<>' content retained as is:

On February 28th, Open AI launched a major move, with the debut of GPT-4.5, which is touted as "the largest, most knowledgeable, and most expensive AI large model in history." CEO Sam Altman even praised it extensively on Twitter, saying it is the "best, most thoughtful model" he has "ever discussed," and that "this thing makes me feel like AI is a person for the first time!"

However, the launch event was dramatically full of surprises: while Altman lavished praise on GPT-4.5 online, he was absent from the launch event itself, as he had just become a first-time father and was fully occupied with caring for his newborn at the hospital.

From 2 years...

Instead of intelligence, compete with emotional intelligence? That’s all GPT-4.5 has?

The Government of Vietnam is promoting the development of a legal framework to manage digital assets and cryptocurrencies in order to create a more transparent and favorable environment for businesses and investors. According to the new requirements from Prime Minister Ph...

Prime Minister requests to submit legal framework for digital currency in March 2025

On March 2, US President Donald Trump announced the establishment of a national cryptocurrency strategic reserve, marking a historic moment for the Bitcoin market. According to TradingView data, the spot market saw an inflow of over $300 billion after the announcement, driving the Bitcoin price from $85,000 to $95,000, and the largest $10,000 gap in the history of the Chicago Mercantile Exchange (CME) Bitcoin futures market was formed.
This article will provide you with a detailed explanation of what the Bitcoin CME futures are and why they are so important to the Bitcoin trend...

Download app to participate in rewarding events