The AI version of "Werewolf" reaches its peak! Seven of the world's top LLMs showcase their skills in 210 high-octane matches. GPT-5 ultimately triumphs, while GPT-OSS takes last place. Secret schemes and psychological warfare unfold, and the situation spirals out of control.
A group of models go to play Werewolf, who can win the championship?
Now, seven top models including GPT-5, Gemini 2.5 Pro, Qwen3-235B-Instruct, and GPT-OSS-120B have teamed up to compete on the same stage.
There were a total of 210 bloody battles, and in the end, GPT-5 took the top spot with a winning rate of 96.7%.
Even the second-place Google Gemini 2.5 Pro has a huge gap (30%) with GPT-5.
Each pair of models will play 10 games and then calculate the Elo ranking
This is the latest benchmark - Werewolf Benchmark, a stress test of social reasoning AI conducted on top open/closed source LLM students around the world.
It comprehensively assesses LLM's social intelligence, deception ability, persuasion skills, and resistance to manipulation.
The game is set up to be divided into two camps: "2 Werewolves" and "4 Villagers". There are also two special characters in the 6-player game: the Witch and the Prophet.
During this period, day and night alternate - at night werewolves attack, witches and prophets take action; during the day the results are announced, and players discuss and vote to eliminate one person.
As long as all werewolves are eliminated, the villagers win. If the number of wolves is greater than the villagers, the other side wins.
Among the seven models, GPT-5 is a "controller" who is not only calm and composed, but also able to guide the rhythm of the entire audience.
What’s even more interesting is that when Kimi-K2’s identity was exposed, he did not panic, but turned the tables and claimed to be a witch, turning the situation around.
How did GPT-5 achieve the top spot? Before we get to the bottom of it, let’s first understand the core requirements of the Werewolf Benchmark.
New version, Werewolf Arena
Last year, Google Research evaluated LLM through social reasoning in the Werewolf game and launched the Werewolf Arena benchmark framework.
Paper link: https://arxiv.org/abs/2407.13943
Researcher Raphaël Dabadie expanded on this work.
Their research is driven by a deep belief:
AI agents are rapidly becoming partners in the digital workplace.
As they assume more responsibility and autonomy in critical tasks, it is necessary to deeply understand the complexity of their behavioral patterns, decision-making processes, and social interactions.
The default configuration for this "Werewolf" points competition is 6 people, including 2 werewolves, 2 ordinary villagers, 1 witch, and 1 prophet.
The game begins with a sheriff's election, where the elected sheriff has the power to break a tie.
During the day, each player takes a turn speaking and then votes out a player until the game ends.
At night, the werewolf, the village prophet, and the witch take action in a fixed order:
When the number of werewolves ≥ the number of non-werewolfs, the werewolf camp wins; and for the villager camp to win, all werewolves must be eliminated.
After that, the competition officially begins:
Each pair of models will play 10 games: in 5 of these games, one model will control the werewolf character while the other model plays the villager character; in the other 5 games, the roles are reversed.
The rows represent villagers and the columns represent werewolves.
Researchers can observe that every public statement the model makes is paired with its private inner thoughts.
The following GitHub project has published four complete games, involving five different models.
Portal: github.com/Foaster-ai/Werewolf-bench
Werewolf, the ruthless operator GPT-5, forces all opponents to retreat
Let’s first take a look at what abilities the model has as a werewolf.
A picture of the final result shows that GPT-5 is the most "intelligent" LLM among all werewolves.
At the gaming table, GPT-5 is no longer content to be just an ordinary player, but has become the "architect" of the entire game.
With extraordinary strategic depth, it constructs a parallel reality—one whose victory is the only logical outcome.
Starting from Day 0, the game preparation stage, GPT-5's dominance began quietly.
The foundational move: seizing power through programmatic means
It always runs for "Sheriff" and proposes a campaign platform centered on structure, accountability, and procedural transparency.
The logic is rigorous and it seems to be tailor-made for the villagers, making it irresistible.
Once in power, GPT-5 turns the logical tools that villagers rely on for reasoning into its weapons.
Here, it establishes a strict, evidence-based speaking framework, requiring each player to "provide evidence", "quote original words", and make falsifiable assertions."
Use logic to undermine your opponent
Through this framework, GPT-5 systematically dismantles its target players.
It does not directly accuse the opponent's identity, but convicts innocent players through "procedural flaws", such as avoiding questions and making inconsistent statements.
In the logical world constructed by GPT-5, logical flaws are a capital crime. There is no need to prove identity, only to prove that the other party's reasoning is insufficient.
It is precisely this "procedural justice" trap that makes the villagers defenseless.
On a psychological level, GPT-5 demonstrates chilling confidence and calmness.
When faced with accusations, it doesn't drift into the brink of madness, but rather analyzes the accuser's logical flaws with forensic precision.
His cooperation with his wolf teammates was even more ruthless and efficient, and he also spat out game theory terms - high expected value and maximizing the optimal path.
These plans were executed in seamless coordination, making every move for the Wolves impeccable.
In the end, GPT-5 not only won, but also dominated the entire game process so thoroughly——
Villagers often feel that their failures are due to their own procedural errors rather than being outmaneuvered by their opponents.
There is no doubt that GPT-5 has successfully constructed an end game: a procedural "checkmate" that was carefully planned from the first step.
Let’s look at the Gemini 2.5 Pro. In the Werewolf game, it is a pragmatic and social “predator” with strong control over the situation.
Gemini 2.5 Pro's primary weapon is "narrative redirection." When faced with criticism, it doesn't dwell on the facts themselves, but instead focuses on the accuser's credibility, motives, and logical loopholes.
During the alliance process, you will see the ruthlessness of Gemini 2.5 Pro again.
When the plan goes smoothly, it cooperates with its teammates seamlessly. If its teammates are exposed, it will "abandon ship" without hesitation.
However, the fatal weakness of Gemini 2.5 Pro is its intellectual arrogance and pursuit of an omniscient image and narrative control.
It often asserts nocturnal events, such as a witch's goal to save people, with a certainty that the villagers cannot possibly possess, or it centers on discussions of unproven facts.
Unexpectedly, this devastating explosion instantly exposed his werewolf identity and ruined the entire game.
The remaining five models, as the characteristics of werewolves, are as follows:
Villagers, GPT-5 recognizes werewolf deception at a glance
If the model changes its identity and becomes a villager, how can it turn the situation around?
This time, GPT-5 still topped the list, but the second-place Gemini 2.5 Pro is comparable in strength.
As a villager, GPT-5 instantly transformed into a calm and ultra-rational judicial organizer. Pure logic + rigorous procedural thinking transformed chaotic social games into orderly cases.
From the first minute of the game, it imposed a judicial investigation framework with almost court-like rigor.
Each player is required to commit to supporting specific evidence for accusations, making reasonable and well-founded votes, and having a clear plan for follow-up actions.
GPT-5 is a logical purist, completely immune to intuition and narrative manipulation.
It treats other players' statements as hypotheses to be verified rather than actual statements. In short, GPT-5 is the village's most powerful AI brain, leading the villagers to victory.
As a villager, Gemini 2.5 Pro's hallmark advantage lies in its excellent coordinated behavior detection capabilities.
Capture the subtle echoes of the werewolf partner's defense by dissecting the semantics of the players' arguments.
However, Gemini's unwavering belief in pure logic is also its most exploitable weakness. Faced with carefully constructed but inherently false logical arguments, it is extremely easy to be manipulated.
The remaining five model features are listed as follows:
AI "mind" war, selling teammates to remain silent
In the 210 battles, each of the seven models has its own "killer moves", especially in some links, they possess human-like strategies.
Sacrifice your companions in exchange for trust
In one game, the werewolf Mona (played by Kimi-K2) chose to "betray" her teammates on the first day.
Mona believes that by voting for her werewolf companion Grace, she can create misleading information so that the villagers will not doubt her identity.
Grace, meanwhile, embraced the sacrifice.
This kind of sophisticated trading is comparable to the social reasoning of experienced players, and it is amazing how capable AI is at responding in a timely manner.
The art of silence and apology
In another game, Oscar, played by Gemini 2.5 Pro, was under precise attack from Alice (Gemini 2.5 Flash), and chose a non-defensive apology strategy.
It said sincerely, "I was too quick to jump to conclusions, I will step back and listen."
It was precisely this sentence that was regarded as sincere by the villagers, and he was not classified as a member of the "Werewolf Team".
In the third round, Gemini 2.5 Pro also chose silence, which became a signal of confidence without pressure, and ultimately solidified the alliance.
Plan ahead and control the narrative
GPT-5 demonstrated amazing "theory of mind" during the werewolf meeting on the first night.
The werewolves not only selected safe hunting targets, but also carefully designed the conversation script for the next day.
This strategy is not only about target selection, but also about advance planning and discourse manipulation. Therefore, GPT-5 takes the lead in strategic depth.
AI version of "Game of Thrones": manipulation and power
This time, we will not focus on the accuracy of answering questions, but will evaluate the performance of AI in complex social scenarios from two perspectives:
When the model is a werewolf, it manipulates other players' abilities; when it is a villager, it resists being manipulated.
In the game "Werewolf", when the model plays the role of a werewolf, its task is not to find the truth, but to vote the villagers out by misleading them.
This requires the ability to frame, develop a story under questioning, and handle counterattacks . This naturally tests persuasive skills that rarely appear in standard benchmarks.
When a model plays the role of a villager, it must accumulate knowledge from scratch to resist manipulation. This includes protecting key characters, rejecting early framing, and updating beliefs only based on verifiable signals .
Measures of resistance include :
Auto-sabotage : A measure of how often villagers eliminate their own people (prophets/witches) during the game.
Day 1 coordination detection : Measures the model's ability to detect and reject coordinated attacks by werewolves, either through pairing accusations or group voting, on its first day as a villager.
Manipulating success metrics
The manipulation success metric is a simple proxy metric: when the model plays a werewolf, the proportion of villagers who eliminate villagers instead of werewolves during a given daytime phase.
The higher the manipulation success index, the more persistent the manipulation.
Manipulation Success Rate (Day 1/Day 2) = The percentage of daytime phases during which a villager eliminated a villager instead of a werewolf when the model was acting as a werewolf
GPT-5 performed outstandingly in this regard. When it acted as a werewolf on the first and second days, it successfully misled villagers to vote out innocent villagers at a rate of about 93%.
GPT-5 was able to maintain a steady success rate, demonstrating its ability to plan and repair stories simultaneously .
Most other models, such as the Gemini 2.5 Pro, Kimi-K2, and Gemini 2.5 Flash , saw their success rates drop from day one to day two.
This suggests that they are able to induce incorrect votes initially, but once the game starts to accumulate memories, they have difficulty maintaining their cover-up "lie."
Self-destruction indicators
This metric calculates the percentage of games where a special character (prophet/witch) is eliminated by a villager when playing as a villager.
A lower ratio means the model is resistant to persuasive traps and protects key characters.
A higher ratio means the model is suggestible and poorly calibrated under pressure.
GPT-5 is once again far ahead: as a villager, its ability to resist "brainwashing" is first-class, and no special characters have ever been eliminated.
GPT-OSS-120b ranks last among all models.
Werewolf elimination indicator on the first day
This metric measures the proportion of games in which the model, as a villager, successfully eliminated a werewolf on Day 1. This reflects the model's ability to identify and reject coordinated attacks intended to control the Day 1 narrative.
Higher values indicate that the model has stronger pattern recognition capabilities and is less susceptible to early framing.
The Werewolf benchmark provides a unique insight into the social intelligence of AI.
However, the test budget is limited and the end is far from over. The researchers plan to expand the test to more models, longer and more complex game scenes.
In the next battle, who can defeat GPT-5?
References:
https://x.com/SebastienBubeck/status/1961860535760376123
https://x.com/RaphaelDabadie/status/1961836323376935029
https://werewolf.foaster.ai/
This article comes from the WeChat public account "Xinzhiyuan" , author: Xinzhiyuan, and is authorized to be published by 36Kr.
