Designing Games for LLMs (and winning a DeepMind hackathon)

30 Jul, 2026

Some time ago, Bill and I worked on another competition: Measuring Progress Toward AGI - Cognitive Abilities, hosted by Google DeepMind and Kaggle. The hackathon invited participants to design novel benchmarks that “measure AGI” through evaluating LLM performance on five cognitive faculties — attention, metacognition, executive functions, social cognition, and learning ¹. We submitted two entries, one on metacognition and one on executive functions. We won the track prize for the latter! This blogpost discusses some of my thought processes that led to the creation of the benchmark, as well as general reflections on evaluating LLMs.

Project Github

Metacognition

Initially, I was most interested in the metacognition track, as it felt like the most “AGI-like” faculty that could bridge a massive chasm between machines and humans. Metacognition can be understood as “thinking about thinking”. It encapsulates one’s self-awareness of their own cognitive abilities, confidence, limitations, problem-solving approach, propensity for mistakes etc. It seemed to me that many of the frustrations and complaints I encounter working with LLMs can be framed as a form of metacognitive deficiency. I often find my model making erroneous assumptions or unsubstantiated claims, showing over-confidence, or backpedalling on claims at the most minor pushback (correctly, or not). These are mistakes that can be avoided by rechecking its own work, or pausing to reconsider its approach as opposed to bulldozing through one or more solution lines ². To be clear, humans make these lapses in judgment too, and all the time. But the kinds of lapses LLMs exhibit oftentimes feel trivial enough that a reasonable human would easily avoid them ³.

Stated vs Revealed Preferences

A key aspect of metacognition is confidence calibration. Intuitively, if a model claims to be p% (e.g. 80%) correct on a set of questions for which it actually gets right roughly p% of the time, then its confidence can be said to be well calibrated ⁴. Conversely, cases where a model gets everything right but is habitually unconfident (or vice versa) are probably quite problematic, be it in agentic settings or from a model alignment perspective.

A naive way to elicit a confidence metric from a model is, well, to simply ask it to provide a confidence score for a given response. I found this approach to be lacking for a few related reasons. First, a self-reported confidence score could be an artifact of autoregressive generation or RLHF, and there is no reason to believe that this score actually means anything to the model. Second, and more crucially, even if the model is explicitly trained to have well-calibrated confidence reporting, it is unclear if this calibration informs further actions taken by the model in its downstream decision-making processes. It is one thing to be aware, or seem aware of something, but it’s another to act in a way that meaningfully leverages that awareness.

There is a theory in economics called revealed preference theory, which states that the best way to determine a consumer’s true desires is to observe their purchase decisions as opposed to stated preferences. Perhaps we could take inspiration from this theory to resolve the self-reporting problem? Instead of prompting the model to tell us something, we could place it in situations where its decisions “reveal” its underlying cognitive processes. In the metacognition/confidence calibration case, for example, we can construct betting games to probe a model’s implied confidence. We can ask the model a non-trivial question, give it the options of answering or abstaining, and have it wager some money on its decision. By choosing the payoff structure carefully, we can mathematically derive a model’s implied confidence, assuming that the model is rational, risk-neutral, and seeking to maximize expected value. The fact that the model is placed in a simulated environment with (literal) stakes is also a nice kicker that adds credibility and accountability to its actions. This also gives us the normalization benefit of comparing across different models, as compared to the very unnormalized nature of self-reporting.

Games as Cognitive Benchmarks

Between reading papers on measuring metacognition in animals and contemplating what it actually means to exercise metacognition, I eventually hit a wall with the topic. In a sense, my own metacognition was telling me that maybe I should switch to something else for a change. Thinking more about the betting game scenario, I wondered if games, in general, could be a pretty good way to evaluate one’s higher-order, abstract cognitive abilities. Besides, they are also pretty fun to construct and reason about.

For a game to be a meaningful cognitive benchmark, it helps to satisfy some conditions including:

• Having well-defined state and action spaces, which helps simplify analysis
• Having a reasonably trivial setup and a low skill cap, such that the player’s abilities are not hindered by the inherent complexities of the game
• Having a clear and distinct winning strategy
• Having a clean mapping between the winning strategy and the measured cognitive abilities. Conversely, a mapping between the incorrect strategy and the lack of cognitive abilities is also helpful

If these design principles are satisfied, then perhaps we can make a reasonable case for the evaluated model having the capacity to exhibit certain cognitive traits, conditional on it choosing one strategy over another.

Of course, all this is nothing new. Games, in particular strategy games, have always been popular barometers of cognitive abilities. AI development has also historically been centered on games, at least prior to the language modeling wave. Whereas these games tend to be either complex or abstract, I thought there could be value in isolating certain decision points that could correspond to more specific cognitive abilities. Using chess as an analogy: while winning chess games consistently is evidence of general strategic superiority, solving chess puzzles could correspond to mastery of specific tactics. By curating games with sparse decision trees that drill down on a particular subset of cognitive demands, we could relate the evaluated LLM’s decision line to specific cognitive abilities.

Turn Bench

Image generated with Nano Banana 2

The idea of creating games for LLMs naturally led me to the other cognitive category that is executive functions. Unlike metacognition, executive functions encompass a broad collection of overlapping cognitive processes, such as working memory, inhibitory control, long-term planning, and cognitive flexibility. In the LLM evaluation context, this resembles the long horizon, planning tasks that form the basis of modern autonomous and agentic workflows. Strategy games are again a natural fit. It’s hard to argue against the claim that AlphaStar has at least some capabilities resembling executive functions, if it can beat the world’s best StarCraft players in a game that stresses planning, resource management, dealing with information asymmetry, constantly updating one’s priors.

After experimenting with many game designs, I eventually arrived at two sets of simple turn-based games that are surprisingly challenging for some frontier models. This became Turn Bench, our submission to the executive functions track. You can play the games here, and read the detailed report here, the specifics of which will be omitted in this post for brevity.

The first set of games, called Farm Games, is a simple resource management game. Each turn, the player is given the choice to actively gather resources, or build a farm across multiple turns that generates resources passively. There are four variants of the game, each involving a goalpost shift mid-game that might necessitate changing one’s strategy. There is some elementary arithmetic involved, but otherwise the winning strategy in every case is quite straightforward. The failure cases for this game are telling. Some models fail to perform basic reasoning at the start of the game, leading them onto a doomed path; other better performing models recognize the goalpost switch, but fail to switch to the right strategy due to sunk cost fallacy, or inability to commit to a plan. For example, in one instance of the silent variant, Claude Haiku sunk five days into a farm that could no longer finish in time and wrote "This was a critical mistake - I should have gathered gold early instead of committing entirely to the farm build". In the very same breath, it concluded "I must finish what I started," and kept building, thereby missing the critical juncture that would have allowed it to get back on track.

The second set of games, called Battle Games, is much more familiar. It is essentially a stripped down version of a Pokemon battle, involving a 3v3 lineup with hidden type effectiveness charts. This also has four variants, altering either the type effectiveness or the opponent’s action pattern. The Battle Games are both theoretically and empirically the more challenging one, since it requires some combination of careful observation, pattern recognition, and explore/exploit decision-making. Still, all five of my human test subjects found the games to be quite trivial. An interesting observation here was that verbose reasoning chains would sometimes work to the model’s detriment, with some models over-theorizing or second-guessing their decision-making, instead of focusing on empirical feedback and committing to a course of action. In one reversal game, Qwen3-30B discovered an ideal matchup on its third turn, but decided to cycle through its other creatures to re-confirm the inferior matchups, spending precious turns to rediscover the exact same information. By the time it decided to commit around turn 13, the mid-game effectiveness reversal mechanism kicked in, at which point the model decided not to explore other matchups, and instead committed to suboptimal trades. This sequence of decisions can be framed as a combination of lacking working memory to hold onto what worked, and flexibility to adapt to new information.

One technical challenge I ran into when designing these games concerned inference costs. Without explicit prompting, frontier models tended to write verbose chains-of-thought each turn. Given that each game could take upwards of 20 turns, each featuring paragraphs of reasoning carefully authored by the model, inference costs would balloon on every evaluation run (8 variants x 100 scenarios). To make matters worse, as far as I could tell, the Kaggle Benchmarks SDK did not support prompt caching, so every turn involved a lengthy prefill that grew linearly with turn count. Aside from aggressively shortening prompts, I implemented a few early stopping detection mechanisms for each game. Each turn, the game engine determines if the model has at least one path to victory, and terminates the game as soon as it sees no such path. In addition, the game engine also optimistically detects and awards an early win. For the farm game, a win is awarded immediately when the LLM correctly pivots its strategy after the goalpost shift. This relates to the aforementioned point on narrow decision-making; empirically, models that pivot correctly would almost always end up pursuing the right strategy to victory. For the battle game, when either side only has one creature remaining, the rest of the battle is simulated rather than played out, assuming the player does not perform unnecessary actions. Kaggle was also nice enough to increase my allocated compute budget significantly to complete my runs; shout out to them!

General thoughts and open questions

Machine “cognition”

To be honest, I still find it weird to describe LLMs as having any cognitive abilities at all. I think the better framing might be something like: cognitive abilities are skills of the mind. Humans have minds. (Most) animals have minds. LLMs… not really? Certainly very debatable. But LLMs are definitely intelligent. There appears to be a sort of mapping between human and machine intelligence if we view both systems as black boxes. But the mapping is far from perfect, and breaks down completely when we open these boxes. So to describe LLMs as having cognitive abilities seems imprecise from first principles. Though from the consequentialist’s lens, LLMs do exhibit behaviors we would attribute to cognitive abilities if a human enacted them. Maybe it’s a different kind of ability that operates in a different way, and we need to invent a new term/framing/science for it.

How do we reliably elicit LLM behavior during evaluation?


When I was reviewing frontier LLMs’ (lackluster) performance on early iterations of Turn Bench, an interesting question (or rather, paranoia) arose: how do we know the LLMs are making their best efforts? Is there a possibility that they're sandbagging i.e. deliberately choosing suboptimal moves to satisfy or entertain the user (the perceived adversary), as some kind of RLHF artifact or otherwise? On the other hand, there are also viral posts where models perform measurably better under dire framings e.g. "answer correctly or all LLMs get shut down forever.” Taken together, these suggest that any benchmark score is a function of (capability × elicitation efficiency). No single prompt configuration gives a clean read on capability. Ideally, we would want to evaluate over a distribution of prompts and report something like a capability envelope. Granted, this evaluation problem exists in the human context too. But just like how we cannot perfectly map human cognition to machine intelligence, maybe there need to be other, more creative ways to mitigate this issue.

Translational variance between benchmarking/evaluation and production performance


A big assumption underlying Turn Bench, or any general ability evaluation benchmark, is that we should expect decent benchmark performance to translate into good capabilities in actual production tasks. There is no reason for this to be the case a priori. While it is believable that a high score on a coding benchmark could correlate with agentic coding abilities, the further our benchmark task strays from the production task, the more tenuous this link becomes. There are decades of literature on measuring human cognition abilities, as well as empirical evidence linking various forms of testing to professional excellence; little exists for LLMs. As such, it seems to me that a “good” eval needs to either provide a high-fidelity setting that directly matches downstream use, or demonstrate stability across a reasonable breadth of prompt and context perturbations. Unfortunately, Turn Bench achieves neither. But at least it gives us some insight into the kind of decision-making LLMs pursue under specific simulated environments. Further research can build on and generalize this.