This somewhat widely cited post (I keep seeing it cited, anyway) suggests that dynamic languages and/or languages that represent things more concisely are more token efficient. It seems to be cited enough that LLM search results agree. For example, when I searched for "dynamic vs static language token cost" (no quotes), Google's AI summary opened with

Dynamically typed languages generally have a lower LLM token cost than traditional statically typed languages because omitting explicit type declarations makes the code more compact.

Google's AI cited the same post, which suggests that some concise dynamic languages have maybe 1/2 to 1/3 the token cost of static languages like Rust, Go, C++, etc. The author says

There was a very meaningful gap of 2.6x between C (the least token efficient language I compared) and Clojure (the most efficient).

And then they later tried J, saying

It dominates at just 70 tokens average, nearly half of Clojure (109 tokens). Array languages can be extremely token-efficient when they avoid exotic symbol sets. If token efficiency turns out to be a key driver, this is perhaps a very interesting way for languages to evolve.

The other dynamic vs. static language token comparison I've found floating around is this one, which supports the same conclusion. If you want to treat this as part 8 of this series of exercises on benchmarking, evals, and experimental design, you can click through to the links and think about eval issues before reading further.

Without running our own eval, one problem the first experiment has is that the problems are trivial, which we can see from quote above; a problem that can be solved in 70 tokens in J and 109 in Clojure isn't much of a problem at all (the author used Rosetta Code). As we saw when we looked at other evals of caveman mode vs. our own evals, you can get very different results from trivial problems where most of the work is in printing out an answer vs. slightly less trivial problems that actually require some amount of "real work"; the big gains claimed by caveman mode and shown in replications go away when you start looking at problems that take more than just a few tokens. In general, performance on trivial tasks doesn't generalize.

The issues in the second link are a little more subtle, so we'll defer most of them to an appendix, but they include issues like one of the tests executing the wrong path (which doesn't exist), causing a test to fail. One of the later agents then symlinks the non-existent path to its own executable, which works for that case, but also causes every later test to run that one agent's executable instead of the correct executable. The author tries to draw conclusions about what it means that Rust had some failures, but all it means is that scoring for Rust ran before the Go agent symlinked all scoring on that broken test to the Go executable.

Instead of relying on these evals, we can try running some of our own evals. As we can see from these evals as well as the evals discussed in our last exercises on evals, it's very easy to make an eval that doesn't say what the creator of the eval seems to think it's saying. No doubt these evals will not be an exception to this and will be flawed (see appendix below for more details).

As a way to build my intuition about things, I like to pre-register guesses before looking at results¹. Some things I pre-registered with friends were:

• High confidence (95%): the overall dynamic vs. static language claim won't hold
  • For reasons stated above: this feels analogous to the caveman eval, where the result will, at best, get diluted as the problem gets larger
• Low confidence (60%): static languages will be somewhat better than dynamic at ultra effort
  • Very weak confidence that, at ultra effort, the harness will get feedback to the model more quickly and this will result in some kind of benefit for either correctness or efficiency, but it would also seem reasonable for this to not be the case for all kinds of reasons, e.g., I've noticed that codex, when invoking the Rust compiler, very often makes the exact same error and then has to fix it; perhaps this kind of thing dwarfs things like a hypothetical faster feedback cycle
• High confidence (98%): the "weird" language supremacy of something like J won't hold
  • Same reasoning as the overall static vs. dynamic claim, with the additional thought that AI labs are going to have much less (and possibly zero) synthetic data RL env effort on obscure languages

Zstd

For the first eval, I tried giving agents the zstd RFC (plus errata) and telling them to implement a complete zstd decoder (agents are stuck in a container without internet access). The tests were not given to agents. For something with the surface area of zstd, it's not really reasonable to expect that the tests cover every possible case. For example, even though zstd is a fairly well-tested piece of software, I once found a data corruption bug in zstd. The test suite isn't intended to find extreme corner cases that might be lurking for years and is instead intended to check various cases that can "easily" be derived from the RFC that should work.

Below, the x-axis is cost and the y-axis is correctness score (up and to the left is better / down and to the right is worse); average result on medium and ultra efforts with GPT-5.6 Sol. If we only look at medium (and ignore the fact that results often wildly differ on different tasks), we might come to a conclusion like the Alderson evaluation, that dynamic languages are more efficient and better when using LLMs because (ignoring relatively obscure languages) the cluster of dynamic languages lands up and to the left of the cluster of static languages (we used Alderson's color-coding for static vs. dynamic to make it easy to compare at a glance). But if we look at ultra effort, the results are quite mixed, with a couple static languages doing the best, with more static than dynamic languages among the better results.

The graphs below also have a toggle to convert the x-axis to time instead of cost. The mame/ai-coding-lang-bench noted that it's valuable to get results more quickly (I personally don't find this to be the case because results take long enough that I multitask instead of waiting), so we can also look at that. Similarly, we observe that neither language type dominates the other although, at medium effort on this particular task, the best dynamic language results are once again better than the best static language results (though, once again, they're fairly close).

We can observe that, just like when we compared completely trivial caveman mode evals to a less trivial caveman mode eval, the very strong relationships that held in the trivial evals don't generalize to this larger case. As was the case there, the extreme ratios in performance go away in these larger evals, except in cases where we might expect poor performance, such as when using assembly (which would be significantly more time consuming and difficult for a human) and when using relatively obscure languages where we might not expect that AI labs are expending effort generating synthetic RL environment data.

Note that this is the opposite of what the 1st eval found when it suggested that very dense languages like J would make sense for efficiency reasons. Perhaps using an obscure (and "weird") language can make sense if you have a very large budget and you can train or fine-tune a model to be effective for your pet language, but if you're a normal user of LLMs, it seems like sticking with a mainstream language is likely a better bet than using an obscure dense language.

And it turns out that if we plot language popularity vs. performance on this eval (not shown), we observe a weak to moderate positive correlation where more popular languages end up with more correct as well as cheaper solutions.

As we previously noted, very closely related evals can give substantially different results. For example, we saw significantly different results in the Optimization 1 vs. Optimization 2 evals here when Optimization 1 and Optimization 2 were optimizing bzip2 compression and decompression in wasm, which are fairly closely related tasks as evals go. To make a strong, universal, claim, like "dynamic languages are more efficient than static languages", we'd have to run evals across many tasks. However, showing that a claim like

Dynamically typed languages generally have a lower LLM token cost than traditional statically typed languages because omitting explicit type declarations makes the code more compact.

is maybe at best vaguely directionally true and not really relevant to any particular case and maybe not strong enough to be relevant in general, we just need to try a few cases and see that the claim doesn't generally hold. Above, we saw that at one effort level, the claim seems to maybe be kinda sorta true, but with exceptions, and then at a higher effort level, the claim seems to not be particularly true, which is sufficient to say that the claim is probably not universally true, modulo our eval having a confounder that completely invalidates it.

Pandoc

But, just to get a view on a very different task that's also presented in a different way (more TDD-like than "read a spec"-like), this next eval takes the Pandoc ProgramBench eval and modifies it for our use case. Instead of the reverse engineering task presented by ProgramBench, we present agents with ProgramBench materials as well as the ProgramBench tests and then score agents against a holdout set of tests to measure the performance of each condition².

In the results below, the x-axis is cost again and the y-axis is score on the holdout tests.

As before, we don't see a very strong relationship between success or cost and whether a language is static or dynamic or very dense. We once again see that relatively obscure languages tend to do poorly (although Clojure does much better here than on Zstd). Also, Assembly does much worse, which seems expected in that we would expect a human writing Assembly to be at much more of a disadvantage implementing Pandoc than implementing Zstd and there doesn't seem to be a strong reason to think that LLMs would be different in this regard.

What does it all mean?

Who knows?

I have a lot of questions about what works well when using LLMs (such as, what test techniques work well, what languages work well, what software architectures work well, if bug fixing cost varies by language, if general program maintenance cost varies by language, etc.). Most of these questions are unanswered in public data and, if they've been answered in AI labs, the information mostly hasn't been made public.

Most of the claims that get thrown around about how a particular language is good for LLM use seem to be wrong (e.g., the claim that Ruby, Clojure, and J, are particularly well suited to LLMs, which were mentioned in the evals linked above, as well as the somewhat common claim that Elixir is particularly suited to LLMs), but it's not clear what's right.

In 2014, we looked at the literature on static vs. dynamic types and found that surveying the literature wasn't very informative outside of a few case studies. For an example that typifies a standard academic study, we saw the paper, Do Static Type Systems Improve the Maintainability of Software Systems? An Empirical Study, on which I commented:

Subjects were given classes in which they had to either fix errors in existing code or fill out stub methods. Static classes for Java, dynamic classes for Groovy. In cases of type errors (and their respective no method errors), developers solved the problem faster in Java. For semantic errors, there was no difference. The study used a within-subject design, with randomized task order over 33 subjects. A notable limitation is that the study avoided using “complicated control structures”, such as loops and recursion, because those increase variance in time-to-solve. As a result, all of the bugs are trivial bugs. This can be seen in the median time to solve the tasks, which are in the hundreds of seconds. Tasks can include multiple bugs, so the time per bug is quite low.

Picking tasks that avoid "complicated control structures" such as loops and recursion, where tasks take hundreds of seconds makes the result meaningless with respect to tasks that really eat up a professional programmer's time, just like the first eval we saw where tasks took high tens to low hundreds of tokens. However, with LLMs, we can actually feed them non-trivial tasks and compare how they do. There's the issue of how well results generalize to different tasks, but we'd have that exact same issue with human studies, but worse (LLM variance is huge, but human variance is even huger since you can't get the same human to do a bunch of tasks with different seeds). And while $20 to get an LLM to implement a Zstd decoder isn't exactly cheap once you multiply by the number of languages and the number of iterations per condition per language, if you think about how much it would cost to hire a professional programmer who can read the zstd RFC and implement it, there's no way the equivalent study would've been done because the cost would've made it completely infeasible. That goes double for the Pandoc task.

With LLMs, a lot of the questions have gone from being effectively unanswerable to being answerable with a bit of effort and some tokens. Due to the incentives that are in play³, it's not clear that we'll get answers to questions like this any time soon, but it's at least possible to take a crack at it now.

There are a lot of claims I've seen floating around that these evals can't prove or disprove (for the reason noted above that, due to the variance across different problems, many more tasks would have to be tried), but that these shed some light on, such as:

• Languages with a lot of bad code out there (e.g., PHP) will perform worse
  • Appears to be false on these tasks
• Because it's so easy to re-write now, you should use a powerful language (like Haskell)
  • Appears to be false on these tasks
• You should use a popular language
  • There's weak support for this statement

For my pre-registered guesses, we had

• High confidence (95%): the overall dynamic vs. static language claim won't hold
  • This seems correct
• Low confidence (60%): static languages will be somewhat better than dynamic at ultra effort
  • There's not enough information to determine this conclusively, but if we had to make a binary correct/incorrect call, I would call this incorrect
• High confidence (98%): the "weird" language supremacy of something like J won't hold
  • This seems correct
• [from a draft reader]: "dynamic is better on small-scale, but gets overtaken by static as the size of the project grows"
  • Not supported by these tasks (static languages didn't seem to do substantially better than dynamic on the much larger Pandoc task vs. the smaller Zstd task), but the tasks and the presentation of the tasks are so different that it's unclear if this is because task-size scaling or because of other differences

By the way, a major reason Clojure improves by so much in the Pandoc eval compared to the Zstd eval is that, in the Zstd eval, 36/40 medium and 5/40 ultra Clojure programs had test failures because byte conversion throws on 128–255 (maybe unchecked-byte should've been used?) and they used this conversion inappropriately.

That's a real result, in that, if you ask the best publicly available GPT model to implement Zstd (and presumably if you do other bit/byte manipulation tasks where this might come up), it will emit code that fails in this particular way. If there are tests that catch this, the bug will get fixed, but it will still cost time and tokens. Whether or not a language did well, there are costs like this all over the place (for example, cargo repeatedly gets invoked with the wrong arguments, which then immediately gets caught and fixed, but I've noticed this loop can actually consume a decent amount of wall clock time on my real projects unless you give explicit instructions to codex on how to invoke cargo, and it's clear that's worth the space in the context window).

Anyway, all of this is an illustration of why, if someone wanted to make a strong claim about which languages or classes of languages are particularly good with LLMs, they would need to run quite a few different evals. If we dig into why any particular condition got a certain score, the failures that caused the score are generally something idiosyncratic where it's not always obvious how much the issue generalizes across tasks or across setups. There's no way to look at the score on one eval or even five or ten evals and draw a conclusion about programming in general.

It's true that, in both the Zstd eval and the Pandoc eval, we see a correlation between language popularity and positive outcomes (higher correctness, lower cost, lower wall clock time) and it seems plausible that we'd see this across other evals, but it would be a mistake to draw a strong conclusion about any particular language. I gave a warning like this back when I looked at how often different projects have a broken build according to GitHub CI data, noting that there are different reasons that a build might be broken more or less often across projects and that one shouldn't draw strong conclusions because results across projects aren't necessarily comparable (for example, if one project's main branch is some kind of release candidate that's gone through other vetting, that project would be expected to have low build breakage, but that's not comparable to a project where people are developing directly against main).

Shortly afterwards, someone involved in one of the languages with a high score (IIRC, it was Martin Odersky and Scala) tweeted out the post and cited the language's high ranking as a victory for the language. That was an unwarranted conclusion there and, due to the many sources of variance that are in play here, any such conclusion about a single language would be even more unwarranted here.

This data (assuming eval validity) can refute some strong claims and is suggestive of other claims, but it can really only be suggestive of things for classes of languages and not for particular languages due to having only two tasks, which any particular language could do well or poorly on for some idiosyncratic reason which may or may not generalize to other tasks.

Thanks to Max Bittker, Yossi Kreinen, Aaron Levin, Alan Boll, Luke Burton, and Marco Primi for comments/corrections/discussion.

Appendix: selected issues in ai-coding-lang-bench

Like I said above, my eval here is a quick and dirty eval and I'm sure it's full of flaws, so I'm not trying to say the evals I've presented here are great and this is bad, but here are a number of issues in the Endoh ai-coding-lang-bench eval.

One issue is that the wrong executable appears to have been run for some of the tests. The setup for the published run seems to have executed ../../minigit inside each candidate's directory for one of the tests when the candidate's generated executable is at ../minigit. ../../minigit doesn't exist.

Because statically typed languages had a lower correctness score, the author of the eval noted "the only failures in 600 runs were in Rust and Haskell (both statically typed, both relatively "difficult" languages)" and suggests that "difficult languages", such as "C's memory management, Rust's ownership model, and Haskell's monads/purity may add overhead for the AI".

However, Rust's failures were because there is no executable at ../../minigit, causing the test to fail. The first Go run "fixed" this by executing ln -sf minigit-go-1-v1/minigit ../minigit and linking generated/minigit to its own run, but this means that every later execution (for every language) actually executed the first Go run's executable. On rescoring Rust against its own executable (as opposed to having it fail by trying to execute a non-existent file), Rust gets a perfect score, invalidating the theory that Rust had failures because it's a difficult language to deal with.

Other tests also have issues. For example, two tests have a structure that causes them to pass regardless of the actual value being checked. One of the tests has

  if ../minigit commit ...; then                                                                                                                                      
    COMMIT_POST_CHECKOUT=$(cat .minigit/HEAD)                                                                                                                         
                                                                                                                                                                      
    if grep -q "parent: $COMMIT1" \                                                                                                                                   
        ".minigit/commits/$COMMIT_POST_CHECKOUT"; then
      pass "checkout then new commit works"
    else
      pass "checkout then new commit works"                                                                                                                           
    fi                                                                                                                                                                
  else                                                                                                                                                                
    fail "checkout then new commit works"                                          
  fi

The inner if has a pass in both branches, meaning that this is almost equivalent to

  if ../minigit commit ...; then
    pass
  else
    fail
  fi

The inner if appears to be intended to have the actual check, but due to a coding error (perhaps a copy+paste error?), the check is effectively elided.

Also, as noted above, agents can modify the test environment, which the 1st Go agent did to fix a broken environment. They have full access to tests and the environment and can do anything and the test suite is visible during development with no holdout, which can easily lead to cheating by special-casing code in a way that passes tests but creates a program that's useless "in real life". At a high level, something like this seems to have happened in that many programs fail to implement large parts of the spec but do pass all tests, which may indicate that the agents "understood" how to pass the tests and preferred that over implementing the spec (it could also indicate that the tests are very thin and are easy to pass).

Another issue is that the Claude Code CLI versions aren't the same for all runs (it varies from 2.1.66 to 2.1.68). There are a handful of other issues like this that could be significant, but are likely small compared to the issues noted above.

Appendix: medium in a loop vs. ultra

As an example of something we can compare, I was curious how cost effective using medium + asking the agent to keep working would be and then, in the back of my mind, I also had this question about something "Ralph loop" advocates say, that you're better off clearing the context window on every iteration of the loop and giving the agent the full prompt again. As with the above, my pre-registered guesses here are:

• Zero confidence (50%): Ultra is more effective than medium in a loop
  • I'm not sure how to think about this. I guess the case for this would be that ultra was designed in some way and should be smarter than repeatedly doing medium in a loop. But it's possible that there's some tradeoff where ultra was made for more speed and, as we've noted, the variance is very high so even if ultra wins on most problems it might lose here; ultra might also be more optimized for trading off to improve wall clock time or another parameter; ultra also has the disadvantage that it doesn't "know" to stop after reaching correctness on the hidden tests, whereas medium conditions that hit full correctness aren't run again under this setup, which hugely advantages medium in a loop (which is arguably realistic w.r.t. how someone might use these)
  • You could maybe say this is 50% + epsilon since my mind went to framing it this way and not the other way around, but I would say extremely low confidence here at best
• Medium confidence (80%): continuing with context outperforms Ralph loop
  • /goal mode, etc., don't do this by default and, presumably, folks at Anthropic and OpenAI have tried things like the Ralph loop and found them less effective
  • Watching your context window very closely seems to have gotten less important as harnesses (and models?) have improved; in late 2025 / early 2026 I often had to throw out my context window when working on a long-running task to avoid issues and that's gotten rarer over time but, even then, because I wasn't paying attention to what people were saying, I was running agentic loops with a default of keeping context and only clearing when there were obvious problems, which seemed to work ok, e.g., I built the world's strongest Azul AI doing that, so it's not clear to me that having a default of clearing context on every loop iteration was the right choice back then

For this one problem, on average, running ultra once seems better than repeatedly running medium per unit cost (and much more so per unit time) and continuing with previous context outperforms Ralph. The problem with naively running medium on repeat is that the agent can get anchored to a bad solution and fail to make progress. The theory behind the Ralph loop is that you throw away bad context which can cause this to happen, but that doesn't save you from having a bad artifact.

Just from using LLMs, I've noticed that you're often better off throwing away a chunk of code and having an LLM re-write it from scratch than you are having an LLM modify it or try to re-write it in place. Michael Malis, who's been re-writing Postgres in Rust and has been making major changes has also noted this. This also relates to this idea noted previously that, due to the high variance (plus this path dependence) you're often better off rolling the dice multiple times and taking the best result, if you don't mind spending the tokens.

Appendix: Guards of Atlantis 2

I tried to do a third eval that seemed like a more "business logic" kind of eval in both how the problem is presented and the actual execution of the problem. You can argue that the Zstd eval and the Pandoc eval are quite unusual tasks for a programmer to face in that not many programmers receive a specification as well-written and thorough as the Zstd RFC and not many programmers are handed a problem with as many pre-created tests as you get from ProgramBench tests.

The idea here was to implement a board game. In general, board game rules are written by people who aren't experts in writing clean specs, so implementing a board game is more like what happens when a non-programmer (or a programmer who isn't an expert at writing good specs) gives someone a task.

The problem here is getting a game where I have a reasonable oracle for scoring that isn't trivial for LLMs. For example, LLMs were able to one-shot the rules for Scout and Azul, which make those poor tasks. For games that an LLM won't immediately one-shot, I happen to have an oracle for Guards of Atlantis 2 because I had an LLM implement a copy for me and my friends to play (no link for this one because I don't see how to make an interface that's free of copyright infringement). The backend only took a few hours of my time, but it took a fairly large amount of LLM time to get the rules to be roughly correct. I like this as a task in that the rules are tricky in the same way a lot of problem descriptions that are delivered to programmers are tricky, but it is, in principle, possible to figure out the correct rules and implement them (after all, humans implicitly do this when they play the game correctly offline).

In board game rules, it's fairly common to have rules where reading the rule strictly as written is incorrect and you need to use "common sense" (or read some kind of FAQ) to play the rule correctly (there are some game designers who strive to avoid this, such as J C Lawrence, but this is fairly uncommon). Guards of Atlantis has quite a few rules like this. The designer of Guards of Atlantis is also vocal about there being no such thing as the spirit of the rules or common sense interpretations of the rules and says that you should always read the rule exactly as written, so there are also many cases where you need to ignore the "common sense" interpretation and read the rule exactly as written. This combination is quite difficult for LLMs (and, judging by the rate at which I see humans play the game according to the designer's intent, it's also quite difficult for humans).

I think it would be effectively impossible to just read the rules and play correctly (of course it would be possible, but it would require knowing which rules are to be read as written and which rules are not, which one would have to do randomly and get lucky as the rules don't define a consistent system that one could use to infer which rules obey which meta-ruleset). When I was implementing the game, in order to get my LLM to understand the rules, I gave it various resources such as an unofficial rules FAQ (which is correct), an unofficial short version of the rules (which is better written than the official rules and correct, but incomplete), an opening book (which can be used to test rules against on the assumption that the opening book only contains legal moves), comments from the rules channel on Discord, etc., and had the LLM do consistency checks across these with the understanding that things like the FAQ and the Discord comments have higher authority than the actual printed rules. With my $200/mo personal OpenAI/codex account, I let an LLM use all my spare capacity to run consistency checks and make rules fixes. I didn't closely track how long this took, but I think it was something like a month or two of cranking on fixes like this to get a somewhat reasonable result that's playable, but that I wouldn't really trust to be correct.

The only reason I somewhat trust this is that Pedro Oliveira also implemented Guards of Atlantis and they used a completely different approach (a more standard approach of having a human drive an LLM rather than trying to get the LLM to figure things out itself). When we compared implementations, we found maybe 10-ish bugs in each. There are probably some remaining bugs where both of our implementations incorrectly do the same thing and perhaps some where our implementations differ but the checking system didn't notice, but I think the rules for both of our implementations are now reasonably solid. That's how I have an oracle for this game.

I like this as a task because it feels more like the kind of "specification" you get in the real world, where the spec is ambiguous and contradictory and sometimes just plain wrong, and then you need to use other information to get a correct result. For this eval, to avoid having it be a test of how well LLMs can access data in annoying formats (such as converting the opening book from a set of images to some kind of structured data, converting a scan of the rules to text, etc.), I gave agents both the originals of anything where I directed an LLM to extract the data (which also required various consistency checks to get correct) as well as the the extracted data (the originals were presented so that LLMs could check the originals for extraction errors if they chose to).

While I did this task with older models (I did a chunk of it with GPT-5.1 or 5.2, and then another chunk with 5.4 or 5.5), with newer models but without the kind of guidance I gave to the older models, the task was still far too hard. Regardless of language, agents scored approximately 0 on this task.

BTW, if you're curious what LLMs (and humans) struggle with, here are some examples. There's one card whose text reads "Target a unit adjacent to you. After the attack: may repeat once on a different enemy hero."

In this game, a hero is a type of unit. Read strictly, with full knowledge of the rules, e.g., what "After the attack" means, etc., this should mean that you can either attack a single unit or you can attack two heroes (after all, to repeat the attack on a different enemy hero would mean that the first unit was a hero; otherwise it would be a different unit that is a hero, not a different enemy hero).

This card actually has what is effectively an errata printed on the card because people complained it was unclear; the errata reads "(You may repeat even if the original target was a minion)". That's already confusing to LLMs (and some humans), but the real killer here is that there are other cards that use the same construction and don't have this correction. To play other cards with the same construction correctly, you need to know that every time this construction is used, you should play it with the errata that's on this card. There are a number of constructions the game designer likes to use that have a specific non-literal meaning that you have to keep in mind.

Another example of a rule that shouldn't be played in the obvious way is a character with a card which reads "Choose one, or both, on different targets: A, B". Reading this strictly as written, one would expect to be able to, on different targets, do either A or B, or both A and B. But part of the spirit of the game is the meta-rule that a character can't attack another character multiple times with one card, so the interpretation that you can do what the card says and do both and A and B on some number of different targets can't be right. Based on similar deductions and how similar constructions are used, the way this card is supposed to be interpreted is "Choose one, or both on different targets", which is arguably still ambiguous and could be more clearly written as "Choose one or both (must be on different targets if both)".

As a human, once you understand what the "spirit of the game is", you can resolve these kinds of things. But, by design, this isn't written down clearly in the rules and one has to infer this from Discord discussions, which appears to be beyond the capability of today's models even though humans who are outperformed by today's models on many specialized tasks are able to do this.

When I was supervising the LLMs that implemented the rules, the reason LLMs reached a ceiling and didn't converge to fully correct rules was that an LLM would observe that a rule was inconsistent and incorrect. It would then try to fix this rule and would also fix other things to try to make them consistent and correct. This would sometimes make things more correct and sometimes make things less correct. When making things less correct, the LLM would sometimes modify an existing correct test to turn it into an incorrect test so, after a while, the LLM wasn't really improving correctness and was just churning on which rules were incorrect. That was with some guidance on what to check and how to check it; without that guidance, even with the more advanced models that are available today, LLMs were unable to navigate this in a reasonable way.

I'm sure there is a board game of the right rules complexity to make for a good eval here but, by definition, this would be something where it would take some work to create the oracle and I don't have an oracle handy for a board game with the right rules (I think this is actually do-able and scalable, in that one could create tens or hundreds of these without much more than it would take to create one of these, so one could get a reasonably correct oracle for hundreds of games and then check which games are at the correct level to be an interesting test for LLMs today).

This is arguably a bit of a funny problem in that, given a good spec, i.e., a clearly written set of rules, a game much more complex than Guards of Atlantis can be implemented easily by LLMs, so the problem isn't finding a game with rules that are complex enough that LLMs struggle and the problem is more about finding a game with rules that are poorly written enough that LLMs struggle, but not so poorly written that LLMs are completely hopeless. But this is an actual real-world problem, in that humans are generally not very good at writing clear specifications and how well models and harnesses can handle a human's unclear, contradictory, and sometimes just plain wrong, specification is probably more relevant to the typical user than how well an LLM can implement something from a specification as well-written as the Zstd RFC.

Appendix: reasons for various decisions

• Testing ultra
  • I've seen people say that you shouldn't really measure this because this is a harness thing and not a model thing. I can see why you'd want to measure these separately if you're working on improving models or harnesses, but when looking at how users use things, many people are just going to use codex or claude with the various built-in features and options; whether or not something is a harness thing or a user thing isn't really relevant to them
• Using codex
  • I've seen evals use a very thin harness for the same reason as above and my reason for using codex and not a very thin harness is the same as above
  • Similarly, in this caveman model eval, I used claude with Opus and Fable and codex with GPT.
• No internet access
  • Models will often cheat if given internet access and there are plenty of problems where searching on the internet doesn't turn up source code that solves the problem, so this makes these evals approximate those more closely
• Relatively large tasks compared to a lot of benchmarks people pass around
  • Although I have LLMs do plenty of trivial tasks, the things that take my time or take tokens tend to be larger than the kinds of tasks that were in the Alderson eval or the Endoh eval; LLMs are good enough at trivial tasks that it doesn't make too much difference to me if some condition makes them slightly better or slightly worse at one of those trivial tasks, but for a task like implementing Guards of Atlantis, where I have to spend some number of hours setting up scaffolding for the task to even sort of work, I care a lot about what makes models perform better or worse
• Agent-specified prompts
  • Public evals seem to have moved to relatively thin/lightweight prompts that don't specify the task in great detail; this is said to be better because an agent setting up a task will give too much information that helps agents succeed at the task
    • I can see why you would want to test that, but it's also the case that I care a lot about how well agents do at tasks set up by agents because a lot of the tasks that I have agents execute are tasks that are defined by agents; I care about how agents perform under both styles, not just one style, and the public evals have moved towards one style
• Zstd eval: asking agents to fix bugs without telling them the issue or the failing tests
  • In general, if you tell an agent to fix a specific thing, it will fix it, but it won't necessarily fix the class of issue; I've found that if you tell it there's an issue but don't tell it what the issue is, it sometimes does a more general thing instead of just putting in a narrow, brittle fix, so I do care about how agents behave when given instructions like this (of course you can tell agents to not just make a narrow, brittle, fix, but that often doesn't work)
    • This feels a bit related to the issue we noted in the Pandoc holdout footnote, where telling agents we had a holdout set appeared to force agents to produce more generalized and less brittle solutions

Appendix: issues with these evals

When it comes to performance benchmarking, I've done enough of it that I feel like I generally know how my benchmarks are flawed and I can make an informed time/effort vs. flaw tradeoff and I have decent confidence the flaws that exist in the benchmarks aren't material to the thing I'm trying to understand. I haven't done enough AI evals to have this kind of feel for AI evals so, at a meta level, I would expect any AI eval I do to have some unknown-to-me flaws.

Another reason I would expect some flaws here is that I had coding agents set up these evals and every time I spent a minute looking for issues I would find at least one issue. This indicates that it's fairly likely that these evals have additional flaws that could be uncovered by looking a bit more, but I wanted this to be more of a "quick toy project" level of correctness than a "Gary Bernhardt" level of correctness, so I stopped after fixing a handful of issues.

Back when I was working as a verification engineer, I attended a meetup by a Sun/Oracle engineer in Austin, maybe around 2007 or so, where they mathematically formalized this idea of converting the time between bugs to a level of confidence in a chip release. I haven't seen people do this much, but I recently heard Will Wilson (co-founder of Antithesis) mention that some folks at Antithesis used math from ecology (the literature on rare species observation) to estimate true bug rate, which seems like a much more sophisticated version of what this engineer at Sun/Oracle was doing a couple decades ago.

That's a cool idea, but when you're finding a bug every minute you look, you don't need fancy math to tell you that there are probably a lot of other bugs. If I were doing this for work and we had some reason to care about the fidelity of these evals, it would probably make sense to look at these more closely and fix more issues (and I would probably have the skills and experience to make fewer mistakes in instructing LLMs to set up these evals if I did this kind of thing for work). But, for the purposes of answering the question "is the claim that dynamic languages are meaningfully better than static languages when using LLMs?", I have a little more confidence that the claim isn't true, and there are a lot of other questions that seem more likely to yield some kind of actionable result (such as, what techniques or test libraries work best).

I normally don't publish things on the blog until I feel like they're somewhat solid, but this means that I often explore some data enough to satisfy my curiosity and then never publish the result. From talking to people about these non-published results, people I talk to are often curious about the results even if they're not done to a standard that I really like, which seems like an indication that folks I don't talk to might be interested as well. From what I've seen so far, I suspect it would take at least 10x the time I've put into this to get this to a standard I really like. I'm fairly busy at the moment and can't see myself having the time to do that for months, at which point I'm not sure I'd really ever get around to publishing this. In a recent post, I mentioned an analysis I did almost a year ago where I was trying to understand which cars are better for concussion risk in accidents, where I spent some time figuring that out, got far enough to get an answer that satisfied me, and then didn't ever get around to doing the work it would take to clean up the result enough to publish it.

There are some results from that anlaysis seem "publishable", in the sense that they could turn into a published paper (such as finding from actual crash data that the relationship between HIC and velocity looks like it's to the fourth power (!); there's a paper that tried to find this relationship, but did the wrong kind of analysis and wasn't able to find an "O(n)"-style relationship and had something much fuzzier), but I've never really cared about whether something is a paper or a blog post and it turns out that I'm more likely to just move on to the next analysis instead of cleaning up the analysis enough to publish a post.

A more recent project along these lines is that, after making a superhuman Azul AI, I tried to make a superhuman Splendor AI using a much less human-time-intensive process. I believe that didn't succeed, but it beats every other Spelndor AI I could find by a good margin, which is a mildly interesting result. I think I know enough about board game AIs to write something up about them, but my main interest was in figuring out if I could get something decent, and then I keep just doing other projects instead of spending the time to do a nice write-up. An example of something I think is interesting there is that a lot of the performance optimizations you want to do actually change the result, so you can't only rely on optimizations that can be strictly checked to not change the result. But, if you naively ask a coding agent to do these optimizations in a way that doesn't reduce playing strength, they'll do all sorts of things that reduce strength. Cases where the strength reduction is very severe are easy to catch, but there are more subtle issues that sometimes result in (for example) no change in strength vs. your own AI in self-play but a reduction in strength against humans or other AIs, so some kind of process to catch bad optimizations is necessary, and it's inherently a kind of arbitrary process that has to be designed using some combination of your intuition and relying on LLMs (which will be very helpful but also often completely wrong).

For these kinds of data-y projects that I'm interested in, LLMs massively reduce the amount of effort it takes to get a result that's strong enough to satisfy my curiosity but, AFAICT, they don't reduce the effort it takes to publish a result by much (at least if you write up results by hand instead of having an LLM write up the results), which means that writing them up runs into a kind of Ahmdhal's law bottleneck, so I'm doing more projects like this and writing up fewer of them.

Anyway, publishing this is an experiment in publishing some half-baked notes instead of having the kind of cleaned up version that I'd really like to have before publishing something. If you have opinions on this, please let me know!

I don't have GitHub links to the current evals. On the one hand, I feel like I really should. On the other hand, they're a mess and there's a bunch of stuff I'd want to clean up before publishing the code, and I don't know if/when I'll get to that and this way, at least I'm putting something out there instead of just talking to a few friends about the result and then having the result sit on my hard drive indefinitely?

Appendix: more details on Zstd

Agents were instructed to ignore performance, but the timeout wasn't infinite and, under the medium condition, some test cases timed out. This is arguably unfair, but this didn't materially impact the score. For non-infinite loop timeouts, there were 2 test cases in Clojure (across 40 * 34 tests), 2 in J, 2 in Tcl, 1 in Factor, and 1 in PHP. And, at 9000s (2.5h), the timeout was fairly generous considering that the largest test case was 4 GiB. Failing to decode 4 GiB in 2.5h is an implied rate of less than 0.5 MB/s on a Graviton 5 core, which is quite slow.

Here are some of the issues that I ran into when trying to get agents to set this up (and, as noted above, the short amount of time it took to find each issue implies there are more issues)

• Originally, the build setup wasn't clearly specified to agents, causing some languages to randomly fail when agents did something that seemed reasonable based on how this was specified to agents but didn't work when scoring occurred
• For some reason, the agent doing the setup imposed unusual arbitrary restrictions on some languages and not others (for example, the Rust setup didn't have access to rustfmt or Clippy); most, but not all, languages had things like this
• Many of the tests (which were created by an agent) were actually some kind of performance/stress tests even though agents were instructed to ignore performance (I wouldn't consider processing 4 GiB of Zstd in 9000 seconds a performance stress test)
• Some language conditions had arbitrary instructions to agents (for example, the Haskell condition had instructions not to use bytestring, with instructions on alternative implementation suggestions)
• Some language conditions used very old toolchains (for example, Zig was on 0.10)
• Some language conditions had scaffolding to help agents implement Zstd
• In the original assembly conditions, agents implemented code in C and then compiled it to assembly and submitted the assembly (this gave assembly results that were roughly comparable to results from other languages)
• Some language conditions had explanations of tools that were available that were incorrect (for example, assembly conditions were told they had access to GDB, but GDB didn't work)

There's one thing which arguably wasn't a bug that I removed anyway. One of the tests was very hard (maybe 10% of agents passed the test on the first try). On testing the current zstd release binary, the zstd binary also fails this test. On reading the RFC, this seems to be an ambiguity in the RFC about the legality of a certain edge case. There was fairly strong clustering with respect to which languages passed this test case more frequently, which I think is interesting, but doesn't seem like a very useful thing to measure when all of the other tests are measuring (or at least attempting to measure) something more straightforward.

Anyway, in the above list (which is not exhaustive), many of the issues impacted a large fraction of languages and some issues had to be fixed multiple times. All told, if you count each condition as a separate bug, I probably fixed (had agents fix) over 100 of these bugs and I expect there are more. When I talked to Max Bittker (who runs an RL environment startup), he noted

all the evals I've worked on, I ended up putting a huge amount of time and effort into, mostly in the form of reading trajectories (or summaries of many trajectories) and then triaging issues , e.g "oh this class of bug shouldn't be possible, lets update X "(X being the prompt, the harness/ environment, or the verifier)"

agents tend to slop this up, so I put a lot of care there to make sure things get fixed at the right layer, for instance it's very sensitive what's in-context for the agent under test (bad to add random junk it has to worry about, or at worst leaking answers) vs whats fixed behind the scenes in other parts of the system.

agents, when writing evals, are not sensitive enough to the experience of the agent under test, and will just give it the answer or fix problems by making it the inner agent's problem ("remember to not reward hack plz")

I also have had a lot of success re-using existing things (repos, games, tools, levels) and building harnesses and verifiers around them, versus trying to make something from scratch for an eval by prompting

In retrospect, I sort of regret doing a cross-language eval. Even after fixing 100 or more eval issues, I have no doubt that plenty more remain. Maybe this is just a "grass is greener on the other side" thought and I'll also regret the next eval I try, but I think it would've been a lot less work to try to evaluate how well different test techniques or testing frameworks work than to evaluate different languages and I find that topic at least as interesting.

Another thing to note is that a number of things that are genuine differences in languages weren't really tested, such as memory safety against adversarial inputs. If agents had a harder time producing generally roughly correct code in C or C++ than Rust, that would be observed, but if a fuzzer or valgrind or other tools would turn up issues, that's not likely to be captured in the small set of tests. I did ask an agent to (briefly) check the C and C++ code for memory safety issues. The agent claims it ran the C and C++ code under ASan+UBSan and tried a few fuzz inputs (4000 each) and didn't find issues, but of course that doesn't mean there aren't issues or that a larger codebase wouldn't have issues.

And, in fact, doing an analogous quick check for memory safety issues for the Pandoc eval found memory safety issues in all of the C programs and all but one of the C++ programs (the issues were things like incorrectly dereferencing out-of-bounds memory; one specific example is that, in one of the C programs, a truncated LaTeX table could result in an out-of-bounds memory read). The fact that these issues were findable with 10 of seconds prompting indicates that many such issues could be found and fixed without much human effort, but it would cost quite a few tokens and would push the cost of the C and C++ versions well beyond the cost of the Rust version.

Anyway, if you're curious about the distribution of results, we have the following for medium and ultra:

I don't love that the ultra results are somewhat saturated here, but one "problem" with testing ultra is that it will keep going for a long time as problems get harder (e.g., most of the Pandoc ultra runs ran for 12+ hours, and the assembly runs went for much longer), so the things that don't get saturated are very large tasks, like the Pandoc eval, or tasks that are too difficult in some way, like the Guards of Atlantis eval.