The first clean measurement of the gap I said was hiding.
The summer continues in hyperdrive at the frontier. In the twenty days since my last post, "Everyone's Watching the Wrong Benchmark," the market saw many new models drop: OpenAI's GPT-5.6 line (Sol, Terra, Luna) went generally available the day of my post (July 9), Thinking Machines' Inkling (July 15), Moonshot's Kimi K3 (July 16), Google's Gemini 3.6 Flash, and Anthropic's Claude Opus 5 (July 24), all on the heels of Anthropic's Fable 5 and Sonnet 5 and Z.ai's GLM-5.2, which landed (to varying degrees of access) in the weeks just before it.
Both the open source and closed source frontiers moved meaningfully, so: a retrospective is due some ~480 hours later.
The most interesting development, to me, is that we now have a natural experiment. Kimi K3 is the new open-source frontier: 2.8 trillion parameters, the largest open model ever released. Claude Opus 5 is the new closed frontier. For the first time, both were measured on the same private, fixed-harness benchmark suites within eight days of each other. This means that for the first time, we can actually see the thing I claimed was invisible.
Three weeks ago I made a set of predictions. Here's the point-by-point accounting, with receipts. One prediction was validated in detail. One was wrong, due to a market shift in generally accepted industry practices and not a technological development. The way it was wrong is more interesting than if it had been right.
Which benchmarks matter for actual business use?
To compare model-to-model capability, I focus on results that are (1) fixed-harness and (2) closed-evaluation. Why so strict? Because this analysis aims to assess pure model capability and hold fixed other implementations like harnesses, tool access, skill access, etc. As models move to reasoning, planning, and agentic work, the implementations surrounding the model increasingly dictate overall capabilities1. But we’re trying to assess the models specifically, because it is those trillions of parameters that cost billions of dollars to train and that contain the intelligence.
Epoch AI's footnotes, which I quoted last time, recognize the other key component: open-weight models perform worse on private benchmarks than public ones, plausibly because they hill-climb harder on what's published.
So, in order to get to an apples-to-apples comparison between open source and closed source, we need to standardize the harness and use closed-evaluations. Two evaluators clear the bar. Vals AI runs all of its own evaluations on private, expert-built datasets across finance, law, tax, healthcare, and coding. It also runs one harness per benchmark, applied identically to every model. Artificial Analysis runs everything through Stirrup, its single open-source reference agent.
Is the domain expertise gap real?
Let's start with the benchmark on which everyone focuses. On the Vals Index, the top three models on earth are Claude Fable 5 at 75.1%, Claude Opus 5 at 74.8%, and Kimi K3 at 74.7%. There is essentially no difference, which explains all the statements about open source having "closed the gap."
But let's decompose the results by domain, when the evaluation uses the same private data and the same fixed harness:
On procedural work and coding, open source matched the frontier. On Vals' SWE-bench Verified, K3 resolves 93.4% of issues, a top-four result globally. On Vibe Code Bench (building working web apps from scratch) it ranks #3 of 76. On CorpFin v2 (extracting answers from 200-page credit agreements), K3 scores 71.6% against Fable 5's 71.8%. K3 offers equivalent performance to the leading model from the leading lab when it comes to coding well-defined targets, orchestrating tools, navigating repos, generating interfaces, and extracting from documents. These domains are closed-loop and verifiable, and therefore particularly trainable with reinforcement learning.
On domain expertise and judgment, open source remains behind. On Vals' Legal Research Bench, built with practicing law firms, Opus 5 leads at 55.3%; K3 sits at 44.2%. On MedCode, Vals' medical-coding benchmark, Opus 5 ranks #1 of 76 and K3 ranks #18. On MortgageTax, Opus 5 ranks #1 of 90; K3 ranks #33.
The single most diagnostic dataset comes from Artificial Analysis's AA-Omniscience, which measures factual accuracy and hallucination across professional domains, keeping the harness fixed across models. K3's domain accuracy: law 37.7%, health 37.5%, business 38.8%, and software engineering 70.1%. Fable 5's aggregate accuracy is 61%, with an Omniscience Index (which rewards accuracy and penalizes hallucination; higher is better) of 40 to K3's 18.4.
K3 is near-frontier at procedure, mid-pack at being reliably right about a domain you can't look up mid-task. My premise from the last post was that closed-source model labs pay billions of dollars to expert labelers and improve model training with that data. Those labelers lay out plans and judge the plans proposed by models, providing the RLHF signal that improves models and agentic workflows. The acute challenge of training long-horizon agents lies in mapping the reward function to the steps in the agentic workflow that deliver the greatest improvement in performance. RL environments provide scalable, lower-cost data generation, but they fail to supply the signal that expert labelers deliver: precise correction of the steps and judgment required to complete common knowledge work.
Step back and the shape is unmistakable. In my last post I argued the labs are spending at least $10–15 billion a year buying expert reasoning from doctors, litigators, analysts, and more, to advance their model capabilities. Twenty days later, the data shows the gap persisting in exactly those domains and closing in exactly the domains where training data is public or verifiable in reinforcement-learning environments.
What did I get wrong?
Last post, I wrote that the distillation door was closing. It seemed the closed-source labs were starting to enforce their terms, or at least bandy about doing so, and lobbying for regulation. That positioning flipped fast and decisively. On July 24, with Washington weighing restrictions on Chinese models, and the White House accusing Moonshot of distilling Anthropic's Fable model to build K3, twenty-five organizations across the AI stack signed an open letter, "Open Weights and American AI Leadership," urging policymakers not to treat distillation as misappropriation, and to address unlawful extraction through targeted legal frameworks rather than broad limits on the technique. The signatories include leaders across the AI landscape, with the exception of one notable frontier closed source lab. Perhaps they’d like to retain the value of all the data they’re paying to create.
It now seems distillation will remain common practice, defended by most of the industry as legitimate engineering rather than theft. That may mean open source can close the gaps referenced above quickly. Otherwise, labs seeking to halt distillation will increasingly need to leverage terms of service, legal enforcement, account validation, and model access controls to a much greater extent than they do today.
What is the significance of the harness?
Model harnesses increasingly dictate agentic capabilities, and the model and harness function best when closely tied together. Take Terminal-Bench 2.1, the standard agentic-terminal benchmark. Moonshot reports K3 at 88.3 on its own Kimi Code harness. Vals, running its fixed harness, measures 80.9. Same weights, same benchmark family, 7.4 points apart. The closed side surfaces the same dependency for top performance: Vals disclosed that Opus 5 ran with a server-side fallback for refusals, and counting fallback-assisted tasks as failures moves its Terminal-Bench score from 84.6% to 81.3%.
The frontier numbers you see quoted are no longer just a model. They are a model-harness pair, tuned together, by the same team. I'm closely watching how Z.ai and Moonshot position their harnesses, because both are shifting toward keeping more of the harness and the hosting optimization proprietary, which would enable them to monetize their leading models via hosted inference or licensing. Z.ai shipped ZCode on July 1 as the "official" development environment for GLM-5.2. ZCode is a proprietary desktop app that grants Coding Plan subscribers 1.5x usage quota. Moonshot's Kimi Code CLI, while MIT-licensed at the client, couples to closed server-side tools and a subscription funnel, with the model post-trained against that harness's tool loop. They get distribution and awareness via leading open models, but the best hosting and performance may only be available on their own infrastructure. That's a potential threat to the hosting providers that have seen massive growth as open-source coding-agent performance accelerated (see, e.g., the OpenRouter rankings).
What does this mean for startups and enterprise customers?
The enterprise argument from my last post holds. If the domain variance above persists, then the opportunity remains for enterprises, as well as the startups that partner with them, to develop and own the proprietary capabilities that differentiate their businesses from the model layer. More importantly, every enterprise, and particularly services industries like law, banking, consulting, etc., should not seek industry-average expertise, but instead seek to provide their own, proprietary insights and expertise developed from decades of work and data. That gap inherently falls outside any benchmark.
The past three weeks catalyzed support for the open source ecosystem that positions its adoption not just as reasonable and sustainable, but as a near-requirement for any serious enterprise AI platform. Our investment interests stand: US open-weight labs built for security and accountability; the tooling layer that lets enterprises encode their own workflows into weights they control; orchestration and routing that turn the open-closed spread into the enterprise's competitive advantage rather than a lab's lock-in; and agentic applications where the compounding value from agent improvement (the feedback, the corrections, the post-trained weights) stays with the customer.
If you're building here, we'd love to talk.
[1] The labs acknowledge that their practice of packaging models with increasingly integrated tech stacks in order to deliver frontier capabilities; according to the footnotes on Moonshot's own K3 model card, K3 was evaluated with the Kimi Code harness, the Claude models with Claude Code or Terminus 2, and the GPT models with Codex, and the benchmark reports the best score across harnesses. Every lab's model was tested inside its own best-case tooling.
Benchmark figures and leaderboard ranks are as of July 23–28, 2026; these move fast — check the live leaderboards.