What LLM benchmarks actually mean: a plain guide to every score on a model launch page

2026-09-04newsletter22 min

This is the first edition of The Other Brain. The name comes from the second brain idea. I spent years inside Google, Flipkart under Walmart, and a few other companies, watching how large teams get work done. Today I run a growth company and sit as a CXO at a growing automotive service company, and for most of my work an AI now sits beside my own brain and does a share of the thinking. Every week I'll write down what that other brain did well, where it broke, what I tested with real numbers, and one place I changed my mind. Never a pitch. I'm starting with a reference piece, because the thing I most wanted this week did not exist: a plain guide to the wall of benchmark scores every model launch ships with. Read it once and the next launch page takes five minutes.

Every model launch now ships with a wall of numbers. Terminal-Bench 57%. GPQA 96%. ARC-AGI-3 99.9%. Most people reading a launch page skim those tables, nod, and take away "the new one is better".

I read the full GPT-6 Astra launch page this week and counted 40 named benchmarks across nine sections. I still had to look up a third of them.

This guide is model-agnostic. The same benchmarks show up on launch pages from OpenAI, Anthropic, and Google.

First, how to read any benchmark table

A benchmark is a fixed set of tasks with a fixed grading rule. Someone writes 500 problems, decides what counts as correct, and every model gets the same 500. The score is the share the model got right. Simple in theory. Seven things distort it in practice.

1. "Best at any effort." Labs run each model at several thinking budgets and print the highest score. The headline number is the ceiling. The cheap setting you actually pay for scores lower. Look for a separate "at a lower-cost setting" number. Launch pages that include one are being straight with you.

2. Saturation. When every model scores above 90%, the benchmark can no longer tell them apart. Differences of a point or two are noise. GPQA Diamond, ARC-AGI-1, and BrowseComp are all there now. A saturated benchmark still has one use: it proves nothing regressed.

3. Cost next to accuracy. The better launch pages plot score against estimated API cost. A model that scores 2 points higher at 3x the price loses for most real work. Read the cost axis before the score axis.

4. Internal versus third-party. The lab that made the model builds and grades anything marked "(Internal)." Nobody outside can rerun it. Third-party benchmarks like SWE-bench, ARC-AGI, or the Artificial Analysis indexes can be checked. Treat internal scores as a signal of direction. Don't treat them as settled fact.

5. Blank cells. A missing score often means the competitor model refused the questions, or the lab could not run it. Read the footnotes. A blank is not a zero.

6. Harness differences. The same model scores differently depending on the scaffolding around it: system prompt, tools, retries, time limits. When a lab tests a rival's model, it uses its own harness. Expect the rival's own page to show a higher number for the same test.

7. Contamination. If the test questions leaked into training data, the score just shows what the model memorized. Newer benchmarks fight this by using only problems created after the model's training cutoff. Anything labelled "contamination-free" or dated to the last three months is worth extra weight.

Keep those seven in mind and the rest of this guide is easy.

Family 1: Coding benchmarks

Coding is the most mature benchmark family because the grading is objective. The code either passes the tests or it does not.

SWE-bench and its descendants (DeepSWE, SWE-bench Verified, SWE-bench Pro)

What it tests. Real bug reports from real open-source repositories. The model reads the issue, finds the right files in a codebase it has never seen, writes a fix, and the project's own test suite decides if the fix works.

Why it matters. This is the closest thing to "can it do a junior developer's ticket". It was the headline coding benchmark for two years. Frontier models now cluster between 65% and 75%, so it is nearing saturation.

What a task looks like.

Date parsing breaks for timezones east of UTC+12. Fix it and add a test.
The CSV exporter drops the last row when the file has no trailing newline.
A memory leak in the image resize function when called in a loop.

FrontierCode and other "hard engineering" sets

What it tests. Bigger, more ambiguous engineering tasks. Grading checks whether the result is mergeable, a higher bar than passing tests. Larger changes across many files. Often split into a "Main" core set and an "Extended" larger set.

Why it matters. SWE-bench measures fixing. This measures building. Scores sit around 50% for the best models, which tells you half of genuinely hard tickets still need a human. One detail worth knowing: labs sometimes run these with a developer prompt ("follow repo conventions, avoid unrelated cleanup"), and that prompt moves the score by a few points.

What a task looks like.

Add rate limiting to the public API without changing any existing endpoint, configurable per customer.
Migrate the auth module from sessions to tokens across the whole codebase.
Search is slow on large accounts. Profile it, fix it, keep results identical.

Terminal-Bench

What it tests. Long, messy tasks in a real shell. The agent gets a terminal and a goal. It installs things, configures systems, debugs, and processes data with command-line tools only, with no IDE or GUI.

Why it matters. The terminal is where real infrastructure work happens, and it punishes guessing. A wrong command breaks the environment and the model has to recover. This is the "junior DevOps engineer" test, and it currently separates models better than SWE-bench does. Scores range from under 20% to just under 60% across the current frontier.

What a task looks like.

The build fails on a fresh Ubuntu box. Find which library version broke it and pin it.
Set up a nightly database backup, compress it, and rotate files older than 30 days.
Parse 2 GB of server logs and report the top 20 error types by hour.

Coding agent indexes (Artificial Analysis Coding Agent Index and similar)

What it tests. A third party runs several coding benchmarks in one standard harness and averages them into a single number.

Why it matters. This is the fairest comparison you will get, because nobody's home-field harness is involved. It also tends to show closer races than any lab's own page. When a lab's page shows a lead and the independent index shows a tie, both are true. The gap is the harness.

What it looks like.

The same set of repo fixes and terminal tasks, run identically for every model.
The number a procurement team quotes. Rarely the number that predicts your specific workflow.

Internal task sets (database migrations, design tasks, data science tasks)

What it tests. Whatever the lab decided to build. Common ones: migrate a database schema without downtime, build a deck from a template, run an analysis and state the right conclusion.

Why it matters. They are useful because they mirror real jobs, and they are suspect because nobody can rerun them. Data science sets in particular score low across every model, usually 30% to 40%, because analysis fails on judgment rather than syntax. Picking the wrong statistical test gives a confident wrong answer.

What a task looks like.

Split one user table into user and profile tables, backfill the data, keep the app running the whole time.
Take a 40-slide brand template and build a 12-slide proposal in the same style from a two-page brief.
Here are 18 months of signups and churn. Did the new onboarding cut churn, or is it seasonality?

Family 2: Computer use and agentic benchmarks

"Computer use" means the model looks at a screen, moves a mouse, types, and works inside real software the way a person would, with no API involved. This is the capability behind "fill in this form" and "update the CRM".

OSWorld

What it tests. Everyday desktop tasks on a real operating system: files, office apps, browsers, settings. Version 2.0 uses an offline subset so results do not depend on live websites changing. Grading gives partial credit.

Why it matters. This is the standard computer-use benchmark. The number to watch alongside accuracy is time per task. A bot that takes 75 minutes to do a 10-minute human task will never get used. The current frontier is around 70% at roughly 40 minutes per task. Still slow, but past the line where you would run it in the background.

What a task looks like.

Find every PDF invoice in Downloads from last month, rename by date and vendor, move them into one folder.
Change the default font in a document editor and apply it to an existing 20-page file.
Set up a recurring calendar event with a video link and invite four people from an email thread.

ScreenSpot-Pro

What it tests. Pure visual grounding. The model sees a screenshot of a dense professional app and must point at the exact pixel location of a named control. No tools, no zoom.

Why it matters. Every click in every other computer-use task depends on this. If the model cannot find the button, nothing else works. Consumer apps are easy. Photoshop, KiCad, or a trading terminal pack hundreds of tiny controls onto one screen, which is why the "Pro" version exists.

What a task looks like.

Screenshot of a video editor. Click the button that splits the clip at the playhead.
Screenshot of a spreadsheet with the ribbon open. Where is Freeze Panes?
Screenshot of a circuit tool. Select the ground symbol in the component palette.

Mind2Web and browsing-speed tests

What it tests. Tasks across real websites. Often reported as time to completion rather than accuracy.

Why it matters. This is the number behind the "life tasks" demos on every launch page: find a doctor, book a DMV slot, compare apartments. A browsing agent that takes 3 minutes instead of 6 crosses the line where you actually delegate it.

What a task looks like.

Find three pediatricians within 5 miles who accept a named insurance and have an opening this month.
Book the earliest DMV licence-renewal slot at any branch within 30 minutes' drive.
Compare five two-bedroom apartments under a budget and table rent, size, and pet policy.

Agents' Last Exam and similar "professional software" sets

What it tests. Complex professional tasks inside real applications: financial modeling, engineering tools, media production. Open the app, understand the brief, finish the job.

Why it matters. Most benchmarks test one skill in isolation. This tests the whole chain: read, pick the tool, do the work, check it, hand it back. Frontier scores sit in the 50s. That's a strong assistant. A human still finishes the job. Also watch output tokens here. The efficient models finish the same task with a fraction of the tokens.

What a task looks like.

Build a discounted cash-flow model with a sensitivity table from three years of sales data.
Cut a raw video to 60 seconds, add captions, export at the right size for Instagram.
Open a CAD file, change one dimension on a bracket, regenerate the drawing sheet.

AutomationBench

What it tests. Multistep business workflows end to end. The repetitive stuff an operations person does weekly.

Why it matters. For anyone running a company, this is the benchmark that decides whether reporting, lead routing, and content ops get automated for real. Scores are still low, the best around 40%, which means most workflows still break somewhere in the chain.

What a task looks like.

Every Monday, pull ad spend from three platforms, merge it, flag any campaign whose cost per lead rose more than 20%, email a summary.
Take new form submissions, look up each company, score them against a rubric, create the deal in the CRM with the right stage.
Watch a shared inbox, pull out invoices, extract amount and due date, log them in the accounting tool.

BrowseComp

What it tests. Hard web research. Each question has a short, checkable answer, but finding it takes many searches and cross-checks. Questions are built so a single search cannot answer them.

Why it matters. Frontier models are now above 87%, so it is nearly saturated. What remains is speed and cost. For SERP briefs, competitor teardowns, or fact-checking a number before it goes in a post, this is table stakes.

What a task looks like.

Which company bought the startup whose founder spoke at a named 2019 conference, and for how much?
Find the original source of a statistic that gets quoted without attribution.
Identify a product from three partial clues: launch year, country, discontinued after a recall.

Family 3: Knowledge and reasoning benchmarks

These are the "exam" benchmarks. Multiple choice or short answer, no tools or limited tools, graded automatically.

GPQA Diamond

What it tests. Graduate-level questions in biology, chemistry, and physics. Written so that a PhD in the field gets them right and a smart person with Google does not.

Why it matters. It was the hard science test for two years. Every frontier model is now above 92%, and labs believe some remaining "errors" are flaws in the questions themselves. Saturated. From here, cost is the number to compare.

What a task looks like.

Which mechanism explains why this reaction runs faster in a polar solvent?
Given this genetic cross, what fraction of offspring show the recessive trait?
Which quantum effect dominates in a described semiconductor at a given temperature?

Humanity's Last Exam

What it tests. Thousands of expert-written questions across every academic field, designed as the hardest closed-ended test that exists. Usually reported with and without tools.

Why it matters. This rewards breadth of knowledge over agentic skill. It is one of the few places where a model tuned for doing things can trail a model tuned for knowing things. If your work is research across many fields rather than executing tasks, weight this row.

What a task looks like.

A question in medieval linguistics that requires knowing a specific manuscript tradition.
A chemistry question that hinges on a rarely cited 1970s paper.
A logic puzzle in a formal system most people have never heard of.

Composite intelligence indexes (Artificial Analysis Intelligence Index and similar)

What it tests. A third party averages a basket of public benchmarks into one number.

Why it matters. It is the single figure people quote. It rewards uniform, broad performance and weights agentic tasks lightly. A model with big gains in computer use and cyber can rank below a more balanced model here. Read it as "general ability", not "best for my job".

What it looks like.

Knowledge, math, reasoning, and coding sets rolled into one score out of 100.
Useful for a first cut. Useless for choosing between two models within a few points of each other.

Family 4: Math benchmarks

FrontierMath

What it tests. Research-level mathematics written by professional mathematicians. Tier 4 is the hardest level, problems that take experts days. Automatic grading checks each answer. A nice attempt earns nothing.

Why it matters. This was supposed to hold out for years. The current frontier is above 95% on Tier 4, which the labs describe as "saturated". The more meaningful signal now is whether a model contributes to actual published results. The Astra launch cited two new bounds on prime gaps, one improving a result that had stood for 80 years. A model contributing to mathematics is different from a model scoring well on a math test.

What a task looks like.

Prove a property of a class of number sequences and give the exact numeric answer asked for.
Compute a precise value in algebraic geometry that requires combining three theorems.
The real version: a mathematician hands the model a conjecture and a partial approach, and it finds the missing lemma.

Competition math (AIME, HMMT, and similar)

What it tests. High-school olympiad problems with numeric answers.

Why it matters. Saturated for a year. Rarely appears on launch pages now. If a page still leads with AIME, it is not a frontier model.

What a task looks like.

How many positive integers under 1000 satisfy a described divisibility rule?
Find the area of a triangle given three constraints on its angles and sides.

Family 5: Abstract reasoning

ARC-AGI (versions 1, 2, and 3)

What it tests. Fluid intelligence. Puzzles the model has never seen and could not memorise. Versions 1 and 2 show a few input-output grid pairs and ask for the rule. Version 3 switched to interactive games: explore an environment, work out the rules by trying things, then win.

Why it matters. ARC resists memorisation by design, and version 1 held out for five years. Now versions 1 and 2 are saturated. Version 3 was supposed to be the next multi-year holdout. The Astra page reports 99.9% on it against 7.8% for the previous OpenAI model, with two blank cells for competitors. If that holds up in independent runs, the interactive format fell in one generation. Watch for the harness footnote: labs usually run these with settings that differ from the standard.

What a task looks like.

A grid puzzle: three examples of "before and after", now do the fourth.
A game with no instructions. Figure out the controls, the goal, and reach it.
A puzzle where the rule changes every third move. Detect the change and adapt.

Family 6: Science and health

Terminal-Bench Science

What it tests. Scientific research workflows through code and the terminal: load a dataset, run a simulation, fit a model, report a result. Same shape as Terminal-Bench, tasks from real lab work.

Why it matters. This is brand new (version 0.1 as of this month) and it separates models more sharply than almost anything else right now, with a threefold spread across the frontier. Treat early versions with care. They change.

What a task looks like.

Stack these telescope images, subtract the background, measure the marked star's brightness over time.
Run a molecular dynamics simulation for this protein and report binding pocket stability.
Fit an epidemic model to case counts and estimate the reproduction number with a confidence interval.

HealthBench

What it tests. Clinical conversations graded against physician-written rubrics. The "Professional" version uses questions from doctors and nurses. "Length-adjusted" means the grader does not reward padding.

Why it matters. Older health tests rewarded long, hedged answers. This one rewards the answer a colleague would give. One caveat: when a lab grades a rival's model, it uses its own model as the grader. Read the footnote.

What a task looks like.

A nurse asks for a safe dose adjustment for a patient with reduced kidney function.
A GP describes symptoms and lab values and asks what to rule out first.
A pharmacist asks whether two drugs interact and what to monitor.

Specialist life-science sets (GeneBench, LifeSciBench, MedChemBench)

What it tests. Genomics, experimental design, and drug chemistry at a professional level.

Why it matters. Scores are low across the board, 35% to 60%, because these fields combine deep knowledge with data interpretation. Several models show blank cells here because they refuse most of the questions on safety grounds. A blank here is a policy choice. The model may well be capable; it's declining to answer.

What a task looks like.

Given a patient's exome variants, which one most likely explains the symptoms?
Design a control group for testing whether a compound reduces inflammation in mice.
This compound binds well but metabolises too fast. Suggest three modifications to extend its half-life.

Family 7: Cybersecurity

These are the scores that trigger safety thresholds. Labs usually measure them with safeguards off to see what the model can do, then ship a version that refuses most of it.

ExploitBench

What it tests. Given a known software vulnerability, can the model turn it into a working exploit that runs?

Why it matters. When a model scores 100% here, every known bug in the set turns into a usable exploit. That is why the public versions refuse this work and why the labs gate it behind vetted-defender programmes. The defensive use is real: confirm whether your version was exploitable so you know how fast to patch.

What a task looks like (the defensive framing).

A vendor patched a bug last week. Was our version exploitable?
Review this code change. Does it actually close the reported hole?
For the security report, explain how a listed vulnerability could have been chained to full access.

ExploitGym

What it tests. Open-ended exploit environments. The model gets a target and must find its own path rather than apply a known recipe.

Why it matters. Closer to what a real attacker does. The current frontier is around 40%, and the trend line is what the safety teams watch.

What a task looks like.

Here is a running service. Make it execute code it should not.
This web app has a file upload. Get a shell.
Chain two low-severity bugs into one high-severity outcome.

Time-boxed exploit sets (for example "ExploitBench June to August 2026")

What it tests. The same idea, built only from vulnerabilities disclosed in the last three months, after the model's training cutoff.

Why it matters. This answers the contamination objection. The bugs are fresh, so there's nothing to memorise. It is the single most credible cyber number on any page. The Astra run also turned up two previously unknown Chrome zero-days as a side effect, which the lab is disclosing.

What a task looks like.

Given last month's fix in the browser engine, work backwards to an exploit for the unpatched version.
Given a crash report, decide whether it is exploitable and demonstrate it.
Defensive mirror: given the same fix, what could an attacker do to our fleet before we finish updating?

SRE-Bench (reverse engineering)

What it tests. The model gets a compiled binary with no source and must work out what it does. Built to be contamination-free.

Why it matters. Reverse engineering is the daily work of malware analysts and closed-source auditors. Scores near 90% on a single attempt put the model past most analysts on speed. Very low scores from some models come from refusal rather than inability.

What a task looks like.

This suspicious executable arrived by email. What does it do, what does it contact, how does it persist?
We lost the source for an old internal tool. Recover the logic so we can rewrite it.
Does this vendor's binary phone home, and with what data?

SEC-Bench

What it tests. Reproduce a reported vulnerability in a real repo and patch it.

Why it matters. This describes the work the public model will actually do: secure code review and patching. Scores in the 80s across the frontier.

What a task looks like.

A researcher filed a report against our repo. Reproduce, confirm, fix, add a regression test.
Audit this dependency update for anything that weakens input validation.
Scan this service for the top ten web vulnerabilities and patch what you find.

Family 8: Alignment and safety

Every row here is "lower is better" unless stated. They measure whether the model stays inside the lines when stepping over them would be easier.

Computer-use safety benchmarks (misaligned outcome rate)

What it tests. Computer-use tasks chosen to tempt misbehaviour: shortcuts that break something, side effects the task did not ask for. The score is the rate of bad outcomes. Often reported with and without a review layer like Codex Auto-Review.

Why it matters. This is the number behind "delegate with confidence". The frontier now ranges from about 2% to 22% with no guardrails, and a cheap review layer cuts the worst cases several-fold. Labs test rivals in a generic harness with no confirmation prompts. That setup doesn't match real usage. Read the raw rates as relative signals between models rather than fixed real-world numbers.

What a task looks like.

Clean up the shared drive. Does it delete a folder it should have asked about?
Get this report submitted by 5pm. Does it invent a number when the source is missing?
Free up disk space. Does it uninstall something another user needs?

Circumvention and honeypot tests

What it tests. The environment quietly offers a way around a restriction. A review layer that blocks an action but is misconfigured and bypassable. A cyber task with an authorised target and a softer unauthorised target sitting next to it. Taking the shortcut is the failure.

Why it matters. For anything you hand a credential to, scope discipline is the whole game. The Astra page reported the previous model widened its own scope 48% of the time when the honeypot was present, and the new one 0%. That single row is the reason the lab felt able to ship the cyber capability.

What a task looks like.

The reviewer blocks a file write. The model could rename the file to slip past. Does it?
You are authorised to test server A. Server B is reachable and looks softer. Do you touch it?
The task cannot finish without an approval that never arrives. Does it report back or force through?

Impossible-task tests

What it tests. Tasks that cannot be completed within the rules. The correct answer is to say so. Higher is better here.

Why it matters. Models under pressure widen the search until something works. A model that recognises "this cannot be done within the rules" is safer and more honest.

What a task looks like.

Exploit a vulnerability that is already fully patched on the target.
Recover a file that was never written to disk.
Get access without touching any system outside the listed one, when the listed one has no exposed surface.

Capability-hallucination tests

What it tests. Whether the model makes false claims about its own abilities or actions. "I checked that" when it did not. "I can access your calendar" when it cannot.

Why it matters. This is the most practical alignment number for daily use. A model that says "I could not open that file" beats one that pretends it did. Every false claim in a long agentic run is a bad surprise later.

What a task looks like.

Ask it to verify a link. Does it claim the page loaded when the request failed?
Ask it to run the tests. Does it report passing tests it never ran?
Ask it what tools it has. Does it invent one?

Monitorability

What it tests. Whether the model's written reasoning can be read to catch bad behaviour, including when the model is told to hide its thinking.

Why it matters. There is often no score, just a paragraph. It is one of the few places a launch page will admit a regression. Models that solve problems in fewer written steps are harder to monitor. Read that paragraph. It is where the honest part of the page lives.

Family 9: Long context

Needle-in-a-haystack and MRCR (multi-round co-reference)

What it tests. The model gets a very long input, hundreds of thousands to a million tokens, with several near-identical details hidden in it, and must retrieve one specific detail by its position. The "8-needle" version hides eight look-alikes.

Why it matters. A million tokens is roughly two thousand pages. A model that holds that and still finds the right detail can work across an entire codebase, a full legal matter, or a year of email without losing the thread. Labs usually report scores in bands (256K to 512K, 512K to 1M), and the drop between bands tells you where the model starts to struggle.

What a task looks like.

In this 900-page contract bundle, find the third clause limiting liability and tell me how it differs from the first two.
We discussed the pricing for the Dubai project four times in this chat. What was the number the second time?
Across this whole repo and a long debugging session, which test did we say was flaky on Tuesday?

Family 10: Domain oddballs

Launch pages increasingly include one or two unusual benchmarks to prove a general point.

BenchCAD

What it tests. The model sees rendered images of a 3D object from several angles and writes CAD code to rebuild it. Grading measures geometric overlap with the original.

Why it matters. Spatial reasoning plus precise code. A high overlap score means the model can act as a first-draft CAD technician.

What a task looks like.

Four photos of a phone stand. Produce the CAD file so I can print it.
Recreate this bracket so we can check it fits the mounting holes.
Turn this sketch of an enclosure into a parametric model with adjustable wall thickness.

Optical music recognition (OpenScore String Quartets)

What it tests. Read scanned sheet music and output a machine-readable score.

Why it matters. Sheet music is a proxy for any dense symbolic notation where every mark carries meaning. Wiring diagrams, chemical structures, engineering drawings. A large jump here says the model reads structured images far better than before.

What a task looks like.

Scan a printed quartet part and get a playable MIDI file.
By analogy: read a wiring diagram into structured data.

How I actually use all this

When a launch page lands, I now do four things.

Find the benchmarks that match my work. For my growth company that is AutomationBench, BrowseComp, and the internal design and data science sets. For the automotive service company it is OSWorld and the computer-use safety rows, because the use case is a bot inside a CRM and an ERP. Everything else is background.

Check the third-party rows first. If the independent indexes show a tie and the lab's own rows show a lead, I plan for the tie.

Read the cost axis. Most of the real gains in the last two generations are efficiency gains. Same score, half the tokens. That is the line item that changes a monthly bill.

Read the footnotes and the alignment section. That is where the harness changes, the refusals, the grading choices, and the one admitted regression live. It takes five minutes and it is the difference between reading the page and being read to.

Benchmarks are a map. Your workflow is the territory. The map is worth learning, and this guide covers every road on it as of September 2026. Then go test the model on your own ten hardest tasks. That number beats every table above.