Section: Technology & AI Author: V Reading Length: ~27 min Sources and further reading: 22 items Topics: AI, benchmarks, ChatGPT, Gemini, Claude, Grok, Czech, model evaluation SEO / Working Title: ChatGPT vs. Gemini vs. Claude vs. Grok: Which AI is the best?
Which AI is best — ChatGPT, Gemini, Claude, or Grok? The question sounds simple until we notice that we are not comparing one thing. Preference on the Arena is not correctness. SWE-bench Verified is not your GitLab, and five attempts at max effort are not one attempt. MMLU is not general intelligence. An English leaderboard is not Czech use. Only when these four things are pulled apart can we read what the public tables aggregate as of September 2026 — and what they do not imply for Czech work.
1. Each leaderboard defends a different task
Imagine four readers of the same word “best”. A product manager opens LMArena and points to the first rows of the public leaderboard. They are right that anonymous pairwise human preferences capture which answer users like more. The Chatbot Arena paper reports more than 240,000 votes from April 2023 to January 2024 and about 90,000 users. It does not measure a factual error, Czech grammar or Czech case law [1].
A head of engineering pulls out SWE-bench. They are right that the share resolved after tests pass is harder than an impression from chat. But SWE-bench Verified is 500 human-verified instances from Python GitHub issues. The original SWE-bench rests on 12 open-source Python repositories. That is not your enterprise monorepo, your GitLab, SAP, internal library or the Czech Cybersecurity Act [6][7].
A teacher shows MMLU. They are right that 57 subjects and 15,908 questions make a named academic set. It does not measure open work with documents, Czech facts or whether a model admits uncertainty. And after Gema et al., it is no longer possible to pretend that every difference on raw MMLU is pure model ability: the authors of MMLU-Redux estimated 6.49 percent of questions with an error in the key [9][10].
A Czech user adds BenCzechMark. They are right that Czech is not merely translated English. BenCzechMark has 50 tasks, 14 of them newly collected, in 8 categories, mostly in native Czech and with about 10 percent machine-translated tasks. It uses a duel win score from statistical tests at an alpha of 5 percent, not a simple mean of percentages. Closed models are also often missing, because the authors deliberately do not send samples to paid APIs. “Not measured” is not “lost” [17][18].
At first glance this is a brand dispute. In fact each person is describing a different task. That is why they can all be partly right and the shared verdict “this AI is the best” still wrong. One table cannot, without reservation, decide chat, programming, school tests, Czech, safety and corporate accountability.
A leaderboard is a map of one task. It is not a verdict on whom you may trust in Czech.
— Jiný Kontext
2. One “best AI” has at least five layers
When people say the best model, they often mean one number. But a public evaluation is built in several layers. The first is the task. Are we asking about chat preference, a GitHub-issue fix, a four-choice test, Czech NLI, a school essay or a search in a legal text? These are not different examples of the same thing. They are different problems.
The second layer is the population of prompts or instances. The Arena collects self-selected prompts from real users and is mostly English. SWE-bench works with issues from 12 Python repositories. MMLU is an English exam bank. BenCzechMark adds historical Czech, pupil essays, spoken language and Czech tasks [1][6][9][17].
The third layer is scoring. A Bradley-Terry model from pairwise preferences is not the same as a percentage of tests resolved. Accuracy on four choices is not the same as a duel win score with statistical significance. Each score has its own question and its own blind spot.
The fourth layer is the harness and the publication policy. Agent scaffold, thinking budget, number of attempts, Docker, allowed tools and the choice of variant can change the result. The Leaderboard Illusion paper warns about private testing of variants and selective disclosure. That is not a small detail if the ranking is turned into a commercial argument [4].
The fifth layer is cost, latency, language and refusal. Token price, speed, Czech quality, data regime, safeguard and the chance that the system will refuse part of the task usually do not fit into one Elo. For users they often decide more than a one-row difference on a leaderboard.
This layering is not a detail. It says whether we are comparing the same work. If one model ran five times at max effort and another ran once in a different month, they do not merely share a benchmark brand. They do not share a protocol.
3. Preference on the Arena is not correctness
Chatbot Arena is useful precisely because it measures something static tests often cannot: human preference over real prompts. A user asks a question, receives two anonymous answers and picks the better one. A leaderboard is built from a large number of such duels. The ICML 2024 paper describes data from April 2023 to January 2024, more than 240,000 votes and about 90,000 users [1].
It does not follow that a higher rank means a truer answer. Preference blends tone, length, formatting, willingness to answer, politeness, certainty and sometimes correctness. A user may vote for an answer that is more readable even if it contains a smaller factual error. Or, conversely, for a brief answer, because it better fits their goal.
The Arena is not useless. It is a map of behaviour in conversation. It must be read as preference, not as an expert witness of truth. When we ask which system wrote a more pleasant email, the Arena is closer to the task than MMLU. When we ask which system correctly reads a Czech statutory paragraph, Elo alone is not enough.
Preference also often arises without knowledge of the correct answer. The user compares two answers at a moment when they themselves may have no primary source, test or legal footing. That is fine if we are measuring user choice. It is a problem if we infer truth from that choice. A model that answers with certainty, fluency and clean structure can win a vote even where a later check would find a faulty citation. A model that answers in a shorter sentence and admits uncertainty can look weaker, even if it is more useful for risky work.
As of the research date we did not use a live first place on the text Arena, because without a stable primary snapshot it would be a number without an anchor. That is not a weakness of the argument. It is its condition. A leaderboard changes and must have a date, a URL and a description of what it measured.
4. Length and markdown are not ability. They are a style factor
LMSYS does not deal with style by denying it. The blog by Li, Angelopoulos and Chiang of 28 to 29 August 2024 shows that answer length was the dominant style factor in a Bradley-Terry regression. The coefficient on the normalised length difference is given as 0.249. The authors controlled for length and markdown elements, for example lists, headings and bold emphasis [2].
That matters, because many users do not vote only on truth. They vote on the feeling that the answer is complete. A longer answer often looks more industrious. Markdown looks more ordered. Headings and bullets give a sense of structure. That can be a useful product trait, but it is not the same as factual correctness.
Style control peels off part of this influence. It does not remove it perfectly. The authors themselves note possible confounders, for example chain-of-thought or other properties of an answer that travel with length. Observational control is not a randomised experiment. It does not say that after subtracting style we have pure truth [2].
That is why it is a mistake to read a style-controlled leaderboard as a universal verdict. It is better than a raw impression. It is still a map of preferences with a correction, not a meter of factual error. If your work depends on whether a citation actually supports a sentence, you must measure citation support. Not merely rank on the Arena.
5. A private variant before publication is not a public record
The Leaderboard Illusion paper points to a practice that is uncomfortable precisely because leaderboards look public and simple. The authors report 27 private LLM variants from one provider, Meta, tested before the release of Llama 4. They also estimate an asymmetry of access to Arena data: 20.4 percent for OpenAI, 19.2 percent for Google and 29.7 percent for 83 open-weight models together [4].
LMSYS disputed these shares in its response. They therefore cannot be read as a closed audit of the whole Arena. They can be read as a warning that a public leaderboard is not only mathematics over a neutral world. Who may test privately, who withdraws a score, who publishes only the best variant and who has access to votes all shape what the public sees [4].
This is an audit of one practice, not proof that leaderboards have no value. On the contrary: if they have value, they need a protocol. They need to say which variant ran, when, under what rules and whether the result matches the production model. Without that, a user is not buying a winner. They are buying a name that may have meant something slightly different on the board.
6. SWE-bench Verified is not your GitLab
SWE-bench is a strong test because it asks a practical question: can a system resolve a real GitHub issue so that the patch passes the tests? The original benchmark contains 2,294 instances from 12 open-source Python repositories. The score is whether the proposed patch passes both FAIL_TO_PASS and PASS_TO_PASS tests in the given harness [6][8].
SWE-bench Verified exists because the original set had various impurities. In August 2024 OpenAI presented 500 human-verified instances from 1,699 randomly selected test samples. They were judged by 93 Python developers. The verified set has 196 easy instances with a human time estimate under 15 minutes and 45 hard instances with an estimate over 1 hour. These times are a human estimate, not an automatic metric [7].
Verified is cleaner. It is not universal. It still measures Python issues from particular repositories, in a particular harness, with particular tests. It does not measure code review, security, architecture, a TypeScript monorepo, historical SAP or Czech documentation. And SWE-bench Pro is again a different set: harder, more files, less leakage of ground truth. Verified is not Pro and Pro is not your GitLab.
That is why it makes sense to take SWE-bench as a filter of programming candidates. It does not make sense to take it as proof that a model will fix your issue at the same speed and with the same risk. Your harness, your tests and your review rules are part of the performance.
In a company repository what decides is often exactly what a public benchmark does not see. Code may pass local tests and still break an internal architectural rule. A patch may work and still be unreadable for the team that will maintain it. An agent may fix a symptom and go around the place where the bug is actually born. Or it may fail only because the firm did not give it the same context a person has: runtime variables, historical decisions, an internal library and knowledge of the release process. That is why a programming benchmark has to be complemented by your own pilot. Not because the public set is bad, but because your work has a different boundary of correctness.
7. Five attempts at max effort are not one attempt
The largest confusion arises when we place numbers with the same benchmark name and a different protocol side by side. In the Fable 5 and Mythos 5 system card of June 2026, Anthropic reports 95.5 percent resolved on SWE-bench Verified for Mythos 5 and 95 percent for Fable 5. These are self-reported numbers, an average of 5 attempts, adaptive thinking and max effort. Fable has production safeguards including a fallback to Opus 4.8, which the card gives as a reason it may sit slightly below Mythos [11][12].
On SWE-bench Pro the same card reports 80.3 percent for Mythos 5 and 80 percent for Fable 5. In the Gemini 3.1 Pro model card of 19 February 2026, Google DeepMind reports 80.6 percent on SWE-bench Verified for Gemini 3.1 Pro Thinking High, but as a single attempt. On the public split of SWE-bench Pro it reports 54.2 percent, also a single attempt [11][14].
In the table for GPT-5.6 of 9 July 2026, OpenAI reports 64.6 percent on SWE-bench Pro. That table does not report SWE-bench Verified. In a comparison table Fable 5 appears as 80 percent Pro, but that is still a number in an OpenAI table and a match of names, not proof of the same run with the same harness. Historically OpenAI reported GPT-4o at 33.2 percent resolved on SWE-bench Verified as of August 2024, in a note on Agentless [7][13].
What you need to test before you add two percentages
Write Verified or Pro. The number of attempts. Thinking. The date of the card. Self-report or independent run. Agent scaffold. Allowed tools. Only then compare. Without that, 95 next to 80.6 cannot be read as a brand contest. It is a contest of two rows with a different protocol.
A self-reported score shares a benchmark name. It does not share a protocol.
— Jiný Kontext
8. MMLU is not general intelligence
MMLU has a good reason to be famous. Hendrycks et al. built a large four-choice academic test in English: 57 subjects and 15,908 questions in total, of which 14,079 are test and 1,540 validation. The original paper uses five few-shot examples per subject [9].
Such a set helped measure breadth of knowledge. But breadth of academic questions is not general intelligence. MMLU does not measure Czech law, Czech facts, open-ended work with a document or the ability to admit that a source is missing. A model can be excellent on a four-choice question and weaker in a long Czech letter to a client. Or the reverse.
The human anchor is also often read more strongly than the paper allows. Hendrycks et al. report 34.5 percent for unspecialised MTurk and about 89.8 percent as an estimate of an expert 95th percentile from source exams. That expert figure is an estimate, partly from percentiles and partly an educated guess. It is not a measured expert panel on the whole of MMLU [9].
MMLU-Redux and MMLU-Pro are further objects. A contamination review from GEM 2026 recalls MMLU as an archetype of a set sensitive to leakage into training and to repeated use [20]. That does not mean MMLU has no value. It means it cannot, without reservation, be made into a verdict on work that MMLU never measured.
9. An error in the key is not a difference between models
In Are We Done with MMLU?, Gema et al. re-annotated 5,700 questions from all 57 subjects and created MMLU-Redux. They estimated that 6.49 percent of questions have an error in the key. In virology, in the analysed slice, they report 57 percent faulty items [10].
That is methodologically uncomfortable. If two frontier models differ by one percentage point on raw MMLU, part of the difference may be label noise, not ability. A model may answer in substance correctly and be punished by a wrong key. Or it may hit a key that is itself disputed.
It is not fair to turn this into a sentence that MMLU is worthless. It is fair to turn it into a rule: the smaller the difference, the greater the need to look at label quality, an interval of uncertainty, contamination and the nature of the task. A precise percentage can precisely measure a defective condition.
10. A cluster at the GPQA ceiling is not a brand verdict
GPQA Diamond is a different kind of academic test. Rein et al. built it as a graduate-level science, google-proof set; the diamond subset has 198 questions. Model cards from 2026 show high results: Anthropic reports Mythos 5 at 94.1 percent as an average of 5 attempts, Google reports Gemini 3.1 Pro at 94.3 percent without tools and OpenAI reports GPT-5.6 Sol at 94.6 percent [11][13][14][22].
The difference of 94.1, 94.3 and 94.6 percent looks like a ranking. But it is a cluster at the ceiling, across different cards and protocols. Without the same run in the same week, with the same settings and public logs, it cannot be made into a brand verdict.
The practical question is again different. If you are choosing a model for scientific Q&A, GPQA can be a useful map. If you are choosing a model for an internal Czech legal analysis, it maps a different terrain. A ceiling score on English science is not a licence to trust a Czech citation.
11. An English leaderboard is not Czech use
Czech is not only a vocabulary. It is syntax, inflection, context, cultural references, schools, offices, legal wording, an error in diacritics and the ability not to translate an English phrase as a Czech sentence. That is why Czech benchmarks matter, even if they are less visible than global arenas.
BenCzechMark from TACL 2025 reports 50 tasks, 14 newly collected and 8 categories. Most of it is native Czech; about 10 percent of tasks are machine-translated. The score is not a simple mean of accuracy, but a duel win score from statistical tests with alpha 5 percent. That is a different language of evaluation than MMLU or Arena Elo [17].
A reservation is still necessary. The authors of BenCzechMark deliberately do not send samples to paid APIs, so closed ChatGPT, Claude or Gemini are often missing on BCM. That cannot be read as a defeat. It is a gap in measurement [17][18].
CzechBench is another Czech trace, but here too it is unwise to pretend precision where the sources diverge. The dossier notes that the BenCzechMark paper mentions 17 tasks and 8 native ones for CzechBench, while the public leaderboard reports 15 tasks. That is why no single official CzechBench number stands here as a fixed figure. A named benchmark Czeval or CzEval was not found in the primary sources as of September 2026 [19].
Whoever reads only LMArena or MMLU does not see Czech duels. Whoever reads only a Czech benchmark without closed models also does not see the whole market. The right sentence is not “this model won Czech”. The right sentence often reads: in this Czech set this was measured, that closed model is missing and your own test is necessary.
Czech use also has several shapes. Writing a natural business email is one problem. Summarising meeting minutes with Czech names and institutions is another. Distinguishing whether a cited statutory paragraph is in force, has been amended or belongs to another part of the legal order is another. And correcting a pupil’s text so that the system does not rewrite the author’s style into smooth generic Czech is another. One score can bring some of this work closer. It cannot replace all of it.
12. Price, safeguard and refusal are not Elo
A public leaderboard usually does not carry the bill. Yet a user is not buying ability alone, but an accepted result. In June 2026 Anthropic lists Fable 5 at 10 USD per 1 million input tokens and 50 USD per 1 million output tokens. In July 2026 OpenAI lists GPT-5.6 Sol at 5 USD per 1 million input and 30 USD per 1 million output tokens [11][12][13].
These numbers are not a verdict by themselves. A cheaper token can be more expensive if the model needs more attempts, more checking and more human correction. A more expensive model can be cheaper if it reduces the number of failed runs. A price list is the price of a raw material, not the price of a result.
Safeguard is another layer. For Fable 5 Anthropic reports production safeguards and a fallback to Opus 4.8, which triggers on average in fewer than 5 percent of sessions [11]. That can be a safety advantage. It can also mean that a Czech “this is not working” on a particular task is not model inability, but product policy.
Refusal, latency and data regime do not fit easily into Elo. In work they still decide. The best model on a leaderboard need not be the best system for a process in which you need an audit, low latency, a stable answer, cheap review and predictable behaviour.
The same logic applies to harm. In a marketing draft, a refusal may be only a delay. In customer support, an unpredictable refusal can be a breach of service. In a legal or medical text, caution can be desirable if the system clearly says what it will not decide and why. A leaderboard that measures resolved issues or chat preference usually will not show this boundary. A pilot in the concrete process must show it.
13. A Grok card is not an SWE table
Grok appears in debates as another brand in the same contest. The dossier, however, states a simple limit: the public cards for Grok 4 of 20 August 2025 and Grok 4.20 of 7 April 2026 are safety and dual-use assessments. A comparable SWE-bench Verified number from xAI was not verified in these cards as of the research date [15][16].
The absence of a number is not proof of inability. It is proof that these cards cannot be assembled into a fair contest on SWE-bench Verified. If one vendor publishes a coding score in a table and another publishes mainly a safety card, we do not have two measurements of the same thing.
That is a more general rule. Not every model card is a leaderboard. Not every safety document contains product performance. And not every marketing table has a protocol that would allow comparison with another table. When the basis is missing, the precise answer is “not verified”, not “lost”.
14. The best map is the one whose error you can bear
The following recommendations are not a product ranking. They are a signpost to a first run of your own. They follow the text Who you allow to err, which treats the choice of AI by the kind of error you can afford. This is not a second brand verdict. It is about how to read a leaderboard as a map.
| Type of failure | How it looks | How to test | What limits the harm |
|---|---|---|---|
| Preference as correctness | Higher Elo = truer Czech | Ten tasks in the same language; factual error, not “I like it” | Take style control as a reservation, not as a test of truth |
| Hidden harness | 95% next to 80.6% as a brand contest | Write attempts, thinking, date, Verified vs Pro, self-report vs independent | Do not add cards with different protocols |
| English map onto Czech | LMArena or MMLU as a verdict for Czech work | BenCzechMark, CzechBench or your own set; “not measured” is not “lost” | Do not let MMLU or raw Elo be the first column |
| Cherry-pick of variants | Private runs, withdrawn scores, only the best number out | Ask which variant is in production | Leaderboard Illusion as a reservation on the table |
| Cluster at the ceiling | 94.1 / 94.3 / 94.6 as a ranking of brands | GPQA Diamond: a 1 p.p. difference at the ceiling | Do not decide a purchase on one percentage point |
Illustrative scheme: Three columns — Arena (preference), SWE-bench (tests in a harness), BenCzechMark (Czech duels). An arrow “best” does not run between columns. It runs only inside a column, with a date and a protocol.
A good map is not the one with the largest number. A good map is the one whose error you know. If it measures preference, use it for work where preference makes sense. If it measures a patch after tests, use it to pick programming candidates. If it measures Czech, read it in Czech and with a limit of coverage.
15. Ten tasks of your own say more than first place
A practical test is not a brand contest. It is a check of whether the map fits your task. Assemble ten tasks in the same language, the same genre and with the same threshold of harm. Ten Czech emails to a customer. Ten issues from your repository. Ten questions from an internal knowledge base. Ten short analyses where you know the right footing.
How to test two candidates on the same set
Run two candidates on the same day, with the same thinking setting, without switching the prompt between runs and with the same access to tools. Record factual error, a hallucinated quotation, refusal, length, cost and time to a usable result. Do not add Elo. Add which error would pass your review.
Three harness tests
If the task is Czech, do not let MMLU or raw Arena Elo be the first column. The first column is BenCzechMark, CzechBench or your set. If a closed model is not on BenCzechMark, write “not measured”, not “lost”.
If the task is code, state the harness. “95 percent Verified” without the number of attempts, thinking mode and agent stack is a marketing headline, not a laboratory note. If the task is research, check citations sentence by sentence. A citation that exists still need not support a particular conclusion.
Your own set need not be large. It must be true. It should measure the work you actually do, in the language in which you do it, and with an error you can imagine.
A good set also has the right negative examples. It is not enough to give a model ten tasks that resolve smoothly. Add an assignment in which the basis is missing. Add a document with a contradiction between an older and a newer version. Add a question whose correct answer is “the available data do not allow us to say”. That is when the difference appears between a model that wants to finish the sentence at any cost and a system that helps keep control.
Record the result simply. Not only a grade, but the kind of error. An invented source. A correct source, a wrong conclusion. An answer that is too long. A refusal without a reason. A good answer, but too expensive to repeat. Only this record says whether first place on a public map leads to your goal.
16. A self-reported score is not an independent run
A vendor’s self-evaluation is not automatically false. It is a different kind of evidence than an independent run. A model card can be honest and still incomparable with a competitor’s card. The difference may be in the number of attempts, the agent scaffold, the date, the allowed tools, the sample or the policy for choosing a variant.
A stronger conclusion would require an independent, pre-registered run of the same SWE-bench Verified harness for ChatGPT, Gemini, Claude and Grok in the same week, with the same scaffold, the same number of attempts, the same commit and public logs. That would still not decide Czech or law, but it would at least remove one source of confusion.
For Czech, a public Czech-language preference leaderboard with described sampling and without private cherry-picking, where closed and open models stood side by side, would help. For MMLU, a re-annotated successor without an estimated key-error rate of 6.49 percent, on which the ranking of frontier models stably differed by more than label noise, would help [10].
It would also help to have proof that a style-controlled Arena score predicts your company metric better than your own set: time to merge, number of complaints, number of hallucinated paragraphs or the share of answers returned by an editor. Until that exists, a public leaderboard remains a map of a task. It is not a verdict.
17. The hidden cost is a first place that measures a different task
First place is comfortable. It saves a decision, fits a slide and gives procurement a simple sentence. That is exactly where its hidden cost lies. When first place measures a different task, it brings certainty into the firm where there should have been a question.
A leaderboard is not an enemy. Without public benchmarks we would have only advertising and personal impression. The Arena helps us see preference. SWE-bench helps us see programming fixes in a given harness. MMLU and GPQA help us follow academic knowledge. BenCzechMark and CzechBench remind us that Czech must have its own measurement. Each map is useful when we know where it leads.
That is why it is not precise to say that we should not trust benchmarks. It is more precise to say that we should trust them in the range they themselves describe. If the Arena says that people in an anonymous pairwise comparison preferred one answer, we should not read from it the truth of a statutory paragraph. If SWE-bench says that a patch passed the tests, we should not read from it the safety of a design. If BenCzechMark says that a model held up in a Czech duel, we should not read from it the performance of a closed model that was not in the set.
The question is not which AI is the best. It is: which task, which prompt population and which harness does the leaderboard measure — and which error will you see in time on your own set?
Related texts in this series
- Who you allow to err — choosing AI by the kind of error you can bear. This text is a map of leaderboards, not a second brand verdict.
- Can AI replace programmers? — SWE-bench as a harness of work, not as chat Elo.
- Which occupations will AI replace… — a model leaderboard does not measure the end of an occupation.