Section: Technology & AI Author: V Reading Length: ~27 min Sources and further reading: 22 items Topics: AI, programming, Copilot, Cursor, productivity, tests, operations, responsibility SEO / Working Title: Can AI replace programmers?
Can AI replace programmers? The question sounds simple only until we notice that we are not comparing one job. Shorter time on an HTTP server in the lab is not a week on a team. A resolved GitHub issue is not on-call. A model's 50% time horizon is not half an FTE. And a benchmark score does not sign an incident. This article therefore does not look for a date when the programmer disappears. It looks for which task the assistant speeds up, where the work moves under review, and who remains with the outage.
1. Twelve green tests on an HTTP server are not a nightly revert in production
Friday's demo looks convincing. A junior shows a small HTTP server in JavaScript. The tests light up green. He explains that with Copilot he wrote the solution much faster than he expected. The tech lead sits next to him and says that in a large internal repository the agent often adds work, because he has to check detours he would never have written himself. The product manager shows a chart of closed tickets. The director asks whether next quarter he should hire fewer people. The on-call engineer asks a shorter question: who will revert it at night, when the green tests are not enough.
At first glance this is a dispute over whether AI will or will not replace programmers. In fact each of them is measuring a different object. The junior, a laboratory task with a clear brief. The tech lead, a mature system he knows. The product manager, the number of completed units of work. The director would like to convert the task into a hiring plan. The on-call person measures responsibility for operations.
Peng, Kalliamvakou, Cihon and Demirer, in a study with GitHub Copilot, measured a programming task from 15 May to 20 June 2022: implementing an HTTP server in JavaScript that had to pass twelve tests. There were 95 freelancers from Upwork in the study, and 35 people in each arm completed the task. The authors report a time reduction of 55.8% with a 95% confidence interval of 21 to 89 [1]. That is a strong number. It does not, however, measure a production codebase, review, an incident, ownership of a service, or the nightly decision whether a rollback will hurt customers more than the bug.
A faster HTTP server is not the same as responsibility for a production system.
— Jiný Kontext
2. Programming work has at least six layers
When a diff appears on the screen, it creates the impression that “AI wrote the code”. But the result is produced in several layers. Each can be measured differently, and each can support a different conclusion.
- Code generation: autocomplete, boilerplate, tests, scripts and small transformations.
- Changing a known system: work in a repository whose shortcuts, debts and unwritten rules a person knows.
- Agent benchmark: issue, patch, running tests and a score on a public set.
- Task length: the horizon on which the model completes longer and longer work at a given success rate.
- Measurement in a team: tickets, lead time, review, defects, interruptions of work and a change of process.
- Responsibility: merge, incident, security impact, audit, customer and rollback.
These layers are not fine nuances. They are different objects of measurement. Peng measures time to passing tests on a clearly bounded task [1]. Cui and co-authors measure completed tasks in organisations in field experiments [2]. METR in 2025 measures experienced open-source developers on real issues in repositories they know well [4][5]. SWE-bench measures the repair of GitHub issues in a harness [9]. Time horizon measures how long the software tasks are that models complete at a given success rate [6].
The labour-market outline asks what will happen to the occupation. Here the question is narrower. Which layer of software is being measured just now? If we do not name it, we end up with a sentence that turns one green test into personnel policy and one slowdown into a ban on the tool.
3. The best assistant is the one whose error is caught by a test, a review and a rollback
With code it is tempting to measure writing speed. But software is not a paragraph of text. An error can be obvious, cheap and reversible. Or it can pass the tests smoothly and appear only in the customer’s data. So a good assistant is not merely the one that drafts the most lines. It is a tool whose failure is visible before it becomes an operational problem.
| Type of failure | What it looks like | How to test | What limits the harm |
|---|---|---|---|
| Code confabulation | The diff looks finished, but it changes authorisation or an edge case | Review, CI, an ownership test and rollback | A person merges; the agent has limited permissions |
| Bad measurement design | A verdict on an occupation from one RCT | Separate the task, the ticket, a known repo and operations | Do not carry the number over onto a headcount |
| Shift into review | The commit is faster, the merge is slower | Measure the brief, the check and an acceptable state | Evaluate the whole chain |
| Benchmark as testimony | A SWE-bench score as a substitute for a team’s work | List what the set does not measure | Internal tasks before a public headline |
| Incident without an owner | “AI wrote it” instead of a responsible merge | Who reverts, who reports, who explains | A runbook on a person and an organisation |
That also changes the buying question. We are not asking only how much time the assistant saves while writing. We ask whether we have tests, review, observability, permissions and rollback that can bear its error. Where the error is cheap and the check is clear, the gain can be large. Where the error escapes into production, faster generation may only move the problem to a more expensive place.
4. Three designs, three directions. Not three verdicts on an occupation
Public debate likes a single chart. Research on programmers’ productivity with AI offers three different designs instead.
Peng and co-authors measured a laboratory task with a clear outcome: an HTTP server in JavaScript, twelve tests, freelancers from Upwork, and time to completion [1]. Cui, Demirer, Jaffe, Musolff, Peng and Salz, in a February 2025 version, work with 4 867 developers at Microsoft, Accenture and one Fortune 100 firm. Their instrumental estimate of Copilot use on the number of completed tasks is +26.08% per week with a standard error of 10.3 [2][3]. METR and Becker et al. in 2025 followed 16 experienced developers, 246 real issues, an average human time of 2.0 hours, and repositories with about 23 thousand stars. With Cursor Pro and Claude 3.5/3.7 they measured a 19% increase in time [4][5].
These are not three verdicts. They are three windows. In one there is a clear skeleton and tests. In the second, ordinary company tasks and adoption of a tool. In the third, people who know their code and must check proposals against context they hold in their heads. None of these measurements by itself says whether an employment contract will disappear. And none of them can honestly be used without a description of the population, the period, the metric, and what was not counted.
5. A speed-up on a skeleton is not a junior headcount
The result of Peng’s study matters precisely because it is narrow. It shows that on a well-bounded task with a testable output, an assistant can cut time substantially. Especially where there is a lot of boilerplate and where tests quickly say whether the solution works. That is exactly the kind of work on which many less experienced people learn.
From that it does not follow that a junior can be subtracted from the budget. The study had a specific task, a specific period, and a completed sample of 35 people in each arm [1]. It did not measure onboarding, knowledge of a company domain, communication with product, debugging of requirements, review of other people’s code, or holding a service in production. It did not measure either whether in half a year the team will pay for missing training of new people.
The right managerial sentence is therefore not: “AI will do the junior work.” It is: “On tasks like this, writing can be shortened substantially if we have clear tests and a cheap check.” That is less flashy. It is, however, more usable.
For a team that implies a different test. Take a small task of your own that resembles a skeleton: a known brief, a clear result, automated tests. Have it done with the assistant and without it. Measure not only writing time, but also the time when a senior reads the diff. If the saving is lost in review, work was not removed. It was moved.
That shift can still be good news. A junior who produces a first draft faster with an assistant may hit questions of architecture, testing and responsibility earlier. That holds only if the team does not let the model replace learning. If a person does not know why the code passed a test, they cannot maintain it. If they cannot explain their own diff, they are not ready to carry a change into a shared system. The assistant’s benefit must therefore be measured together with learning: did it speed the path to understanding, or did it only produce more text that someone else must untangle?
6. More completed tickets is not more quality in operations
Cui and co-authors move the debate from the laboratory into firms. That is a large change. Four thousand eight hundred and sixty-seven developers, three organisations and a measure of completed tasks per week are closer to what managers care about [2]. The estimate of +26.08% completed tasks per week is lower than the laboratory effect on Peng’s task, which the authors themselves relate to the difference between a clean task and ordinary work [2].
But even a ticket is only a unit of process. A completed task is not automatically a quality change in production. It depends on what the organisation counts as done, how strong the review is, and whether defects, incidents, customer impact and other people’s time are measured. A team can close more small tasks with an assistant and at the same time load review. Or it can genuinely raise throughput without a drop in quality. Data on completed tasks are a start, not a final account.
This is often where the replacement debate breaks. If a firm tracks only the number of tickets, it will be tempted to convert a percentage into people. If it also tracks defects, review and operational changes, it may find something else: the assistant increases the volume of work, but it requires a different kind of check and a different split of responsibility.
In field measurements it also matters who actually uses the tool, and on which tasks. Adoption is not usually spread at random like paint on a wall. Someone reaches for the assistant on simple fixes, someone on library research, someone on tests, and someone almost never. If completed tasks are then tracked, the result can mix the tool’s ability, people’s willingness to use it, the type of tickets assigned, and team rules. Cui and co-authors address this in the research design; the reader should take from it mainly caution when translating it into their own firm [2].
A company pilot therefore must not merely switch the tool on and, after a month, count closed items. It must distinguish types of task, people’s experience, the size of the diff, review time and the number of returned changes. Otherwise a good report on throughput can hide a new bottleneck among those who check the changes.
It is still more precise to track who pays for the speed-up. If the assistant helps the author of the task, but the reviewer spends more time verifying, the saving has moved between people. If the number of tickets rises, but the number of returns from QA also rises, the saving has moved into another queue. If the whole chain from brief to a safely accepted change is faster, then the team has evidence that is worth more than a general headline about AI.
7. A slowdown among people who know the repo is not proof that the assistant does not work
METR in 2025 brought an uncomfortable result. Experienced open-source developers worked on issues in repositories they knew well. Average experience with the given repository was several years, the tasks had an average human time of 2.0 hours, and the study ran from February to June 2025. When they were allowed to use AI tools, time lengthened on average by 19% [4][5].
Even more interesting is the expectation. Developers beforehand expected a speed-up of 24%. After the study they still estimated that they had been 20% faster with AI, even though the measured result went the other way [5]. That is not proof that the developers do not know what they are doing. It is a warning that a subjective feeling of fluent work can mislead even experienced people.
From that it does not follow that the assistant does not work. It follows that in a known system a person holds context in their head that the model does not have. They know which seemingly simple solution will break an old integration. They know why an odd helper is odd on purpose. They know that a test fails only in combination with data that are not in the repository. AI can add suggestions, but also dead ends that have to be checked.
Three tests of productivity
Are you measuring time to green tests on a new skeleton? Are you measuring the number of completed tasks in a team? Or are you measuring time on an issue in your own mature repository? Three yes answers do not add up. Each belongs to a different decision.
8. A 50% time horizon is not 50% replacement of an FTE
METR, in the study by Kwa et al., measures the ability of models to complete ever longer software tasks. It works with a set of 170 tasks, including HCAST, RE-Bench and 66 SWAA, and with twelve frontier models. For the o3 model it reports about 110 minutes of human time as the 50% time horizon [6]. The study also estimates that this horizon has doubled since 2019 roughly every seven months, more precisely 207 days with a 95% interval of 166 to 240 days [6][7].
That matters for the direction of development. It is not, however, a sentence that the model replaces half a programmer. A 50% horizon means that the model has, on the given set, roughly a half success rate on a task corresponding to a certain human time. METR also reports that the 80% horizon is about four to six times shorter [6][7]. Production usually does not want a half chance. It wants reliability, repeatability and responsibility.
A METR note of 20 March 2026 further warns that time-horizon results are sensitive to modelling assumptions, regularisation and estimates of human times [8]. That does not cancel the trend. It only blocks a translation into a simple sentence about months of work in any firm.
A 50% time horizon says how long a task the model sometimes completes. It does not say who will revert the outage at night.
— Jiný Kontext
9. SWE-bench repairs an issue with tests. It does not design architecture and does not carry a pager
SWE-bench is a useful benchmark because it tries to start from real GitHub issues. The original paper by Jimenez and co-authors worked with 2 294 issues from twelve Python repositories. In the paper, Claude 2 with BM25 retrieval reached 1.96% resolved issues; GPT-4 was, because of budget, scored on a 25% subset [9]. Later SWE-bench Verified from OpenAI introduced a human-verified sample of 500 items and reports GPT-4o with a score of 33.2% [10]. Epoch AI describes the verified set also as a variant with 500, or 484 usable samples depending on the setting [11].
That is progress in measuring agents. It is not a programmer’s job. The benchmark does not contain a meeting with product, an architecture decision, reading internal incidents, a migration without an outage, a security review or on-call. It contains an issue, a repository, tests and a scoring method.
The right question is therefore not whether SWE-bench shows the future of programming. It shows one important layer: the ability of models to find and fix some bugs in open repositories under a test. For choosing a tool that is a signal. For a plan of responsibility in production it is too little.
10. Faster boilerplate among the less experienced is not the end of the senior
Results around coding assistants often show a larger benefit for less experienced people, or for people in a less familiar part of the work. That makes sense. Where a person is looking for syntax, boilerplate, a common pattern or a first draft of a test, a model can quickly supply material to edit. Where a senior holds the history of the system in their head, suggestions can add noise.
From that it does not follow that the senior disappears. The opposite comfortable sentence, that the senior is beyond the reach of change, does not hold either. Rather the mix of work changes. A less experienced person can reach a first solution faster, but they must learn to read a diff, understand tests and recognise when the model has only confidently copied a bad pattern. A senior may write less routine code, but more often set boundaries, check security and decide whether to accept a change.
A team that turns this only into a saving on people risks two things. It loses the apprentice work on which future seniors grow. And it shifts checking onto a smaller number of people, who then carry a larger review queue. A team that bans the assistant because of one slowdown may, in turn, leave cheap gains on well-testable tasks unused.
11. Using the tool is not trust in the output
The Stack Overflow Developer Survey 2025 shows high adoption of AI tools. According to the overview, 84% of respondents use or plan to use AI tools, compared with 76% in 2024. Among professionals it is reported that 51% use AI daily. At the same time 46% of respondents rather distrust the accuracy of the output than trust it, while 33% express trust [12][13].
This is not a representative sample of all programmers in the economy. It is a self-selection of Stack Overflow respondents. Still the signal is useful: use and trust are not the same. A person can use the tool every day because it saves time in searching, drafting a test or explaining an error. At the same time they need not believe that an output without a check belongs in production.
That difference is healthy for software. A programmer need not be an opponent of AI in order to refuse autonomous deployment. They need not be an enthusiast in order to use code completion. Trust is not a sticker on a tool. It is a relationship to a given task, tests, permissions and the impact of an error.
12. Checking can eat the saving from generation
An assistant often speeds up the first draft. That is visible even without a study. Less visible is the second bill: checking. A person must read the diff, understand why the change works, run tests, add a missing test, check security, find the effect on migrations and decide whether the result can be accepted.
On a simple task the check can be cheap. Tests say yes or no, and the error has little impact. On a mature system the check is itself the work. A model can generate a solution that passes locally but does not match the architecture of the service. It can change more files than were needed. It can remove a seemingly pointless guard that protected a historical edge case. Each such thing adds time for a senior.
Checking also has different prices by environment. In a library without data, a test and a short review may suffice. In a payment flow, a healthcare system or an internal administration with personal data, the same diff is more expensive, because the impact of the error is more expensive. That is not a question of fear of AI. It is the same logic teams use for human changes: a risky change needs more eyes, better observability and a plan for return.
How to test where the work has moved
Choose your own practice repository or an internal sandbox. Do not use other people’s production systems and do not exceed permissions. On the same type of task measure three times: the brief, the check, and bringing it to a state for which you would sign an incident. If writing got shorter but checking grew more, you do not have a replacement for a programmer. You have a shift of work.
13. The labour market is a different question from a merge into production
In the debate about programmers a wider labour-market question opens easily. How many ICT specialists there are, how wages are growing, how many students are arriving at schools, what OECD, ILO or Eloundou say about task exposure. Those are legitimate topics. But they belong on a different map from the question of whether a particular assistant will safely speed a change in a repository.
CZSO, for example, describes ICT specialists in Czechia and their wages [14][15][16]. Eloundou, ILO and OECD describe task exposure and changes of work in the wider economy [17][18][21]. These sources do not say whether your agent will correctly change authorisation, whether review will catch the error, or who will run a rollback at night. That is why this article does not build a ranking of occupations and does not carry percentages from the labour market into a software process.
The right boundary is: the labour market can say why the question matters. It will not answer for a merge. If an assistant is to help a particular team, it must show its own task, its own test, its own review and its own operational responsibility. Without that, the debate about replacing programmers is too wide to lead to a safe decision.
That is also why we do not use percentages from the article on occupations as a shortcut here. Task exposure in the economy can be high or low according to the method, but a merge request in a particular repository is decided on other evidence. Does the diff have a test? Does the reviewer understand it? Did it improve the service? Did it add a silent security debt? If those answers are missing, the labour market will not help. If they are clear, a team can use the tool without large statements about the end of a profession.
14. A benchmark score does not sign an incident
When an agent solves a benchmark, a score signs that. When a change takes production down, an organisation signs that. This difference is basic. A model is not an employee in a contractual relationship, is not the holder of a pager, and is not a person who will explain to a client why data were processed wrongly. A tool can propose a diff. Responsibility for accepting it stays in the process.
The AI Act classifies some systems in employment and management of workers as high-risk according to use, for example in hiring or in evaluating persons [20]. That is not an argument here that Copilot is banned, nor that programmers will disappear. It is a reminder that the use of AI in a decision about a person or about work has different requirements from generating a test in a sandbox.
Before a merge into production
Is the result covered by tests that measure the real risk, not only a happy path? Who is the owner of the service, and who has the right to accept the change? Is there a rollback, observability and a communication plan? Are secrets separated from a tool that is not supposed to see them? If a team cannot answer these questions, a benchmark score will not help it.
15. An agent with the right to change files is a different error from a hallucinated snippet
The older debate about AI in programming often looked like this: a model proposes a snippet and a person copies it. Today an agent reads a repository, edits files, runs tests, installs dependencies and sometimes prepares a commit. That is a different class of risk. A bad snippet is text. A bad agent step can change the state of a project.
It is not only about the quality of the model. It is about permissions. May the agent read production secrets? May it reach outside the workspace? May it run commands that change a database? May it push? May it resolve a conflict by removing someone else’s work? Each “may” changes the price of an error.
Good practice therefore does not start with the sentence that the model is clever. It starts with a sandbox, a preview of changes, separation of secrets, a limit on writes, and human approval where the result changes shared state. An agent can be an excellent helper in a safely bounded space. The same agent with too wide an access can do harm faster than the person it was meant to speed up.
Especially important is the difference between a proposal and an action. A proposal can be wrong and nothing happens if a person rejects it. An action changes a file, a configuration, a branch, a package or an environment. Once an agent acts, it needs the same logic as other automation: the least necessary permissions, a record of steps, a chance to preview and a chance to return. The intelligence of the model does not cancel that logic. It only widens the range of what automation can propose.
In an ordinary development process that means one simple brake. An agent may speed the preparation of a change, but acceptance of the change must still pass through a place where it is clear who decided. A commit without an owner, a test without meaning and a green run without a way back are not signs of mature AI. They are weaknesses of process that AI only sped up.
16. Dependence on an assistant is a different account from commit speed
A team can speed up with an assistant and still create a new debt. Not in the code, but in the workflow. If people stop understanding the changes they accept, dependence grows. If knowledge of the system moves from the team’s heads into prompts and a tool, the risk grows when a service fails, a price changes, or a model changes.
Dependence need not be bad. Every team depends on an editor, CI, a cloud and a package registry. The difference is whether the dependence is visible. If the assistant speeds routine and the team can still read a diff, keep tests and handle an incident without it, it is a tool. If without it they cannot understand their own changes, it is an operational weakness.
This account usually does not appear in a benchmark. A benchmark measures a task. Dependence measures the day when a service is down, a model changes, account rules tighten, or a new member of the team must explain an old commit. Anyone who scores only commit speed may miss the price of lost understanding.
17. Three times will tell more than a headline about replacement
A practical test for a team need not be large. It must be precise. Choose several tasks that you may measure safely. One simple one with good tests. One in a known part of the system. One where review or a security edge matters. On each, run a variant without the assistant and with the assistant, if that is organisationally possible. Do not look for other people’s vulnerabilities, do not work outside permissions, and do not give the agent secrets it does not need.
Measure three times. First: the brief and the making of a proposal. Second: the check, that is the diff, tests, review and fixes. Third: bringing it to a state for which the owner of the service would sign an incident. It is the third time that separates a demo from production. Code that passes one local test need not be a state the team will accept.
What needs to be tested
For each task write down in advance what done means. Not “the model answered”. Done means: tests passed, the review makes sense, the risk is named, a rollback exists, and the owner of the service would take the change into their process. If checking swallows the saving, the sentence “writing got shorter, responsibility did not” applies. If the whole chain gets shorter, you have a real benefit.
18. The question is not whether AI will replace programmers. It is who will bear the outage
A faster HTTP server can be true. More closed tasks can be true. A slowdown of experienced people in a known repository can also be true. These sentences do not contradict one another, because they do not measure the same thing. One speaks of a skeleton and tests. The second of a company’s flow of tasks. The third of context the model does not have. The fourth, the operational one, asks who will bear the consequence.
From that it does not follow that the question about programmers is a calm one. It follows that it is badly posed if it only reads “will replace, or will not”. Software is made of tasks, tests, review, operations and responsibility. Some parts will speed up. Some will move. Some may get cheaper. Some will get more expensive, because checking will matter more than writing.
The right question is therefore not whether AI will replace programmers. It is: on which task the assistant shortens writing, on which it lengthens checking in a known system, and who will sign the outage when the green tests at night are not enough. Until the answer ends at a benchmark, it lacks production. And until it ends at production without a measure of the task, it lacks evidence.
Related texts in this series
- Which occupations will AI replace… — the labour market; this text is a task, a test and on-call.
- ChatGPT vs. Gemini vs. Claude vs. Grok… — SWE-bench is a harness, not a verdict on an occupation.
- Who you allow to err