Section: Technology & AI Author: V Reading Length: ~27 min Sources and further reading: 22 items Topics: AI, Programming, Copilot, Cursor, Productivity, Tests, Operations, Accountability SEO / Working Title: Can AI replace programmers?
Can AI replace programmers? The question sounds simple until we notice that we are not comparing one job. Reducing time on the HTTP server in the lab is not a week in the team. The resolved GitHub issue is not on-call. The 50% time horizon model is not half-time. And the benchmark score does not sign the incident. Therefore, the article does not look for a date when the programmer will disappear. It looks for which task the assistant will speed up, where the work will move during the review, and who will remain during the downtime.
1. Twelve green tests on the HTTP server is not a nightly revert in production
Friday's demo looks convincing. Junior will demonstrate a small HTTP server in JavaScript. Tests are green. He explains that with Copilot he wrote the solution much faster than he expected. The tech lead sits next to him and says that in a large internal repository, the agent often adds work to him because he checks branches that he wouldn't have written himself. The product manager shows a graph of closed tickets. The director asks if he should hire fewer people next quarter. On-call engineer laysshorter question: who will revert it at night when the green tests are not enough.
At first glance, it is a dispute about whether or not AI will replace programmers. In reality, everyone measures a different object. Junior laboratory assignment with a clear assignment. Tech lead mature system that knows. Product manager number of completed units of work. The director would like to transfer the task to the recruitment plan. On-call person measures responsibility for traffic.
In a study with GitHub Copilot, Peng, Kalliamvakou, Cihon, and Demirer measured a programming task from May 15 to June 20, 2022: implementing an HTTP server in JavaScript that had to pass twelve tests. There were 95 Upwork freelancers in the study, and 35 people in each branch completed the task. The authors report a reduction in time of 55.8% with a 95 % confidence interval of 21 to 89 [1]. That's a strong number. However, it does not measure the production codebase, review, incident, ownership of the service, or the nightly decision whether a rollback will hurt customers more than a bug.
A faster HTTP server is not the same as being responsible for a production system.
— Jiný Kontext
2. Programming work has at least six layers
When the diff appears on the screen, it gives the impression that "the code was written by AI". But the result is created in several layers. Each can be measured differently and each can bring a different conclusion.
- Code generation: padding, boilerplate, tests, scripts and minor transformations.
- Modifying a known system: working in a repository, whose abbreviations, debts and unwritten rules one knows.
- Agent Benchmark: issue, patch, test run and score on the public set.
- Task length: the horizon at which the model completes increasingly longer work with some success.
- Measurements in the team: tickets, lead time, review, defects, work interruption and process change.
- Responsibility: merge, incident, security impact, audit, customer and rollback.
These layers are not subtle nuances. They are different objects of measurement. Peng measures the time to pass the tests on a clearly defined task [1]. Cui and co-authors measure task completion in organizations in field experiments [2]. METR in 2025 measures experienced open-source developers on real issues in repositories they know well [4][5]. SWE-bench measures the patching of GitHub issues in the harness [9]. Time horizon measures how long software tasks the models handle with a given success rate [6].
The labor market outline asks what will happen to the occupation. Here the question is narrower. Which software layer is currently being measured? If we don't name it, we end up with a sentence that turns one green test into personnel policy and one slowdown into a tool ban.
3. The best assistant is the one whose error is caught by test, review and rollback
It's tempting to measure typing speed with code. But software is not a text paragraph. A mistake can be obvious, cheap and reversible. Or it passes the tests smoothly and only appears in the customer's data. Therefore, a good assistant is not just the one who drafts the most lines. It is a tool whose failure is seen before it becomes an operational problem.
| A kind of failure | What does he look like? | How to test | What will limit the damage |
|---|---|---|---|
| Code confabulation | Diff looks done but changes authorization or edge state | Review, CI, proprietary test and rollback | The person merges, the agent has limited permissions |
| Poor measurement design | Vocation verdict from one RCT | Separate task, ticket, known repo and operation | Do not transfer the number to a part-time job |
| Move to review | Commit is faster, merge is slower | Measure input, control and acceptable status | Evaluate the entire string |
| Benchmark as a statement | SWE-bench scores as a substitute for team work | List what the kit does not measure | Internal tasks before the public caption |
| Incident without owner | "AI wrote it" instead of responsible merge | Who reverts, who reports, who explains | Runbook per person and organization |
This also changes the purchasing question. We're not just asking how much time the assistant will save typing. We ask if we have tests, review, observability, permissions, and rollback that will handle his error. Where error is cheap and control clear, the payoff can be great. Where a bug leaks into production, faster generation may just push the problem to a more expensive location.
4. Three designs, three directions. Not three vocation verdicts
Public debate likes one chart. Research on the productivity of AI programmers offers rather three different designs.
Peng and co-authors measured a lab task with a clear outcome: an HTTP server in JavaScript, twelve tests, Upwork freelancers, and time to completion [1]. Cui, Demirer, Jaffe, Musolff, Peng, and Salz, as of February 2025, work with 4 867 developers at Microsoft, Accenture, and one Fortune 100 company. Their instrumental estimate of Copilot usage per task completed is +26.08% per week with a standard error of 10.3 [2][3]. METR and Becker et al. in 2025, they tracked 16 experienced developers, 246 real issues, an average human time of 2.0 hours, and repositories with approximately 23 thousand stars. Measured 19 % time increase with Cursor Pro and Claude 3,5/3,7 [4][5].
That's not three verdicts. There are three windows. In one, there is a clear skeleton and tests. In the second, common corporate tasks and tool adoption. In the third, people who know their code and have to check designs against the context they have in their heads. None of these measurements alone tell whether the employment contract will end. And none of them can be honestly used without describing the population, period, metric, and what didn't count.
5. Speeding up on a skeleton is not a junior's job
The result of Peng's study is important precisely because it is narrow. It shows that on a well-defined task with a testable output, the assistant can significantly reduce the time. Especially where there is a lot of boilerplate and tests will quickly tell if the solution works. This is exactly the type of work that many less experienced people learn to do.
It does not follow that the junior can be deducted from the budget. The study had a specific task, a specific period, and a completed sample of 35 people in each branch [1]. It did not measure onboarding, knowledge of the company domain, communication with the product, debugging of requirements, review of foreign code or keeping the service in operation. She didn't even measure whether the team would pay for the missing training of new people in half a year.
Therefore, the correct managerial sentence is not: "AI will do junior work." It reads: "For tasks like this, writing can be significantly shortened if we have clear tests and cheap control." This is less effective. However, it is more usable.
It's a different test for the team. Take your own small task, which resembles a skeleton: known assignment, clear result, automatic tests. Have it done with and without an assistant. To measure not only the writing time, but also the time when the senior reads the diff. If the save is lost in the review, the work has not been removed. She was moved.
Still, this move could be good news. A junior who produces a first draft faster with an assistant may encounter questions of architecture, testing, and accountability earlier. But this is only true if the team does not let the model replace the learning. If you don't know why the code passed the test, you can't maintain it. When he can't explain his own diff, he's not ready to carry the change to a shared system. The contribution of the assistant must therefore be measured along with the learning: it accelerated the path tounderstanding, or just produced more text for someone else to untangle?
6. More tickets completed is not more quality in traffic
Cui and co-authors move the debate from the laboratory to the corporate world. That's a big change. Four thousand eight hundred and sixty-seven developers, three organizations, and measuring completed tasks per week is closer to what managers care about [2]. The estimate of +26.08% of completed tasks per week is lower than the laboratory effect for Peng's task, which the authors themselves relate to the difference between pure task and routine work [2].
But even the ticket is only a unit of the process. A completed task is not automatically a quality change in production. It depends on what the organization considers complete, how strong the review is, whether defects, incidents, customer impacts and other people's time are measured. The team can close more small tasks with the assistant and burden the review at the same time. Or it can actually increase throughput without compromising quality. Data on completed tasks is the beginning, not the final account.
This is where the replacement debate often breaks down. If a company only tracks the number of tickets, it will be tempted to convert the percentage to people. If he also tracks defects, reviews and operational changes, he may discover something else: the assistant increases the volume of work, but requires a different way of control and a different distribution of responsibilities.
In field measurements, it is also important who actually uses the tool and for what tasks. Adoption is not randomly spread like paint on a wall. Some reach for the assistant for simple repairs, some for library research, some for tests, and some hardly at all. If the number of completed tasks is then tracked, the result can be a mix of the ability of the tool, people's willingness to use it, the type of tickets assigned, and team rules. Cui and co-authors are researching itby design; the reader should take from this mainly caution when translating to his own company [2].
Therefore, the corporate pilot does not have to just turn on the tool and count the closed items after a month. They must differentiate between task types, people's experience, diff size, review time and number of reverted changes. Otherwise, a good pass report may mask a new bottleneck for those who control the changes.
It is even more accurate to track who pays for the acceleration. If the assistant helps the author of the task, but the reviewer spends more time verifying, the saving has shifted between people. If the number of tickets increases, but the number of returns from QA increases, the saving has moved to another queue. If the entire chain from input to safely accepted change is accelerated, then the team has proof that is worth more than a general headline about AI.
7. Slowdown for people familiar with the repo is not proof that the assistant is not working
METR in 2025 produced an inconvenient result. Experienced open-source developers worked on issues in repositories they knew well. The average experience with a given repository was several years, the tasks had an average human time of 2.0 hours, and the study took place from February to June 2025. When they were allowed to use AI tools, time increased by 19 % on average [4][5].
Even more interesting is the anticipation. The developers expected a 24% speedup in advance. After the study, they still estimated that they were 20% faster with AI, even though the measured result went in the opposite direction [5]. That's not proof that the developers don't know what they're doing. It is a warning that the subjective feeling of smooth work can deceive even experienced people.
This does not mean that the assistant does not work. It follows from the fact that in a known system, a person has a context in his head that the model does not have. He knows which seemingly simple solution will break the old integration. He knows why the special helper is special on purpose. It knows that the test will only fail in combination with data that is not in the repository. AI can add suggestions, but also dead ends that need to be verified.
Three productivity tests
Are you timing the green tests on the new skeleton? Do you measure the number of completed tasks in the team? Or do you measure time per issue in your own mature repository? Three positive answers do not add up. Each belongs to a different decision.
8. 50% time horizon is not 50% of full-time compensation
METR in the study by Kwa et al. measures the ability of models to complete increasingly longer software tasks. It works with a set of 170 tasks, including HCAST, RE-Bench and 66 SWAA, and twelve frontier models. For the o3 model, it lists approximately 110 minutes of human time as the 50% time horizon [6]. The study also estimates that this horizon has doubled roughly every seven months since 2019, to be exact 207 days with a 95 % interval of 166 to 240 days [6][7].
This is important for the direction of development. However, it is not a sentence that the model replaces half of the programmer. A 50% horizon means that the model has approximately half the success rate on a given set for a task equivalent to a certain human time. METR also states that the 80% horizon is roughly four to six times shorter [6][7]. Production usually doesn't want half a chance. He wants reliability, repeatability and responsibility.
In addition, the METR Note dated March 20, 2026 notes that time horizon results are sensitive to modeling assumptions, regularization, and estimation of human times [8]. This does not break the trend. It just prevents translation into a simple sentence about months of work in any company.
The 50% time horizon tells how long a task the model sometimes takes to complete. It does not say who will reverse the outage at night.
— Jiný Kontext
9. SWE-bench fixes issue with tests. He does not design architecture and does not carry a pager
SWE-bench is a useful benchmark because it tries to come out of real GitHub issues. The original work by Jimenez and co-authors worked with 2,294 issues from twelve Python repositories. In the paper, Claude 2 with BM25 search achieved 1.96% resolved issues; GPT-4 was scored on a 25 % subset due to budget [9]. OpenAI's later SWE-bench Verified introduced a human-verified sample of 500 items and reports GPT-4o with a score of 33.2% [10]. Epoch AI also describes the verified set as a variant with 500 or 484 usable samples depending on the settings [11].
This is an advance in agent measurement. It's not a programmer's job. The benchmark does not include product consultation, architecture decisions, reading internal incidents, migration without downtime, security review or on-call. Contains issue, repository, tests and scoring method.
So the right question is not whether SWE-bench shows the future of programming. It shows one important layer: the ability of the models to find and fix some bugs in the open repositories under test. It is a signal for instrument selection. This is too little for a production liability plan.
10. Boilerplate acceleration in the less experienced is not the demise of the senior
Results around coding assistants often show a greater benefit for less experienced people or people in a less familiar part of the job. That makes sense. Where one is looking for syntax, boilerplate, a common pattern, or a first draft of a test, the model can quickly supply material for editing. Where the senior holds the history of the system in his head, suggestions can add noise.
This does not result in the demise of the senior citizen. The opposite convenient sentence that the senior is beyond the reach of change does not apply either. Rather, the composition of work is changing. A less experienced person can get to the first solution faster, but they have to learn to read the diff, understand the tests, and recognise when the model has just confidently copied the wrong pattern. A senior may write less routine code and more design boundaries, review security, and decide whether to accept change.
A team that makes it all about saving people risks two things. They will lose the apprenticeship work on which future seniors grow up. And it transfers control to a smaller number of people, who then carry a larger review queue. A team that bans an assistant because of one slowdown can leave cheap gains on well-testable tasks lying around.
11. Using the tool is not trusting the output
Stack Overflow Developer Survey 2025 shows high adoption of AI tools. According to the report, 84 % of respondents use or plan to use AI tools, up from 76% in 2024. Among professionals, 51% report using AI on a daily basis. At the same time, 46 % of respondents do not trust the accuracy of the output rather than trust it, while 33% express trust [12][13].
This is not a representative sample of all programmers in the economy. It is a self-selection of Stack Overflow respondents. Still, the signal is useful: usage and trust are not the same thing. One can use the tool every day because it saves time in searching, designing a test or explaining an error. At the same time, they don't have to believe that output without control belongs to production.
It is this difference that is healthy for software. A programmer does not have to be an opponent of AI to reject autonomous deployment. He doesn't have to be a geek to use code completion. Trust is not a sticker on a tool. It is related to a certain task, tests, permissions and impact of the error.
12. Inspection can eat up generation savings
The assistant often speeds up the first draft. This can be seen even without a study. Less visible is the second account: control. One has to read the diff, understand why the change works, run the tests, write the missing test, check the security, find the impact on the migrations and decide if the result can be accepted.
For a simple task, inspection can be inexpensive. The tests say yes or no, and the error doesn't have much impact. In a mature system, control is work itself. The model may generate a solution that passes locally but does not conform to the service architecture. It may edit more files than needed. It can remove the seemingly unnecessary guard that protected the historical edge case. Every such thing adds time to the senior.
Inspection also has different prices depending on the environment. In a library without data, a test and a short review can be enough. In a payment flow, healthcare system or internal administration with personal data, the same diff is more expensive because the impact of the error is more expensive. It's not a question of fear of AI. It's the same logic teams use for human change: risky change needs more eyes, better observability, and a rollback plan.
How to test where a job has moved
Choose your own practice repository or internal sandbox. Do not use third-party production systems and do not exceed permissions. For the same type of task, measure three times: enter, review, and bring to the state you would sign off on the incident. If typing has become shorter, but checking has grown more, you don't have a replacement programmer. You have a job transfer.
13. The labor market is a different issue than merging into production
In the debate about programmers, the broader issue of the labor market is easily opened up. How many ICT specialists are there, how are wages growing, how many students are coming to schools, what do the OECD, ILO or Eloundou say about exposure to tasks. These are legitimate topics. But it belongs to a different map than the question of whether a specific assistant will safely speed up changes in the repository.
For example, the CZSO describes ICT specialists in the Czech Republic and their salaries [14][15][16]. Eloundou, the ILO and the OECD describe the exposure of tasks and changes to work in the wider economy [17][18][21]. These sources don't say whether your agent will correctly modify the authorization, whether the review will catch the error, or who will trigger the rollback at night. Therefore, in this article, we do not build a ranking of professions and do not transfer percentages from the labor market to the software process.
The right line is: the labor market can tell why the question is important. It won't respond to merge. If the assistant helps a specific team, he must show his own task, his own test, his own review and his own operational responsibility. Without that, the debate about replacing programmers is too broad to lead to a safe decision.
This is also why we don't use percentages from the occupations article here as shorthand. The exposure of tasks in the economy can be high or low according to the methodology, but the merge request in a particular repository is decided according to other evidence. Does it have a diff test? Does the reviewer understand him? Did he improve the service? Didn't he add a silent security debt? If these answers are missing, the labor market will not help. If they are clear, the team can use the tool without making big declarations about the demise of the profession.
14. Benchmark scores do not sign an incident
When an agent solves a benchmark, it signs the score. When a change throws off production, the organization signs off on it. This difference is fundamental. The model is not an employee in a contractual relationship, is not a pager holder, and is not the person to explain to the client why the data was processed incorrectly. The tool can suggest a diff. Responsibility for its acceptance remains in the process.
The AI Act classifies certain employment and workforce management systems as high-risk based on their use, for example in recruiting or evaluating people [20]. This is not an argument here that Copilot is banned, nor that programmers will disappear. It's a reminder that using AI in a human or job decision has different requirements than generating a sandbox test.
Before merging into production
Is the result covered by tests that measure real risk, not just luck? Who is the owner of the service and who has the right to accept the change? Is there a rollback, observability and communication plan? Are the secrets separated from the tool that is not supposed to see them? If the team can't answer these questions, the benchmark score won't help them.
15. An agent with the right to change files is a different error than a hallucinatory snippet
The older debate about AI in programming often went like this: the model suggests a snippet and the human copies it. Today, the agent reads the repository, edits files, runs tests, installs dependencies, and sometimes prepares a commit. That's a different class of risk. The wrong snippet is text. A wrong agent step can change the state of the project.
It's not just about the quality of the model. It's about permissions. Can an agent read production secrets? Can it extend outside the workspace? Can they run commands that change the database? Can he push? Is he allowed to fix the conflict by removing someone else's work? Each "may" changes the error cost.
Therefore, good practice does not start with the sentence that the model is smart. It starts with sandboxing, previewing changes, separating secrets, restricting writes, and human approval where the result changes shared state. An agent can be an excellent helper in a safely defined area. The same agent with too broad an approach can do damage faster than the person it was meant to speed up.
The distinction between proposal and action is particularly important. The proposal may be wrong and nothing will happen if one rejects it. An action changes a file, configuration, branch, package, or environment. Once an agent acts, it needs the same logic as other automations: least permissions needed, record of steps, preview option, and rollback option. The intelligence of the model does not override this logic. It just increases the scope of what automation can design.
In the normal development process, this means one simple brake. The agent may speed up the preparation of the change, but the acceptance of the change must still go through a place where it is clear who made the decision. A commit without an owner, a test without meaning, and a green run with no rollback are not signs of a mature AI. It's the weaknesses of the process that AI has only accelerated.
16. Helper dependency is a different account than commit speed
A team can accelerate with an assistant and still create new debt. Not in code, but in workflow. If people stop understanding the changes they are accepting, addiction grows. If knowledge of the system moves from the head of the team to the prompts and the tool, the risk increases in the event of a service outage, price change, or model change.
Addiction doesn't have to be bad. Each team depends on the editor, CI, cloud, and package registry. The difference is whether they see the addiction. If an assistant speeds up the routine and the team can still read diffs, maintain tests, and resolve an incident without it, it's a tool. If they can't understand their own changes without it, it's an operational weakness.
This account will not usually appear in the benchmark. A benchmark measures a task. Dependency measures the day a service goes down, a model changes, account rules tighten, or a new team member has to explain an old commit. Those who value only the speed of the commit may overlook the cost of lost understanding.
17. Three times will tell more than a replacement headline
A practice test for a team doesn't have to be big. Must be accurate. Choose a few tasks that you can safely measure. A simple one with good tests. One in a known part of the system. One where review or safety margin is important. For each, run a variant without an assistant and with an assistant, if it is organizationally possible. Don't look for other people's vulnerabilities, don't work outside of authorization, and don't give an agent secrets they don't need.
Measure three times. First: entering and creating a design. The second: control, i.e. diff, tests, review and corrections. Third: putting the service owner in a state that would sign the incident. It is the third time that distinguishes the demo from the production. Code that passes one local test may not be the state the team accepts.
What needs to be tested
For each task, write down in advance what it means to be done. Not "the model answered". Done means: the tests have passed, the review makes sense, the risk is named, the rollback exists and the service owner would take the change into their process. If the control swallows the saving, the phrase "writing has shortened, responsibility has not" applies. If the entire chain is shortened, you have a real benefit.
18. The question is not whether AI will replace programmers. It sounds like who will bear the fallout
A faster HTTP server may be true. More closed tasks can be true. Slowing down experienced people in a known repository can also be true. These sentences are not mutually exclusive because they do not measure the same thing. One talks about skeleton and tests. The second about corporate task flow. The third about the context that the model does not have. The fourth, the operational one, asks who will bear the consequences.
It does not follow that the question of programmers is calm. It follows from the fact that it is misplaced if it only reads "will replace or not replace". The software consists of tasks, tests, review, operation and responsibility. Some parts will speed up. Some will move. Some may be cheaper. Some will become more expensive because checking will be more important than writing.
So the right question is not whether AI will replace programmers. It reads: on which task the assistant shortens the writing, on which lengthens the check in the known system, and who signs the outage when the green tests at night are not enough. As long as the response ends at the benchmark, it lacks production. And as long as it ends up with production without measuring the task, it lacks proof.
Related texts in this series
- Which occupations will AI replace… — labor market; this text is an assignment, test and on-call.
- ChatGPT vs. Gemini vs. Claude vs. Grok… — SWE-bench is a harness, not a vocation verdict.
- Who do you allow to make mistakes?