AI Can Write the Code. But Code Was Writing Us.

What happens to software engineering when we automate not only the work, but also the struggle that helped shape the engineer?

Earlier this year, in Coding is not fun anymore?, I wrote about an unfamiliar feeling: I still enjoyed building software, but I was writing less and less of the code myself. I hadn't forgotten how. Writing it had stopped being the bottleneck.

In The Bottleneck Moved to Review, I described where that work had gone. Agents handled much of the implementation. I spent more time specifying requirements, designing systems, checking security, reviewing changes, and verifying results.

In my own work, I consider that progress. I wouldn't generalize that experience into a claim about everyone's productivity. Lately, though, I've been thinking about something those posts left unresolved:

If AI writes most of the code, what happens to the engineer who used to be shaped by writing it?

Terence Tao's recent comments on AI and mathematics helped me see the question more clearly.

A correct answer isn't the whole achievement

In a thread on September 3, 2026, Tao used Navier–Stokes to illustrate a troubling possibility. Imagine a powerful AI system with enormous computational resources. It explores possible approaches, encounters obstacles, revises its attempts, and eventually produces a proof of a major open problem. That proof might even be formalized in Lean and checked by a machine.

The problem is solved. But the question I take from Tao's argument is: how much has mathematics learned?

In an earlier thread, on April 22, he separated the problem-solving part of mathematical research into three activities:

• Proof generation: finding a solution.
• Proof verification: checking that the argument establishes the result it claims.
• Proof digestion: understanding why the argument works, connecting it to existing mathematics, and extracting ideas that might help with other problems.

Tao's argument is that AI and formalization are accelerating the first two faster than people can absorb the results. This is his assessment of mathematical practice, rather than a general measurement of the cost of solving problems. (Part 2 of the thread)

Under the standard he proposes, a proof that has been generated and verified is still an incomplete achievement until someone can explain it and place it within the field's understanding. (Part 3)

I'm using the Navier–Stokes scenario as a thought experiment here, not making a claim about the problem's current status. What interests me is the gap between obtaining a correct result and learning something from it. Software engineering may be heading toward a version of the same problem.

We thought the product was the code

Viewed only through its output, programming looks straightforward. There is a problem. A programmer thinks about it, writes code, and gets it running. A product appears.

In that account, implementation is an intermediate cost. If a machine can do it faster, handing it over seems entirely sensible.

Peter Naur offered a different account in his 1985 essay, Programming as Theory Building. For Naur, programming produces more than a program. The people doing it also develop a theory of the system: what problem it addresses, how its parts relate, why particular choices were made, and how it should respond when circumstances change.

Here, a theory of the system means a practical understanding of how it works and why it was designed that way: enough to predict behavior, justify a design choice, and reason about a change. The source code is one part of the result. Another part lives in the people who built it.

In this view, giving two engineers the same repository doesn't give them the same ability to change it. Someone who helped build the system may know things the documentation hasn't captured. They know the design rationale: why those decisions made sense at the time.

Think of the engineer who knows which failed design led to an abstraction, why the retry policy works the way it does, or why an awkward module cannot yet be refactored. They remember the apparently simple requirement that once produced a race condition. Their advantage is more than a memory of individual lines of code; they have an explanation for why the system has this shape.

Naur argues that passing on this understanding requires opportunities to work alongside people who already have it, discuss unusual behavior, and practice under guidance. Exposure to code and documentation alone is insufficient in his account. (“Program Life, Death, and Revival”)

Building the software, becoming the engineer

This is the distinction I want to carry forward from Naur:

Building software also builds the engineer.

Consider a hypothetical project. You write an initial version before you fully understand the problem. It's slow. You profile it, and the numbers contradict your expectations: moving data through memory is the bottleneck.

You change the data representation. Memory use falls, but tail latency, the response times of the slowest requests, gets worse. You add batching, which helps one workload and hurts another. You read, experiment, and revise.

Suppose the system is better after three weeks. I'd hope the engineer now understands it better too. The next time, they might recognize the memory bottleneck earlier. They might see a family of related problems where they previously saw an isolated bug. By engineering judgment, I mean being able to decide what matters, anticipate consequences, and choose a useful next experiment even when the answer is uncertain.

The software is visible. The judgment developed while building it is much harder to see.

Practice creates opportunities to learn; it doesn't guarantee deep understanding. The point connecting Naur's argument to the studies below is that finishing software and developing understanding are two outcomes worth examining separately.

Suppose an agent builds a better system in two hours than I could write in two days. I've delivered more software. I haven't necessarily become a better engineer in the process.

That separation is what I think we're underestimating.

The concern goes beyond learning less

I explored a related tension in AI Is Making Us Faster Learners and Worse Thinkers at the Same Time. Some programming experiments now offer direct evidence that task completion and understanding can move in different directions. Their findings are narrower than the claim that AI makes engineers worse.

In an Anthropic experiment published in January 2026, 52 engineers, mostly junior, learned the Python library Trio. The AI group averaged 50% on an immediate assessment, compared with 67% for the group working without AI assistance: a 17-percentage-point gap. The largest difference was on debugging questions. The AI group finished slightly faster, but the time difference was not statistically significant.

Participants who asked conceptual questions or actively worked to understand the answer tended to perform better than those who delegated much of the work. That was a qualitative observation, not evidence of a causal relationship between a particular interaction style and the result. The experiment also measured immediate understanding of a new library, not long-term changes in engineering ability.

A second study, More Code, Less Understanding?, published in IEEE Transactions on Software Engineering, asked 69 participants to complete two programming tasks, one with AI and one without. With AI, participants completed more of the assigned work but were less able to answer technical questions about the code they had just produced. The sample included students, researchers, and professional developers. Again, the assessment captured understanding immediately after the task, rather than engineering competence as a whole.

Those findings matter. If the concern is that people understand less of the code they produce, code walkthroughs, explaining a change to a colleague, and documenting design decisions are plausible interventions. Margaret-Anne Storey proposes such practices to address cognitive debt. They are practical proposals, not proven remedies for every effect of AI assistance.

But I worry about something harder to recover: the ideas we never discover because we never have to work through the implementation.

The questions we find while building

Imagine that I want to reduce latency in a model inference server. I care about p99 latency, the 99th-percentile response time: 99% of measured requests finish at or below it. An acceptable average can hide a long tail of slow requests, which is why Google's SRE guidance recommends examining the distribution. I could give an agent this task:

Profile the server, optimize batching and memory use, keep p99 latency below X, benchmark the alternatives, and implement the best one.

In this hypothetical example, a capable agent might try several configurations and return a report like this:

Throughput increased by 41%. p99 latency fell by 28%. All tests passed.

Those are illustrative numbers, not measurements from a project. They describe an outcome I would be happy to receive.

Now imagine taking a more direct role in the same work. I open the profiler expecting computation to dominate and discover that copying data is the problem. I change the representation, then notice a tradeoff between waiting to form a batch and the latency of individual requests.

That tradeoff is real: NVIDIA's Triton documentation recommends experimenting with larger batches or queue delays to trade additional latency for throughput. Which configuration is useful still depends on measurements of the workload. Suppose mine contains requests of very different lengths. I start thinking about the scheduler.

A new question appears: should its scheduling policy adapt to the mix of incoming requests, latency targets, and GPU utilization?

I started with “How do I reduce latency?” Working through the implementation led me to “Could we build a scheduling policy that adapts to real traffic and operating cost?”

That second question might become a library, a paper, a product, or simply a way of thinking I use for another ten years. In this scenario, I didn't know to ask it at the start. I found it along the way.

I have described a related, real experience in The GPU Was Already There: Four Silent Bugs in On-Device Training. Claude Code helped with much of the mechanical implementation in AnyLearning. The insights that mattered to me came from measuring what the system actually did.

In the measurements documented there, device-selection logic overlooked the Mac's GPU. is_bf16_supported() returned True on an RTX 2070 even though BF16 arithmetic was emulated. FP16 was slower than FP32 on the M1 I used. These were observations on my particular machines and workloads, not rules for every GPU. (The AnyLearning case study)

Another surprise was that GradScaler can skip optimizer.step() when gradients contain inf or NaN. That behavior is explicitly documented in PyTorch's AMP examples. “Silent” described my application's failure to log skipped steps; the mechanism itself is intentional.

For me, that project illustrates the value of staying close to the system's behavior. A profiler result that contradicts my prediction, a security defect, or a mismatch between a requirement and how people actually use the software can all be reasons to reconsider the question. That is my interpretation of the value of practice, not a claim that accumulating more failures automatically produces a better engineer.

Friction isn't always waste. Sometimes it carries information.

By friction, I mean the resistance that makes me revise my understanding: a failed approach, an unexpected measurement, or a requirement that turns out to be incomplete. Repeating a tedious edit is a different kind of effort. The distinction I want to make is about what the difficulty gives me a chance to learn.

Here is the hypothesis I want to explore: if AI resolves every obstacle before we encounter it, the software may improve while our opportunities for discovery shrink. The studies of assessment scores above did not measure that loss.

I call it discovery debt. This is my name for a possible risk, not an established metric or an experimental finding.

It helps to distinguish it from the three forms of debt in Storey's From Technical Debt to Cognitive and Intent Debt:

• Technical debt makes future changes more expensive because of decisions embodied in the code.
• Cognitive debt is the erosion of a team's shared understanding of the system.
• Intent debt is the absence or loss of documented goals, constraints, and design rationale that people and agents need to change the system.

The distinction I want to draw is this: cognitive debt concerns how little we understand the system we have; discovery debt concerns what we never thought to investigate. In the inference-server example, understanding the final batching code would address the first problem. Never asking whether the scheduling policy itself should change would illustrate the second.

I use “debt” as a metaphor for a missed opportunity to develop understanding, not a balance we can calculate or assume we can repay. A technical problem can become a ticket. A question you never discovered is harder to put in a backlog. You may not even know what you're missing.

Didn't compilers take work away too?

There is an obvious objection. If struggle matters so much, why not write assembly? Why use garbage collection, a database, or PyTorch instead of implementing everything ourselves?

Programming already gives us tools that spare us from building every layer. In LLVM, the register allocator maps virtual registers to physical registers or memory locations. PostgreSQL provides B-tree indexes and write-ahead logging. Application developers can use those mechanisms without implementing them, while still needing to understand the guarantees and limitations relevant to their applications.

So “write everything yourself” cannot be the answer.

The useful question for me is which decisions I still engage with after delegating a task. Automating an implementation step and handing over the choices that define the problem are different degrees of delegation. An agent can be given responsibility for choosing what to do next. In Anthropic's description of agent architectures, the model directs its workflow, uses tools, and adjusts its actions based on feedback from the environment.

Consider a broad delegation: I let an agent choose the architecture, dependencies, and data model, then read only the final diff. I miss the alternatives that were rejected. That is a risk of this arrangement, not an unavoidable property of agents.

I've delegated much of the design exploration: trying alternatives and learning why some don't work. In my reading of Naur, making and revising those choices is one opportunity to build a theory of the system.

That is also why specification has become more important in my work. In Ask, or It Will Guess, I described a failure mode I worry about: an agent receives an ambiguous request, chooses an interpretation, and starts building.

A study of question answering offers a useful parallel. Su and Cardie (2026) found that the models they evaluated generally answered ambiguous questions directly rather than asking for clarification. They studied question answering using a specific set of models, not coding agents working in repositories. Applying that result to how I delegate software work is my inference.

The risk is an excellent implementation built on an assumption nobody deliberately chose. When I delegate that choice without examining it, I may also lose an opportunity to understand the problem better.

Plenty of code, too little understanding

This suggests a software version of Tao's concern about proof abundance.

Suppose code generation continues to get cheaper. One engineer runs five agents, each trying three implementations. Continuous integration (CI) runs builds and tests, static analysis looks for problems, and another model reviews the changes. Each check has a limited scope. Passing them is not equivalent to a formal proof that the entire system is correct.

If this workflow works well, code that passes automated checks could accumulate faster than people can understand it. That is a scenario to consider, rather than an established forecast. Within it, a further constraint emerges: turning working code into a working understanding of the system. For software, I'd call this building a mental model: an understanding of the system that lets me explain a design, question its assumptions, and reason about changes. That is the counterpart to Tao's “proof digestion.”

The analogy looks like this, with an important limit: testing software and verifying a mathematical proof offer different kinds of assurance.

In The Bottleneck Moved to Review, I argued that my bottleneck was shifting from writing code to reviewing it. I still think that is true. But the deeper bottleneck may be how much understanding we develop through that review. Careful review can itself be a way to build a mental model; accepting a correct diff does not, by itself, tell me how much I learned.

The longer-term question could be: how quickly can we turn what machines produce into understanding we can use?

I worry that teams may change how they assign work before they work out how people will learn from it.

How does the next engineer gain that judgment?

This is the part that concerns me most. Imagine an experienced engineer advising a junior colleague:

Let AI handle the repetitive implementation. Focus on architecture.

That can be sensible advice. But consider how the experienced engineer got there. Perhaps they designed a bad API, chose the wrong abstraction, tracked down a deadlock, or chased a memory leak. They shipped something that didn't scale, resolved a production failure the tests hadn't caught, or watched an elegant diagram fall apart when it met real data.

Those are examples of a possible path to understanding, not a checklist everyone must complete before being allowed to evaluate architecture. They fit Naur's emphasis on guided practice and discussion as part of learning a system.

If early-career engineers no longer encounter that work, what replaces its educational role?

We could automate the work junior engineers learn from before we have another way to help them become experienced engineers.

Imagine how this might play out within a company. This is a thought experiment, not a labor-market forecast. A company is highly productive in 2026 because its senior engineers use agents well. Five years later, it discovers that the next cohort is excellent at coordinating AI but has had too little contact with real failures to develop the judgment needed for an unfamiliar situation.

In that scenario, the company improved software production while underinvesting in the people who would eventually supervise it. Anthropic's experiment raises a short-term concern about learning a new skill. It does not establish this five-year outcome. That longer horizon is the question I want teams to take seriously.

Keep the friction that teaches you something

I don't intend to return to the way I wrote code in 2021. Nor do I think junior engineers should be banned from using Copilot, Claude Code, or Codex. I want a better distinction between repetitive effort and effort that builds understanding.

My working rule is to use AI to remove repetition while protecting the work that helps me understand. That means thinking about who gets the learning opportunity as well as who gets the task. I want to keep the moments that challenge my model of the system, without preserving every inconvenience around them.

1. Define the requirements and constraints

Before an agent implements something, I want to be clear about the requirements and constraints:

• Where are the trust boundaries?
• What latency budget do we have?
• Which data is authoritative?
• How should the system behave when something fails?
• What privacy requirements constrain the design?

If I cannot answer those questions, I am not ready to delegate the implementation.

2. Make a prediction before asking

When a task offers a useful chance to learn, I try to predict the behavior or bottleneck before asking AI about it.

There is an experimental basis for trying an answer before receiving one. Kornell, Hays, and Bjork (2009) found that attempting an answer, even unsuccessfully, improved subsequent learning compared with seeing the question and answer together. Their materials were knowledge questions and word associations, not profilers or coding agents. Predicting a bottleneck before asking AI is my application of the idea; that specific practice wasn't tested in their experiments.

3. Stay involved when the results surprise you

I'm happy to delegate boilerplate, migrations, routine tests, and straightforward refactoring. When a profiler, error, or benchmark produces something unexpected, though, I want to look at it myself before asking for an explanation.

4. Keep a record of the experiments and decisions

For a substantial change, I want the alternatives that were tried, the assumptions that shaped the design, the error logs, and the benchmark results. A plausible explanation after the fact isn't enough. I want to be able to connect it to experiments that actually ran.

The final diff doesn't give me that record of what was tried and what happened. Keeping that evidence available for review is consistent with Anthropic's recommendations on environmental feedback and agent transparency.

5. Explain it without the diff

After reviewing an important change, try closing the diff and explaining how the system works and why the change was made. Chi and colleagues' study of self-explanation (1989) observed that students who learned with greater understanding produced more explanations connecting solution steps to underlying principles. The study concerned mechanics problems; the following code-review exercise is my proposed application, not a procedure it validated.

Try answering these questions without looking:

• Where does the data go?
• Where does state live?
• How can the system fail?
• Why was this architecture chosen?
• What future requirement would make the design unsuitable?

I'm trying to build this into my own work with agents. Alongside “Does it run?” I want to ask, “Can I explain why it works?” If I can't, I have more work to do before I can responsibly change or troubleshoot it. That is what I mean here by taking ownership: being able to explain and maintain the code, regardless of who typed it.

The aim is to preserve feedback that develops judgment, rather than making work slower for the sake of a difficult lesson.

Knowing how to code is a way to investigate

If AI can write code, do we still need to learn to program?

I find that question too narrow. Remembering syntax, churning out CRUD endpoints, and recalling framework APIs are not enough, on their own, to define the competence I want in an engineer working with AI.

Code is also a way to investigate a system. Follow execution. Set a breakpoint. Change an assumption. Run a benchmark. Write a small experiment. Look beneath an abstraction to see how it works.

Viewed that way, writing code is closer to experimentation than typing.

The analogy I have in mind is a physicist using equipment someone else built while still taking responsibility for the questions and the interpretation of measurements. Imagine a version of the experiment reduced to an instruction to AI followed by a finished chart. I would want to know how much the person conducting it understood about the phenomenon.

That is the ability I want to retain in software: going deep enough into a system to develop an independent explanation of how it works. I don't need to write every line to do that. I do need to remain able to investigate.

More than an orchestrator

In Coding is not fun anymore?, I described a shift from implementing software myself to orchestrating agents: defining tasks, coordinating their work, and reviewing the results. I still think that describes where my work is going. It just doesn't describe everything I want to preserve.

The engineer I want to be does more than divide up tasks, read reports, and approve results. They build explanations, notice contradictions, and recognize a broader problem inside a particular failure. They turn a workaround into an abstraction. They discover that the original requirement asked the wrong question. They start implementing a solution and come away with a better question.

That is discovery, and it is where I think individual judgment could matter most in a world with abundant code. I care about the mental models an engineer develops, the questions they notice, the experiments they devise, and the knowledge they can take from one system into the next.

Imagine two engineers using the same model, IDE, agent setup, and compute budget. Both finish the feature. One also leaves the project with a new way to understand an entire class of problems.

That is a difference worth protecting.

The code was writing us

The AnyLearning work showed me a benefit I want more of: AI helped me finish work I'd been putting off. I want agents to write migrations, fix repetitive bugs, and try ten configurations. I want the cost of turning an idea into working software to keep falling.

As it does, I think we need to evaluate something that project delivery metrics can overlook: how much the engineer learned.

That is what I take from Naur. Working software matters, and so does the understanding of the people responsible for it. While we debugged, refactored, measured, failed, and tried again, we were also changing how we thought.

We were writing code. The code was writing us.

The studies on AI and immediate understanding give me a reason to be more deliberate about keeping those activities connected.

I want to ask more than how much code AI can write for us. Which parts of writing code were quietly writing us, and how do we preserve them when a machine can do the implementation?

There is room for a hopeful answer: engineers freed from repetition who still develop curiosity, depth, and judgment. People who don't have to walk every road, but know when to get out of the helicopter and look at the terrain.

That is the kind of engineer I want to become. It is also the kind I think we need to learn how to develop.

---

Related posts

• Coding is not fun anymore?: where enjoyment and professional identity might come from when AI handles much of the implementation.
• AI Is Making Us Faster Learners and Worse Thinkers at the Same Time: the tension between completing better work and developing understanding.
• The Bottleneck Moved to Review: my shift from writing code to reviewing and verifying it.
• Ask, or It Will Guess: retaining responsibility for requirements, assumptions, and reasoning when working with agents.
• The GPU Was Already There: a case study in what measurement reveals even when implementation is largely automated.

References

• Terence Tao: Navier–Stokes and the value of searching for a solution, September 3, 2026; proof generation, verification, and digestion, April 22, 2026.
• Peter Naur: Programming as Theory Building, 1985.
• Anthropic: How AI Assistance Impacts the Formation of Coding Skills, January 29, 2026.
• Alberto Martin-Lopez and colleagues: More Code, Less Understanding? On the Impact of AI Assistants on Developers’ Productivity and Code Ownership, 2026.
• Margaret-Anne Storey: From Technical Debt to Cognitive and Intent Debt: Rethinking Software Health in the Age of AI, 2026.
• Nate Kornell, Matthew Jensen Hays, and Robert A. Bjork: Unsuccessful Retrieval Attempts Enhance Subsequent Learning, 2009.
• Michelene T. H. Chi and colleagues: Self-Explanations: How Students Study and Use Examples in Learning to Solve Problems, 1989.
• Su and Cardie: Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions, 2026.
• Anthropic: Building Effective Agents.
• Google SRE: Service Level Objectives, including latency distributions and percentiles.
• NVIDIA: Triton dynamic batching; PyTorch: Automatic Mixed Precision examples.
• LLVM: Register allocation; PostgreSQL: B-tree indexes, write-ahead logging.
• GitHub: GitHub Actions, code scanning.