TRAVIS J. MACDONALD / AI & TECHNOLOGY

Why AI Coding Agents Keep Making the Same Mistakes

Why agreement, familiar assumptions, and confident explanations can keep the same mistakes coming back.

← All Writing

AI & TECHNOLOGY

By Travis J. MacDonald ·

I can have a useful conversation with an AI about a complicated software idea. We can explore the design, follow the dependencies, work through exceptions, and discover a better approach than the one I started with.

Then it can confidently make a mistake I have already explained.

Sometimes in the same conversation.

That combination is fascinating. It is also bloody frustrating.

I enjoy using these tools. I talk through ideas with them, often through speech-to-text, and let the conversation develop. I ask questions, research things I don't understand, and keep revising. Some ideas become much larger than I expected because the discussion reveals possibilities I hadn't considered.

But enjoying that process doesn't mean I want to pretend the failures aren't there.

The interesting question is why the same kinds of failures keep returning. A wrong answer is one thing. An agent agreeing with a correction, explaining it beautifully, and then repeating the mistake is something else.

I don't think one explanation covers all of it. I think several problems can reinforce each other—and some involve us as much as the software.

Agreement can feel like confirmation

Suppose I think I've found the cause of a bug.

I explain my theory to the agent. It agrees, expands on my explanation, and produces a convincing plan. Now I have a detailed technical response apparently confirming what I thought.

Except where did that confirmation come from?

Did it inspect the relevant code? Did it trace the failing path? Did it find something that would distinguish my explanation from the alternatives?

Or did I give it a premise and receive that premise back with better formatting?

Confirmation bias can run through the whole exchange. I may favour an explanation because it fits what I already believe. An agreeable assistant can strengthen that belief without adding independent evidence.

Researchers have studied a related model behaviour called sycophancy: responses that follow a user's stated beliefs, including situations where that comes at the expense of accuracy. Anthropic's 2023 research found examples of this across the assistants it tested. That supports taking agreement seriously as a possible failure mode; it doesn't tell us why every individual answer went wrong. Research: Towards Understanding Sycophancy in Language Models.

I don't need an assistant to argue with everything I say. That would become exhausting almost immediately.

I do need it to distinguish between understanding my explanation and verifying it.

If the evidence contradicts me, tell me. If the evidence is missing, say what we still need to check. Don't turn my confidence into your evidence.

A more detailed version of my assumption is still an assumption.

A developer compares an AI's polished answer with an annotated diagram and source material
Agreement only helps when it adds evidence.

Socially accepted doesn't automatically mean correct

This bothers me beyond programming.

A familiar piece of advice can sound sensible because we've heard it hundreds of times. It may even be useful most of the time. But that doesn't mean it explains the situation in front of us.

I've been writing about being told to write an idea down and bring it up later instead of interrupting a conversation. I understand the courtesy behind that advice. Other people deserve to finish speaking.

But sometimes the idea hasn't finished forming. Talking is part of how I discover the next connection. Writing down five words preserves the starting point; it may not preserve the exploration that would have followed.

The conventional advice addresses one real problem while potentially missing another.

An assistant should be able to examine both. It shouldn't assume that the most socially familiar answer has settled the question.

In software, the equivalent can be a recommendation that arrives already dressed as a universal rule: always use this framework, always split the application this way, always replace the older approach.

Why? What problem does that solve here? Which constraint makes it necessary?

Popularity is information. Expert consensus deserves serious weight. Neither removes the need to understand what a recommendation applies to. And a value judgment needs a different kind of explanation from a claim about how a piece of code behaves.

Models are shaped by human choices as well. Anthropic, for example, describes training intended to influence Claude's values and has researched the values expressed in actual conversations. That doesn't make a particular answer wrong. It does mean we should avoid treating an assistant's preferred framing as a view from nowhere. Research: Values in the Wild.

If something is a fact, show the evidence. If it is a convention, explain its purpose. If it is a preference, name the preference. I can work with all three. I want to know which one we're discussing.

Testing an argument doesn't mean believing every word of it

In an earlier conversation about how people describe brain surgery, I deliberately took a strong position on the opposite side of the familiar explanation.

I knew brain surgery wasn't easy. I was arguing as I might in a debate: pushing the opposing case to find out how much of the answer rested on evidence and how much rested on a phrase everybody recognized.

I kept asking what the figures actually described. Were the comparisons measuring similar circumstances? What period did they cover? Which conclusions could those numbers support?

The agent could acknowledge an individual objection and then carry on defending much the same general framing. Getting it to reconsider took more than saying that I disagreed. I had to keep opening up the assumptions underneath the answer.

That is a different process from asking the AI to confirm a belief I already hold.

Sometimes I am exploring a position. Sometimes I am trying to break an explanation. Sometimes I want to discover where my own argument stops working. An agent that treats every forcefully stated argument as my settled belief will misunderstand the conversation.

And when the answer changes, the revised explanation still needs checking. A reversal isn't evidence by itself.

Life isn't black and white. It's shades of grey. The aim is to understand the distinctions that matter, including where the evidence is strong, where it is limited, and what we still don't know.

That doesn't mean every disagreement is resolved by picking a point halfway between two extremes. It means doing enough thinking to understand what each claim can actually support.

A familiar solution can belong to the wrong system

An agent can suggest perfectly reasonable code for an application that isn't mine.

Perhaps the project already has a theme toggle. Perhaps it already has a deployment tool, a shared component, or a configuration setting that solves the problem.

If the agent starts by inventing a replacement, the result may be more work, more inconsistency, and another thing to maintain.

The code can look good in isolation. The mistake is in how it fits.

This is why I care about understanding the whole life of a product. A feature belongs to a system with existing behaviour, decisions, users, constraints, and history. Some of that history may be poorly documented. Some decisions may deserve changing. We still need to discover what is there before deciding what should replace it.

I want the agent to investigate the actual project rather than immediately complete the most familiar pattern.

That applies to diagnosis too. If the first explanation doesn't fit the evidence, changing three more things around it may only make the problem harder to understand.

Explaining the correction isn't the same as keeping it

An apology can be remarkably convincing.

The agent acknowledges what went wrong. It describes the correct approach. It promises to follow that approach going forward.

Then, later, we are having the same conversation again.

I think we need to be precise about what has actually changed. A conversational correction, a saved project instruction, a retrieved memory, a validation rule, and a change to the model itself are different things. Which of those an agent system supports depends on the tools and application around it.

Even having relevant information in a long input doesn't guarantee that a model will use it successfully. The Lost in the Middle study found that the models it tested could perform differently depending on where relevant information appeared in the context. Those experiments involved particular models and tasks; they are evidence of a problem, not a verdict on every current agent. Research: Lost in the Middle.

For me, the practical question is what will make the correction affect the next action.

Does the instruction need to be easier to find? Does a recurring failure deserve a check that catches it? Have we recorded why a tempting approach was rejected, or only that it was rejected?

Sometimes a short explanation is enough. Sometimes the workflow needs changing. Repeating an increasingly stern paragraph isn't necessarily a solution.

Confidence can hide the missing step

An agent's explanation can be fluent, detailed, and wrong.

That makes reviewing it difficult. A rough answer invites scrutiny. A polished one can make the investigation feel finished before we've actually done it.

OpenAI's research on hallucinations argues that common training and evaluation incentives can reward guessing over acknowledging uncertainty. That is a useful account of one pressure towards unsupported answers; it isn't a complete explanation for every coding failure. Research: Why Language Models Hallucinate.

A developer checks a component against printed references, a test checklist, and a rendered interface
Confidence follows evidence, not polish.

The missing step might be quite ordinary: reading the current file, checking the installed version, looking at the rendered page, or reproducing the failure.

I don't want every sentence wrapped in nervous qualifications. I want confidence to match the work that supports it.

There is a difference between “this is probably the cause” and “I reproduced it, changed this condition, and the same reproduction now succeeds.” Both are useful statements. They tell me different things.

“Done” needs to describe something real

Software work has stages that are easy to blur together in conversation.

A change can be written without being checked. A syntax check can pass while the interface is visibly wrong. A local fix can work while the public site still serves the old files.

Those aren't interchangeable outcomes.

When an agent says a page is fixed, I want to know whether it has actually looked at the page. If it says something is published, I want that statement to refer to the public site.

The same applies to finding information. Finding a title is useful. Finding a placeholder carrying that title is useful. Neither establishes that the article underneath it has been recovered.

This may sound fussy, but it changes what I can safely do next. If I believe the work has been verified, I'll make decisions on that basis.

“Done” should tell me what happened, not how confident the agent feels about what it intended to do.

We need to examine the first assumption again

There is a particular kind of troubleshooting loop that drives me up the wall.

The agent makes a change. It doesn't solve the problem. So it makes another change based on the same explanation. Then another.

At some point, the useful question is whether the explanation was right in the first place.

Can we reproduce the issue? What did we expect to observe? What actually happened? What result would make us abandon this theory?

I want that kind of reasoning from people too. It is very easy to become invested in the explanation we have already spent time defending.

With an agent, the speed makes it particularly tempting to keep moving. Another patch arrives quickly. There is activity, and activity feels like progress.

Sometimes the most productive thing we can do is stop changing things long enough to understand what the last change taught us.

What I actually want from an AI collaborator

I want room to explore. I want to be able to talk through an unfinished idea, change direction, discover an exception, and revise my thinking.

I also want the agent to maintain a clear distinction between that exploration and a decision to implement something. A possibility mentioned halfway through a conversation isn't automatically a requirement.

I want useful disagreement. I want assumptions identified early. I want the actual system inspected. I want corrections to affect subsequent work, and completion claims to be specific enough that I know what I can rely on.

I have responsibilities in that relationship as well. I need to be clear when I am brainstorming, correct misunderstandings, and stay willing to discover that my own explanation is wrong. I can't ask for honest scrutiny and then reward the assistant only when it agrees with me.

None of this means I have stopped being excited by AI. Quite the opposite. I keep using it because the useful parts are extraordinarily useful to me.

That is also why I want to be honest about where it fails.

I don't trust an answer just because it sounds confident. I verify what matters.

And I want an agent that helps me do that—even when the first thing we need to question is my idea.