I Asked AI to Fix the Same Bug 40 Times

3 hours ago

6 min read

A composite account, assembled from the version of this story that arrives in our inbox most weeks. The details are changed. The shape never is.

I want to be upfront that this isn't one person's story. It's the story we get sent over and over, with different tools and different apps and remarkably consistent details. I've stitched it into a single account because the pattern is the useful part, and because nobody sends it to us until the end, when they're too worn down to describe the middle.

So here is the middle.

Week one: it works

The app is a client portal. Small agency, maybe eighty customers, each logging in to see their own projects, invoices and files. Built on Lovable with Supabase behind it, over about three weeks of evenings. It looks good — genuinely good, better than most agency portals that cost five figures.

He tests it constantly while building. Logs in, clicks around, everything correct. Shows two clients. They like it. He opens it to everyone.

For eleven days, nothing happens. Then an email: "Why am I seeing Rachel's invoices?"

The first fix

He can't reproduce it. He logs in as himself, sees his own data. Logs in as a test client, sees theirs. Everything is correct, every time he looks.

So he does what the tool has trained him to do for three weeks, and he describes the problem: users are sometimes seeing other users' data after logging in.

The AI responds confidently. It adds a filter to the dashboard query — explicitly filtering by the current user's ID. Reasonable. It even explains why this fixes it.

He deploys. Emails the client. Fixed.

Nine days later, someone else sees it.

Attempts two through eight

This is where the loop starts, and where it stops being obvious that it's a loop.

Each round has the same structure. A report arrives, he describes it again, the model produces something plausible and different. It adds a check on page load. It clears state on logout. It adds a loading guard so the dashboard doesn't render before the user object exists. It adds a key prop to force a re-render. Each change is sensible in isolation. Each one ships.

And each one appears to work, because the bug is intermittent. Two weeks of silence reads exactly like a fix.

This is the trap, and it's worth naming precisely: an intermittent bug cannot be disproven by not seeing it. Every fix gets confirmed by an absence. The loop has no way to fail, so it never stops.

By attempt eight the app has eight overlapping guards around the same area, none of which are addressing the cause, all of which are now part of the code that has to be reasoned about.

Attempts nine through twenty-two

The character of the thing changes here. He stops describing the symptom in business terms and starts pasting error output, session objects, snippets of code the model wrote weeks earlier and he doesn't recognize.

The model, now looking at a file with eight competing guards in it, starts producing fixes for problems introduced by previous fixes. Logout breaks — clicking it leaves you on a blank screen. Fixed. Then the loading spinner never stops on slow connections. Fixed. Then the dashboard flickers between two states on first load, which a client mentions, which is a genuinely new bug that didn't exist in week one.

Somewhere around fifteen, he starts prefixing his prompts with warnings. Don't change anything else. Only fix this. Do not touch the login flow. He's no longer building. He's containing.

He also stops opening two files entirely, because last time he asked for a change near them, three other things broke.

The bit nobody talks about

Around attempt twenty-five he stops telling anyone.

The clients who reported it think it's fixed. His business partner thinks it's fixed. It's easier to keep fixing it quietly than to explain that the app they've been demoing for two months has an intermittent data-leak bug and he doesn't know why.

This part shows up in almost every version of this story we receive, and it's the part that makes it expensive. The problem stops being technical and becomes a thing he's carrying alone. People describe it as embarrassing. It isn't — it's a structural property of the tool, not a personal failing — but that's not how it feels at 1am on the fourth weekend.

Attempts twenty-six through forty

Diminishing, increasingly desperate. He switches tools, pastes the codebase into a different AI, gets a fresh set of confident suggestions that don't work either. He rewrites the dashboard from scratch, which takes a weekend and changes nothing. He considers rebuilding the whole app and gets far enough to price it.

Attempt forty is not a fix. It's the point where he stops.

What it actually was

When someone finally reads the codebase — not the file with the bug, the whole thing, which is how every AI app rescue starts — it takes about forty minutes.

The Supabase client is created once, at module level, and shared. The dashboard's data fetching caches by query name, not by user. And session restoration is asynchronous, so on some page loads there's a brief window where the app has cached data from a previous session and a user object that hasn't resolved yet.

On a shared computer, or a browser where someone logged out and someone else logged in without a hard refresh — the agency's clients often had two people in one office — the dashboard renders the previous user's cached data for a moment before correcting itself. Long enough to see. Long enough to screenshot.

None of it lives in the dashboard file. The cause is spread across three files that were written in different weeks, by a model that saw each of them separately and never saw them together.

And this is exactly why forty prompts failed. Every prompt described the dashboard, so every fix landed in the dashboard. The bug was in the relationship between three files, and a relationship is not a place you can put a patch.

The fix, once identified, was small: key the cache by user, and hold rendering until the session resolves. Under an hour. The other four hours went to removing eight rounds of guards that were now doing more harm than good.

Why this is structural, not bad luck

Three things made this inevitable, and none of them are about anyone's competence.

The tool sees files, not systems. It works inside a context window — whatever it decided to look at, plus what you just said. It has no held model of how your app fits together. Ask about the dashboard and you get dashboard changes, because that's the world it can see.

Intermittent bugs break the feedback loop. The whole prompting model depends on you being able to evaluate the output. When the failure only appears under conditions you can't reproduce on demand, you can't tell a fix from a coincidence. You're guessing, with a very confident co-pilot.

Every failed fix makes the next one harder. Eight guards in a file is eight more things the next diagnosis has to account for. The problem gets worse as a direct function of how long you keep going.

That last one is why forty is so much worse than four. Not forty times the cost — considerably more, because a real share of the eventual fix is spent undoing the attempts.

The signal he missed

He told us afterwards that he knew around attempt three.

Not intellectually — he didn't have a framework for it — but something in him registered that this wasn't converging, and he kept going anyway because the next prompt was always cheaper than the alternative. Five minutes versus admitting he needed help. That calculation is correct thirty-nine times in a row and catastrophically wrong in aggregate.

If there's one thing worth taking from this, it's the threshold: the same bug, fixed, returning three times, means the loop has stopped working. Not that you're bad at prompting. That the cause is somewhere the loop can't see.

Two can be coincidence. Three is information. Forty is a story you tell later, a little too casually, to someone who has heard it before.

If you're somewhere between three and forty right now: the fastest way out is someone reading the whole codebase rather than the file you think the bug is in. Most causes of this kind are visible in an afternoon — and every attempt you add between now and then is a thing that has to be removed later.