The fix worked in dev and failed in prod. The code was identical.

The fix worked in dev. It failed in prod. Both were true at the same time. The code was identical. The running code was not.

We shipped a batch of fixes. In development, every one of them worked. In production, the same bug persisted as if nothing had changed. The source matched line for line.

The trap: the source on disk is not necessarily the code that is running. In our case, a background worker was still executing stale code. The fixes were applied to a process that was not the one doing the work. Everything looked correct. The running container was weeks behind the repo.

How I now diagnose “the fix does not reproduce on prod”

1. Stop debugging the logic. The logic might be fine.
2. Exec into the actual running container and read the actual file. Not the repo, the container.
3. Check whether the deploy actually deployed. Did the right service restart? Did the worker pick up new code? Is there a cache, a baked image, a second copy?
4. Verify the version that is serving traffic, not the version in git.

The deeper lesson

When a fix reproduces in dev but not in prod, your first hypothesis should be “the deployment did not happen the way I think it did,” not “my fix is wrong.” Confirm what is actually running before you touch the code again.

I lost hours re-debugging correct code. Do not be me.