Restart vs recreate: how one config value nearly took down production

I almost took down production by raising a single config value. The danger was not the value. It was the word “recreate.”

I needed to raise one serving parameter. The catch: that parameter cannot change at runtime. The container has to be recreated with the new value.

In Docker, restart and recreate are not the same thing, and confusing them is how you take down routing.

1. Restart keeps the container and its internal IP.
2. Recreate gives the container a new internal IP.

If your routing, say an nginx upstream, cached the old IP, traffic keeps flowing to an address that is now a dead or different container. Silent breakage until something restarts.

The zero-downtime sequence

1. Confirmed nginx was using a DNS resolver with dynamic upstream resolution, so it re-resolves the address instead of caching it. I had hardened this after an earlier incident.
2. Updated the config value.
3. Recreated only that container, not the whole stack.
4. Verified the network alias resolved to the new IP.
5. Confirmed the published port still responded.
6. Ran the full regression harness before declaring victory.

The lesson for anyone running Docker with nginx: if your routing depends on container IPs, even indirectly, you must know which operations reassign them before you run them. Use DNS-based resolution so a recreate does not strand your traffic.