Self-host or external API? The real numbers said 5.4x

Self-host or external API? I ran the real numbers. The gap was 5.4x. Here is exactly how I got there.

Someone on the team mentioned, casually, that an external API would be “easier to maintain” than our self-hosted model. Probably true operationally. But nobody had run the numbers on what “easier” would cost at scale.

Real token budgets, not estimates

So I did, and I refused to estimate. I extracted the real token budgets from our actual codebase: every endpoint’s system-prompt length, conversation depth, and output budget. Real usage, not blog-post averages.

The result:

1. Self-hosted on our own GPU: a fixed monthly cost, flat regardless of volume.
2. External API at 10,000 users: roughly 5.4x more expensive.
3. Break-even sat below 2,000 monthly active users, far under our trajectory.
4. Per-call AI cost on our own infrastructure: effectively zero, at any volume.

I packaged it as a short executive brief plus the full methodology, so a non-technical leader could verify every number.

Costing the system is the ML engineer’s job

The infrastructure decision got made with the right numbers instead of a vibe. That is the real point: financial analysis of ML infrastructure is the ML engineer’s job, not finance’s. You are the only person who knows what the system actually costs to run. If you do not quantify it, the decision gets made on someone else’s guess.