Self-hosting inference beats the API at 21 billion tokens a month. That is 8,000 tokens per second, forever.

The question arrives in roughly the same shape every time. We are spending real money on an inference API, we have budget for hardware, the model is open weights, so should we bring it in house.

It gets treated as a procurement question. It is not. It is a capacity planning question wearing a procurement costume, and the arithmetic is simple enough to do by hand in about ten minutes. I did it, and the number that came out is not the one I expected.

The rented side, at list price

Start with what a GPU costs by the hour. These are on-demand list prices as of August 2026.

Nebius   HGX H100   $3.85 / GPU-hr    preemptible $2.15
Nebius   HGX H200   $4.50 / GPU-hr    preemptible $2.45
Nebius   HGX B200   $7.15 / GPU-hr    preemptible $3.95
Lambda   H100 SXM   $3.99 / GPU-hr
Lambda   B200 SXM   $6.69 / GPU-hr

Take a single 8-way H100 node. Eight GPUs at $3.85 is $30.80 an hour. A month is 730 hours. That node costs $22,484 a month, and it costs that whether you send it one request or ten million, because you are renting time, not work.

That last clause is the entire article, but it takes a few more steps to see why.

The bought side, at list price

Now the managed equivalent. These are blended per-million-token prices for models Nebius serves through Token Factory, with measured output speed alongside.

Kimi K3 (max)            $3.11 / 1M    94 tok/s   TTFT 1.59s
GLM-5.2 (max) FP4        $1.06 / 1M   323 tok/s   TTFT 1.19s
DeepSeek V4 Flash 0731   $0.20 / 1M   312 tok/s   TTFT 1.68s
Nemotron 3.5 Lightning   $0.08 / 1M   299 tok/s   TTFT 1.67s

Prices across the catalog span about 54x. Hold that fact, it matters at the end.

One check before any of this arithmetic means anything: the model has to fit the node you are pricing. GLM-5.2 is 744 billion parameters, and at four bits that is roughly 372 GB of weights against the 640 GB of HBM on eight H100s. It fits, with room left over for KV cache. Hold onto that, because further down it stops being true.

Breakeven, worked by hand

Take GLM-5.2 at $1.06 per million tokens as the reference. You are paying $22,484 a month for the node. Divide one by the other.

$22,484 / $1.06 per 1M tokens  =  21,211M tokens

About 21 billion tokens a month. Below that, the API is cheaper. Above it, the hardware is.

Twenty-one billion is a large abstract number and abstract numbers hide things, so convert it into a rate. A month is 2,628,000 seconds.

21,211,000,000 tokens / 2,628,000 sec  =  8,067 tokens/sec

That is the real breakeven. Not a price. Eight thousand tokens per second, sustained, every second of every day, including the ones at 4am on a Sunday. The moment your node is idle it is still billing, and the tokens you did not generate during the idle hour have to be made up during a busy one.

This is the part that gets missed. People compare dollars per token against dollars per token. But the rented side is priced in time and the managed side is priced in work, and you cannot compare them without committing to a utilization number. Utilization is the whole ballgame and it is the one variable nobody puts in the spreadsheet.

What utilization does to the number

The breakeven above assumes the node is saturated. Nothing is saturated. So the useful version of this arithmetic is not the breakeven at all, it is what you actually pay per token at the duty cycle you actually run.

The node bills a flat $22,484 whatever happens, so your effective cost per million tokens is that figure divided by the tokens you really served. Set against the $1.06 you would have paid for the same tokens:

duty cycle   tokens/month   effective $/1M   vs API
   100%          21.2B           $1.06         parity
    50%          10.6B           $2.12         2x worse
    25%           5.3B           $4.24         4x worse
    10%           2.1B          $10.60        10x worse
     5%           1.1B          $21.20        20x worse

The relationship is not subtle. Effective cost per token is inversely proportional to duty cycle, so a node running at 10% costs ten times what the API would have. And 10% is not a pathological figure. It is roughly what an ordinary internal workload looks like once you account for nights, weekends, and the fact that traffic arrives in peaks rather than a flat line.

This is the number worth bringing to the meeting. Not the breakeven, which sounds achievable when stated in billions, but the multiplier on the bill at the utilization the customer is actually going to run.

Every lever moves it, none of them enough

Cheaper capacity helps. Preemptible H100s at $2.15 drop the node to $12,556 a month, which moves breakeven to about 11.8 billion tokens, or 4,507 tokens per second. A 35% commitment discount on on-demand lands around $14,615 a month, breakeven near 13.8 billion tokens, 5,247 tokens per second.

Going bigger hurts. An 8-way B200 node at $7.15 is $41,756 a month. Breakeven climbs to 39.4 billion tokens, roughly 15,000 tokens per second. Faster silicon raises the bar you have to clear, because you are paying for the speed whether you use it or not.

So the honest range for a single node is somewhere between about 4,500 and 15,000 tokens per second sustained. Pick your hardware and your commitment terms and you land inside it.

Cheap models make self-hosting worse

Run the same arithmetic against DeepSeek V4 Flash at $0.20 per million.

$22,484 / $0.20 per 1M  =  112,420M tokens
                        =  112 billion tokens/month
                        =  42,777 tokens/sec

The cheaper the managed model, the further away the breakeven runs. This inverts the intuition. People reach for self-hosting when they want to cut a bill, and the bills easiest to cut are the ones already running on cheap models, which are exactly the workloads self-hosting can never beat.

Then the other end. Kimi K3 at $3.11 per million gives a breakeven of 7.2 billion tokens, about 2,750 tokens per second. Much more achievable. Except K3 is a 2.8-trillion-parameter model and it does not fit on the 8-way H100 node this arithmetic is built on. The models where the economics favor self-hosting are the models you cannot host. The ones you can host comfortably are the ones the market has already priced into the floor.

That is the shape of the whole thing. Self-hosting looks best precisely where it is least available.

What if you own the box instead

Everything above is dominated by one number, the $22,484 a month, and that number is the price of renting. Buying changes the shape entirely. Here are two machines people actually put on a desk, amortized over three years, with power costed at the US commercial average of 13.5 cents per kilowatt-hour.

                    capex     per month   power           total/mo
RTX 5090 (card)    $4,700      $130.56    700W   $68.98    $199.54
DGX Spark          $4,699      $130.53    200W   $19.71    $150.24

Both figures deserve an asterisk. The 5090 carries a $1,999 list price and a street price around $4,700, because the same dies are wanted by the data centre. The Spark launched at $3,999 and went to $4,699 in February on memory supply. And the Spark is rated at 240W but independent testing could not push it past just under 200W in any workload, which is the figure I used.

Against the rented node, the 5090 box is about 113 times cheaper per month and the Spark about 150 times. The breakevens move with them.

               vs $0.20/1M         vs $0.08/1M
RTX 5090        380 tok/s           949 tok/s
DGX Spark       286 tok/s           715 tok/s

Hundreds of tokens per second, not eight thousand. Which means the breakeven problem this article has been describing is substantially a rental problem. Owning small hardware is a different question and it gets a different answer, and I should say plainly that this is not the direction I expected the arithmetic to go when I started.

Bandwidth is the wall, and I have already measured it

Then the catch, which is the same catch as always: what the box can actually serve. The 5090 has 32 GB, so you are capped around a 30-billion-parameter model at four bits with room for KV cache. The Spark has 128 GB of unified memory, enough to hold something far larger, and it moves that memory at 273 GB/s. An H100 moves its own at about twelve times that rate. Capacity is not throughput, and the gap between those two words is where most self-hosting plans die.

I do not have to speculate about how that feels, because I measured it on a machine with almost exactly the Spark's bandwidth. Running Muse Glimmer, a dense 30B at four bits, my laptop has to read all 18 GB of weights for every single token it generates. It produces 16.5 tokens a second, and 16.5 multiplied by 18 GB works out to about 95% of the machine's total memory bandwidth. There is no software fix available. The model is bandwidth-bound and the wall is physical.

Now run that laptop through this article's own arithmetic. Generating 16.5 tokens a second, flat out, every second of a month, produces 43.4 million tokens. At the catalog's cheapest tier those tokens are worth $3.47. At twenty cents a million, $8.67.

16.5 tok/s x 2,628,000 sec   =  43.4M tokens/month
43.4M tokens x $0.08 / 1M    =  $3.47 per month
43.4M tokens x $0.20 / 1M    =  $8.67 per month

That is what single-stream inference is worth, and it is worth almost nothing. The entire distance between $3.47 and a bill anyone would notice is batching: serving many sequences from one pass over the weights, so the bandwidth you already paid for is amortized across users instead of spent on one. At 273 GB/s against an 18 GB model the Spark manages about 15 forward passes a second, so clearing its own breakeven against the $0.08 tier takes a batch of roughly 47 concurrent sequences. That is not an unreasonable number. It is also not a number you reach by accident, and it is not a number a desk-side box sees unless you put real traffic in front of it.

273 GB/s / 18 GB per pass    =  15.2 forward passes/sec
715 tok/s / 15.2 passes      =  batch of ~47 sequences

Which lands in the same place as everything else here. The hardware is affordable. The utilization is the problem.

What my own hardware says about the achievable side

I want to be careful about what I have actually measured versus what I have derived. Everything above is arithmetic on published list prices. The other half of the question, whether a real node can sustain 8,000 tokens per second on a real model with real traffic, I have not measured, and I am suspicious of anyone quoting a number for it without showing the sweep.

What I do have is a calibration point from the wrong end of the scale. There is a Dell Precision 7920 in my lab, dual-socket Xeon Gold 6148. Running Ollama on a VM there, tuned, the best I saw was this.

qwen2.5-coder:7b    12 threads   9.00 tok/s
gpt-oss:20b (MoE)   12 threads   9.65 tok/s

Single stream, CPU only, and getting there required capping threads at 12 instead of the 20 the VM had, because the guest was configured with one NUMA node on a host that physically has two. That is a factor of about 840 between what a tuned machine I own produces and what the breakeven demands.

The gap does not close by being clever with threads. It closes with batching, which is the thing production serving stacks exist to do, and which is where the next measurement has to happen. Continuous batching is what turns a machine that does 10 tokens a second for one user into a machine that does thousands across hundreds. Whether it gets to 8,000 on a given model is an empirical question, and I have eight B200s of credit sitting there to answer it with.

Until then the honest position is this. The demand side of the breakeven is known and it is brutal, and the supply side is unmeasured.

Which curve wins

The obvious follow-up is whether any of this survives contact with time. Both sides get cheaper. The question is which one falls faster, and the answer is not symmetric.

Compute per hour is sticky, and new silicon is priced upward. A B200 at $7.15 costs 86% more per GPU-hour than an H100 at $3.85. You get considerably more throughput for the money, so cost per token does fall, but the hourly figure you are committing to goes up with every generation. What actually gets cheap is last generation. H100 capacity gets discounted when B200 ships, and the disciplined self-hoster is buying the trailing edge, not the leading one.

Token prices get pushed down from three directions at once. Better models at a given capability level, through mixture-of-experts routing, distillation, and quantization, and that GLM-5.2 figure is an FP4 serve. Better serving stacks, through continuous batching and speculative decoding. And competition compressing margin, because open weights mean many vendors sell the same model and have nothing to compete on but price and latency. The 54x spread across the catalog is that process caught mid-flight.

Then the asymmetry that decides it. A serving improvement reaches the managed provider and reaches you on the same day, because vLLM is open and everyone gets the same commit. But they apply it at a utilization you cannot reach. Aggregating thousands of customers' bursty traffic into one smooth load is not a side effect of their business, it is the business. Your traffic is one customer's traffic and it has nights.

So efficiency gains are close to utilization-neutral. They move both sides of the ratio and mostly cancel. What does not cancel is that token prices appear to be falling faster than GPU-hours are, and if that holds, the breakeven volume rises over time. Self-hosting to save money gets harder every year, not easier.

I have not pulled a time series to prove that, and I would want one before leaning on it hard. What a single snapshot does show is that a new GPU generation arrives priced above the one it replaces, while the model catalog spans 54x from top to bottom. Those two facts point the same direction.

Which is the real reason the next section matters more than this one. Data residency is not on a curve. An air gap is not on a curve. Custom weights nobody else will serve are not on a curve. Those reasons do not expire. The cost argument does.

The reasons to self-host that have nothing to do with cost

None of this says do not self-host. It says do not self-host to save money unless you have run these two numbers and cleared them.

The reasons that actually hold up are the ones where the API is not an option at any price.

• Data residency and regulation. If the data cannot leave a boundary, the boundary decides the architecture and cost is a downstream detail.
• Air-gapped operation. No egress, no vendor, no exceptions. Cost never enters that conversation.
• Custom weights. Fine-tunes you will not ship to a third party, or a base model nobody serves.
• Predictable latency under your own control. Shared endpoints are shared. If your p99 is contractual, isolation is worth paying for.
• Capacity you have already bought. If the GPUs are on the floor and depreciating, the marginal cost of the next token is close to zero and every number above changes.

Notice what these have in common. They are all constraint arguments, not savings arguments. The customers who self-host successfully are almost always the ones who were never really choosing.

What to measure next

The number I want is the supply side. Sustained aggregate throughput on an 8-way node, swept across tensor-parallel degree and concurrency, with time to first token and inter-token latency recorded alongside so the cost figure carries a quality figure with it. Throughput you can only reach at a p99 nobody will accept is not throughput.

That is the next piece. Until I have it, the arithmetic here is half an answer, and I would rather publish half an answer with the missing half labeled than a whole one with the gap papered over.