A 2.8-trillion-parameter model fits in 8 GB of RAM. It writes one token every 32 seconds.

Two projects showed up this week doing the same unlikely thing. Colibri runs GLM-5.2, which has 744 billion parameters, on a consumer laptop. A project called kimi-k3-in-c runs Kimi K3, at 2.8 trillion parameters, on a single CPU in 8 GB of RAM. The whole engine is 176 KB of C. No GPU, no BLAS, no PyTorch. It depends on libm and OpenMP and nothing else.

I went looking for the catch. It is not hidden and the author does not bury it. At the 8 GB setting the engine produces about one token every 32 seconds. Call it 0.03 tokens per second. A hundred-token answer takes most of an hour.

My first reaction was the obvious one. What is the point of this. I think that reaction is wrong, though not for the reason I expected, and working through the arithmetic changed how I think about what a model costs to run.

Capacity stopped being a memory problem

The old rule for local inference was simple. Weights have to fit in memory. If the model does not fit in your RAM or your VRAM you cannot run it, and that made "can I run this" a hardware question with a yes or no answer.

Both engines break the rule the same way. Mixture-of-Experts models only fire a small fraction of their weights per token. K3 activates roughly 104 billion of its 2.8 trillion parameters for any given token. GLM-5.2 activates about 40 billion of 744 billion. The weights are enormous. The working set is not.

So you stop holding the model in memory and start holding it on disk, paging in the parts that fire. kimi-k3-in-c packs the 93 dense layers into a single 109 GB file addressable at known byte offsets. The routed experts ship pre-quantized at four bits, half a byte per parameter, and get multiplied without ever being expanded back out. A per-layer LRU cache keeps whichever experts the router keeps picking. Colibri does the same shape of thing to GLM-5.2, with the dense parts pinned in RAM and 19,456 routed experts streamed off SSD with async readahead.

People reach for Apple's unified memory as the analogy and it is the wrong one. Apple's UMA is hardware: one physical pool, uniform latency, and the copy does not exist. This is demand paging. Virtual memory, applied to experts instead of process pages. Apple eliminates the transfer. These engines hide it. Capacity comes from NVMe and you pay for it in read latency.

Which means the claim worth caring about is not "744B on a laptop" or "2.8T in 8 GB". It is that capacity became a storage problem instead of a memory problem, and storage runs about two orders of magnitude cheaper per gigabyte than HBM.

Why it is slow, in one line of arithmetic

The speed collapse is worth working out by hand, because it tells you what would and would not fix it.

K3 activates about 104B parameters per token. At four bits that is roughly 52 GB of weights that have to be read for every token generated. The engine sustains about 6 GB/s off NVMe using O_DIRECT. Divide one by the other and you get about 8.7 seconds per token as a floor.

The measured figure on the fastest preset is 10.69 seconds per token. The arithmetic closes. This is not a badly written implementation. It is a memory-bandwidth problem wearing an inference costume, and the disk is the wall. The published benchmarks agree: I/O eats between 41% and 71% of wall clock across the presets.

One detail surprised me. O_DIRECT reads cold beat the buffered path warm on the author's machine, 3.2 GB/s against 2.3 GB/s. When you stream tens of gigabytes per token the page cache stops helping and turns into an extra copy.

C is not the slow part

I want to be careful here, because "written in C" and "slow" briefly welded together in my head and they should not.

llama.cpp is also C and C++. It also carries no framework dependencies. It runs a 7B model on a MacBook somewhere between 30 and 100 tokens per second, and it is the engine under Ollama, LM Studio, and most local AI anyone actually uses. Same language, same philosophy about dependencies, completely usable.

Rewrite kimi-k3-in-c in Rust, in CUDA, or in hand-tuned assembly and you get the same 0.03 tokens per second, because none of those make an SSD faster. The bottleneck is physical, not linguistic. What makes this project slow is what it attempts, a 2.8T model that does not fit, and not the language it is written in.

What this looks like on hardware I own

I ran into a smaller version of all this the week before, and it is the reason the arithmetic above landed the way it did.

There is a Dell Precision 7920 in my lab, a dual-socket Xeon Gold 6148 box. I gave a VM on it 20 vCPU and 24 GB of RAM and pointed Ollama at it. The first real test was gpt-oss:20b, a 20B MoE with about 3.6B active, roughly 13 GB on disk. It loaded. It answered. It ran at about 3 to 4 tokens per second, and I did what everyone does at that point, which is start thinking about what to buy.

That was the wrong move. I ran a thread sweep first, and the numbers went the opposite direction from what I expected.

qwen2.5-coder:7b      4 thr   5.3 tok/s
                      8 thr   7.8
                     12 thr   9.0
                     20 thr   5.1

gpt-oss:20b (MoE)     8 thr   8.3 tok/s
                     12 thr   9.65
                     16 thr   9.2

Twelve threads beat twenty by roughly a factor of two. Giving the job more cores made it slower. The default, which is to use all 20 vCPU, was producing 3.1 to 4.5 tok/s, so capping the thread count was a free 2x to 3x that cost nothing and required no new hardware.

The reason is the same reason the C engine is slow, scaled down by three orders of magnitude. The VM was configured with a single NUMA node on a host that physically has two, so the guest allocated memory without knowing it was split across both sockets, and a large share of every read crossed the interconnect between them. Past about twelve threads, memory contention grew faster than the extra compute could pay for it. The DIMMs also run at 2133 MT/s rather than the 2666 the CPU supports, which does not help. The host itself was not busy: 58 of 80 vCPU allocated, load average around 13, governor set to performance. Compute was never the constraint. Memory bandwidth was.

The second thing I saw was this article's thesis in miniature. Three identical runs of gpt-oss:20b, back to back, gave 4.55, then 3.85, then 3.14 tokens per second. It got slower every single time it ran. A 13 GB model in 24 GB of RAM leaves nothing for anything else, so it started swapping, and swap is just disk with extra steps. That is precisely the failure the paging engines are built around, except they schedule the disk access on purpose instead of discovering it by accident.

So: a working model, slow tokens, and every instinct I had about the cause was wrong. Not the CPU. Not the core count. Not the model. The layout of the memory underneath it.

What a GPU does and does not buy

The obvious next move is to bolt a GPU on. I sized it out and the answer is less than you would hope.

A GPU's value in this job is memory bandwidth, not FLOPs. Single-stream decoding uses each weight once per token, an arithmetic intensity of about one, so the tensor cores sit idle no matter what you do. The GPU only accelerates weights that physically live in its HBM. That leaves three options.

• Put the dense trunk and attention on the GPU and leave the experts on CPU and disk. This is the pattern llama.cpp exposes as --n-cpu-moe. It accelerates the part that was never the problem. Call it 1.1x to 1.3x.
• Use the GPU as a hot-expert cache. An 80 GB card holds about 5.5% of a 1.45 TB expert pool, and routing is skewed enough that you might reach a 20% hit rate. Call it 1.25x.
• Speculative decoding, where a small draft model on the GPU proposes several tokens and the large model verifies them in one pass, so a single weight read serves multiple tokens. This is the real trick, worth maybe 1.5x to 2.5x, though MoE routing eats into it because verifying more tokens touches more experts.

Stack all three optimistically and you land near 2.5x. One token every four seconds instead of every eleven. Better, and still not something you can hold a conversation with.

The gap you are trying to close sits between NVMe at 6 GB/s and HBM at 3,350 to 8,000 GB/s. That is a factor of five hundred to thirteen hundred. You do not close it by adding one GPU to a disk-bound system. You close it by putting all 1.45 TB in HBM, which takes roughly eight B200s, and at that point you have left consumer hardware so far behind that the premise is gone.

There is no middle configuration. The model is either fully resident or it is slow.

Compatibility: what it actually needs

For anyone who wants to try it, the requirements are specific and Linux only.

• Linux x86-64 with AVX2 and FMA. AVX-512 is optional, and the published benchmarks ran without it.
• 8 GB of RAM at the floor preset. The ladder runs up to about 224 GB if you have it.
• Roughly 1.7 TB of free storage, and it needs to be fast local NVMe. Network storage will not do.
• GCC 9 or newer, or Clang 10 or newer, plus GNU Make or CMake. The engine depends on libm and OpenMP and nothing else.
• Python 3.9 or newer for the tooling only, not for the C engine, plus a Hugging Face token to pull the checkpoint.
• No GPU support at all, which is a choice rather than an omission.

One good decision: you can validate the whole engine without downloading 1.5 TB of anything. Building and running the test suite checks it against a committed 13-layer reference model in under a minute. More projects should ship that path.

git clone https://github.com/FareedKhan-dev/kimi-k3-in-c.git
cd kimi-k3-in-c
make -j && make test

The claim I find most interesting is that all five memory presets produce byte-identical output. Hand it 8 GB or hand it 224 GB and the tokens are the same tokens. Memory is a latency dial, not a quality dial. Behind that sits a three-gate validation suite covering teacher forcing, greedy decoding, and incremental generation with cache, all checked against PyTorch fixtures committed to the repo.

So what is it actually for

Not serving. 0.03 tokens per second is not slow but workable. It is slow enough that overnight batch work does not pencil out either, and I would not plan anything around running this particular engine.

What it is for comes down to four things.

• A readable reference. Karpathy's llama2.c is about 700 lines and nobody runs production on it. The value is that you can read the whole thing in an afternoon and come out understanding transformer inference, with no framework abstractions between you and the arithmetic. 176 KB of C sits in that tradition.
• Establishing that the floor exists. "This is possible at all" changes what people try next. llama.cpp started as a slow weekend project and got orders of magnitude faster over two years of community optimization. The first version being slow is the normal shape of this.
• Deployment with no supply chain. No Python, no CUDA, no PyTorch, no dependency tree with several hundred packages in it.
• Auditability. You can read 176 KB of C. You cannot meaningfully audit PyTorch plus CUDA plus a serving framework plus everything those pull in behind them.

Those last two stop being aesthetic preferences the moment you work somewhere that has to answer for what is running.

Where this stops being a curiosity

I spend my working life around VMware Cloud Foundation, and that is the lens where this class of work starts to look like something other than a stunt.

Start with the air gap. In a disconnected estate the alternative to a slow local model is not a fast local model. It is no frontier model at all, or a GPU cluster that arrives with export controls, lead times, and a power and cooling conversation with facilities. Against that baseline, slow and present beats fast and unavailable for anything asynchronous. Overnight document processing, batch classification, offline evaluation. Nobody sits and watches those type.

In those environments the dependency story matters more than the speed story. Getting PyTorch plus CUDA plus a serving framework through a security review means accounting for a tree with hundreds of packages in it, and doing it again at every upgrade. A single binary with libm and OpenMP behind it is a different conversation. I have watched teams spend more calendar time on that paperwork than on the deployment it was gating.

Then there is the refresh cycle. The standard assumption is that a host without GPUs has no AI future, so it goes to the secondary market or gets recycled. If capacity comes from storage instead of HBM, a decommissioned box with a lot of RAM and a lot of NVMe is not useless for MoE inference. It is slow, and slow costs close to nothing when the hardware is already paid for and already racked. That is worth thirty minutes of thought before it leaves the building.

It cuts the other way on the buy side. If you are speccing a refresh with any of this in mind, the line items that matter are NVMe bandwidth and memory channels, not only GPU count. Those are cheaper lines, and they serve the rest of the platform whether or not the inference plan works out.

On vSphere specifically, two things decide whether this performs at all. The first is the storage path. When I/O is 41% to 71% of wall clock, the choice between local NVMe, vSAN, and anything network attached is the entire performance story, and it dwarfs every other tuning decision you might make.

The second is NUMA, which is the lesson from my own box further up. Size the VM to the NUMA node before you size it to the core count, and check what the guest thinks its topology is rather than what the host knows it to be. The benchmarks in the repo ran on a dual-socket EPYC, so the same trap is sitting in the same place for anyone who tries to reproduce them.

None of this makes kimi-k3-in-c a production workload. It makes the technique a planning input, which is a smaller claim and a more useful one.

The counter-risk

Part of this genre is a flex. "X in C" is a recognized format now, half engineering contribution and half portfolio piece, and it would be dishonest to pretend the performance numbers are anything but bad. A 2.8T model at one token per 32 seconds is a demonstration, not a tool. If you need something to use this month, use llama.cpp with a model that fits in your RAM.

The transferable part is the technique, not the binary. Four-bit experts multiplied without dequantizing. Weight files addressable at known offsets, so residency becomes a runtime parameter. Memory as a dial with output invariance underneath it. Those ideas work just as well inside a fast engine, and that is where I expect them to turn up.

What I am watching

Two things decide whether this line of work matters or stays a curiosity.

First, whether residency as a dial gets absorbed into the engines people actually run. llama.cpp already does partial offload, and the step from there to "declare a memory budget, get byte-identical output, pay the difference in latency" is small and useful.

Second, whether the hardware moves underneath it. The whole approach bets that storage bandwidth is cheap and memory capacity is expensive. If large unified-memory machines keep getting cheaper, which is the direction Apple pushed and everyone is now copying, the paging trick matters less. If they do not, streaming from NVMe is how most people end up running frontier models, and these two projects are early sketches of what that looks like.

I would like to run kimi-k3-in-c properly, but 1.7 TB of fast local NVMe is a real commitment for something I already know emits a token every 32 seconds. The build and test path needs no checkpoint at all, and that one I will do.