I tried to make my laptop AI twice as fast. I found the wall instead.

Yesterday I got Meta's open model, Muse Glimmer, running on my laptop — private, offline, no bill. Today I tried to make it fast. This is a build log about walking into a wall, finding out exactly where it is, and why that was still a good day.

The starting point was 16.5 tokens a second. That's quicker than I read, so it's perfectly fine to talk to — but I wanted to know whether I was leaving speed on the table or genuinely hardware-locked. There's only one way to find out, which is to try to beat it.

First, the physics. Generating each token means reading all 18 GB of the model's weights out of memory. Multiply 16.5 tokens a second by 18 GB and you land at roughly 95% of my laptop's total memory bandwidth — the raw rate at which it can move data around. That's the tell: the bottleneck isn't sloppy software, it's that there's a hard ceiling on how fast bytes move, and I'm already brushing it. You cannot optimize your way past a bandwidth wall.

But there is one clever trick that sidesteps it: speculative decoding. You bolt a small, fast 'drafter' model onto the big one. The drafter cheaply guesses the next several tokens; the big model then checks all of them in a single pass instead of one at a time, keeps the guesses it agrees with, and throws away the rest. The output is identical — it's lossless — you just get more tokens out of each expensive memory read. Meta ships a drafter built for exactly this, called DFlash, and measured it running about 1.8x faster on Apple Silicon. That was the prize: roughly 30 tokens a second, for free.

So I went and got it. I pulled the model's GGUF build and the DFlash drafter, moved them onto the laptop, and ran them on llama.cpp — a different, very mature inference engine — with everything on the Mac's GPU.

The first result was the useful one, even though it looked boring. llama.cpp on its own, with no drafter, generated at about 17 tokens a second — the same number I already had. That is not a letdown; it is a confirmation. When two completely independent engines land on the same speed, you know you're measuring the hardware, not a quirk of one of them. The wall is real, and now I know exactly how tall it is.

Then the drafter. It loaded fine, but the current tooling couldn't wire it to the main model correctly — so the drafter ended up guessing blind, and the big model rejected almost everything it proposed. Here's the counterintuitive part: speculating and missing is slower than not speculating at all, because you still pay for every wrong guess. Instead of speeding up, it fell to 11.7 tokens a second. The feature exists in name; it isn't finished yet on this stack. The version that does work is Meta's own reference setup — a heavier build for another day.

So I left it at 16.5 and called it. Which sounds like defeat, except I walked away with three things I didn't have this morning: the exact ceiling, hard proof that it's bandwidth and not software, and the knowledge that the one door to a 2x — speculative decoding — is real but not yet open on my machine. You can't route around a wall until you've found it. Today I found it, and I mapped the door I'll walk through when the tooling catches up. Everything's staged to flip it on in an afternoon.

That's the quiet lesson, and it's an old one: a negative result is still a result. The hopeful benchmark that hides the wall is worth less than the honest one that shows you exactly where it stands. Meanwhile the model on my desk keeps answering, Wi-Fi off, costing nothing — which was the whole point before I ever went chasing the extra speed.