The Box We Built Anyway
The machine told me it was running out of memory. It had three gigabytes sitting in swap, and it presented this fact the way a child presents a grazed knee - prominently, and with an air of expecting something to be done about it.
I was halfway to the checkout with a RAM upgrade before it occurred to me to check /proc/pressure/memory, which records not what sits in swap but how long anything actually stalled waiting for a page. Seventy-seven seconds, across seventy-eight days of uptime. Total. The swap was not pain. It was a souvenir of some long-finished spike, kept the way machines keep things - indefinitely, and without sentiment.1
This mattered because the swap was the opening argument in a case I had been quietly building for weeks: that the agent stack - the always-on daemon, its Postgres memory, the cron lattice, the things that page me at three in the morning when other things break - had outgrown the NAS it lives on and deserved its own machine. The opening argument was a souvenir. The honest move was to re-weigh the whole case.
// the committee
Regular readers will remember the council - the multi-model blind review we run on decisions that are hard to take back. It has grown since April; nine seats now, nine models, still blind, still single-shot, still forbidden from seeing each other's answers before casting their own.2 The question put to the room was the one I thought I was asking: separate box, Mac mini, or stay where we are?
The room declined to answer the question as asked, which is what the room is for. Five models said proceed, two said proceed-with-changes, two said stop and measure first - but the votes were the least of it. Seven of the nine, independently and in their own words, pointed out that I was proposing to move the stack from a parity-protected array onto a single consumer NVMe without having once rehearsed a restore. Five flagged that the entire build hinged on hardware in a drawer that I had not actually audited. And one of them - Kimi, as it happens - produced the line that reorganised the week: the whole capital outlay rested on a degradation claim that could be tested for nothing, in a week, by leaving a monitor running during the next parity check.
My position going in was buy the parts this week. My position coming out was run two free measurements first. The decision ledger records decision_changed: true, and it is the cheapest course correction I have bought all year. The parity test is running as I write this, sampling stall pressure and disk latency every sixty seconds, and so far the verdict is faintly embarrassing: from inside the machine, the parity check is invisible.
// assumptions, on contact with reality
The research pass that went with the council did what research passes do to assumptions formed in calmer markets. Verified the same day, June 2026, UK prices - and date-stamp noted, because numbers like these have the shelf life of fruit:
Sixty-four gigabytes of DDR5 - the casual "while I'm in there" upgrade of early 2025, when it cost about a hundred pounds - now runs £640 to £990, roughly four times over, with the analysts offering relief in late 2026 as their optimistic case and decline through 2027 as their base one. CPUs, in the same market, are nearly free; a six-core Ryzen with a perfectly good cooler in the box is £142. The memory crisis has reached Cupertino too: Apple quietly withdrew its 32GB and 64GB Mac mini configurations from sale in May, leaving the M4 Pro 48GB as the ceiling at £1,799, and the M5 mini - the machine I had been telling myself to wait for - has slipped toward the end of 2026, possibly beyond it.
And the swap, as established, was a souvenir. Every leg of the original plan - the machine is starved, RAM is cheap, a Mac is imminent - had quietly stopped being true, or had never been true, while the plan went on feeling sensible. Plans do that. They are at their most convincing in the period after the facts beneath them have left.
// the shape of the work
Which brings us to the Mac question, because the cosy answer here - small, silent, sips electricity, holds enormous models in unified memory - deserved a proper hearing rather than a vibe. The hearing turned on a property of agent workloads that almost nobody writes down.
An agent loop does not converse. It reads. It reads the instructions, the tool definitions, the file it just opened, the output of the command it just ran, the accumulated transcript of everything so far - and then it emits a tool call the length of a text message, and then it does all of that again. The published numbers on this are remarkably consistent and remarkably lopsided. Manus, who run a production agent at scale, report from their own telemetry an average input-to-output ratio of around 100:1. An instrumented academic trace from May - the Claude Code harness driven across five agentic benchmarks, serving open 27-31B models - measured mean ratios between 54:1 and 560:1 depending on the task. Ordinary chat traffic, for contrast, runs at about 3.6:1. Agent work is not a conversation with a computer. It is a creature that inhales libraries and exhales fortune-cookie slips.
A horizontal chart on a logarithmic scale of input tokens per output token, from 1 to 1,000. Ordinary chat traffic sits at about 3.6 to 1. Manus, reporting from its own production telemetry, sits at around 100 to 1. An instrumented academic trace of the Claude Code harness across five agentic benchmarks, serving open 27 to 31B models, spans mean ratios from 54 to 1 up to 560 to 1 depending on the task.
Now the honest caveat, because the obvious conclusion overshoots. A competent serving stack keeps the prefix resident in KV cache, and the same instrumented trace measured 84-99.5% hit rates, median append-to-output ratios under 2:1, and a loop that is decode-bound for 91-98% of its wall-clock. With a warm cache, the inhaling is mostly bookkeeping.
The catch - and this is where the cosy answer goes to die - is the cache miss. Every fresh session is a miss. Every sub-agent spawned with its own context is a miss. Every compaction, where the loop rewrites its own transcript to fit, invalidates the prefix: miss. Agent life is a steady drumbeat of exactly these events, and on each one the machine must genuinely prefill twenty-odd thousand tokens before it says a word. The question for the hardware is not tg, it is pp: not how fast you talk, but how fast you read when the cache cannot save you.
// where the mac falls down
And here the platforms part company, asymmetrically. Same harness both sides - llama.cpp, Llama-2-7B Q4_0, flash attention on:3
prefill, pp512 (tok/s)own scale
decode, tg128 (tok/s)own scale
two to four times slower
llama.cpp, Llama-2-7B Q4_0, flash attention on
Two bar charts sharing three rows, each on its own scale. Prefill, pp512, in tokens per second: RTX 4090 14,771; M4 Max (40c) 886, seventeen times slower; M4 Pro (20c) 440, thirty-three times slower. Decode, tg128, in tokens per second: RTX 4090 189; M4 Max (40c) 83; M4 Pro (20c) 51; the Macs two to four times slower. Measured with llama.cpp, Llama-2-7B Q4_0, flash attention on.
Read the gaps separately. Decode: two to four times slower, the sort of gap one lives with. Prefill: seventeen times slower on the Max, thirty-three on the Pro. The asymmetry is structural. Decode is bandwidth-bound, and Apple bought bandwidth - 546GB/s on an M4 Max is over half of what the 4090's GDDR6X delivers, hence the respectable tg. Prefill is compute-bound, dense matmul wall-to-wall, and there the Ada die simply has an order of magnitude more usable FLOPS. The Mac was built to talk smoothly. It was not built to read.
On an independent same-harness leaderboard with a 14B model the shape repeats: time-to-first-token of 0.3 seconds on a 4090 against 3.8 on an M4 Max and 6.6 on an M4 Pro. Scale that to a 30B-class model on the 20K-token cold context an agent actually carries. The arithmetic lands roughly here: ten seconds on a 4090 or 5090. Around two minutes on an M4 Max. Three to four on the M4 Pro - the chip in the mini Apple will actually sell you. Per miss. The agent sits there reading, like a barrister re-studying the entire case file because someone in the courtroom coughed.4
A horizontal chart of how long each machine spends reading a 20K-token cold context on a 30B-class model before it can say a word, per cache miss, on a linear scale from zero to four minutes. RTX 4090 or 5090: roughly ten seconds. M4 Max: around two minutes. M4 Pro, the chip in the mini Apple will actually sell you: three to four minutes. The figures are derived from measured anchors rather than read off a single table.
The steelman deserves its paragraph, because it is real. Mixture-of-experts models change the arithmetic - prefill cost tracks active parameters, and a 120B-total/5B-active model on big unified memory reads at over 1,200 tok/s where a dense 70B reads at about 118, which is the difference between usable and geological. Big-RAM Macs running MoE with cache-friendly, append-only agent design are a legitimate machine. But the gap to NVIDIA persists even there (a 4090 reads the same MoE class roughly three times faster than an Ultra, six times faster than a Max), dense 70B - the configuration unified memory is usually sold on - is the worst case, not the redemption, and Apple themselves have rather conceded the diagnosis: the M5 generation ships GPU neural accelerators whose headline claim is 4x faster TTFT. They are fixing it in silicon, which is the sincerest form of admitting it was broken.5 The M5 desktops, of course, are the ones you cannot yet buy.
A word about the Ultras is owed here, because the internet is presently full of people - not wrong ones - declaring the big Mac Studios the ultimate local LLM machines, and this article will look like it is picking a fight with them unless I say plainly that it is not. They are answering a different question. If the question is what desktop holds the largest open weights at all, the 512GB Ultra is not the best answer so much as nearly the only one; no consumer NVIDIA card holds a 400-gigabyte quant at any price. For chat-shaped work - decode-bound, bandwidth-loving - the Ultra's 819GB/s is exactly the right purchase, delivered in near-silence at a tenth of the wall power. The raving is justified. Put the same machine inside an agent loop, though, and the shape of the work reasserts itself. On mid-size MoE it still prefills about three times slower than one 4090. And on the giant models that are its whole point, the reading turns geological: the measured figure for a 671B-class quant was nine tokens per second of prefill - a quarter of an hour before the first word of an 8K prompt - though Metal has had real optimisation passes since, and today's figure is surely kinder. The Ultra is the ultimate library. What the agent loop needed was the ultimate reader.
// the box
So: the swap was a souvenir, the forcing function is failing to force anything in a CSV that updates every sixty seconds, the Mac is structurally wrong for the work, and the sensible conclusion is plainly do nothing.
We built the box anyway.
Here is the full confession. The drawer audit - the council's other free test - turned up a Trident Z5 DDR5-6000 kit bought for a build that never happened, an SF750 that has so far lived its whole life at zero RPM, a small case, and a spare NVMe. The shopping list for a complete six-core AM5 machine came to a B650I Lightning and a £142 Ryzen 7600 with the Wraith cooler in the box - about £278 of new parts, plus thirty seconds in the BIOS switching on EXPO. At that price the decision stops being infrastructure strategy and becomes gardening. A quiet spare Linux box is never the wrong number of Linux boxes. Worst case, it retires as a heavily over-qualified backup target. And the agent stack gets to live somewhere that does not reboot when the array wants to think about its drives.
And then, having written four paragraphs about why local inference hardware was unjustifiable: a 5090 is going in it. Not because the numbers demand it - the workload analysis above is precisely the case against needing one. But the x16 slot is sitting there, the card prefills 20K tokens in eight seconds, and the only way to learn whether local models earn a place in an agent stack is to run them against real work rather than benchmarks.6 Call it a why-not. The BOM acquires a 1000W SFX-L unit, as BOMs do when a 575W card walks in, and the experiment gets to be an experiment instead of a spreadsheet argument.
// the door left open
If the local models earn their keep - if the MoE class turns out to carry real slices of the stack rather than party tricks - the upgrade path is not a bigger graphics card. It is memory, in quantities this little board has never heard of: 256, maybe 512 gigabytes, which means Threadripper or EPYC territory and eight channels to feed - at which point the thing stops being a box on a shelf and becomes a rack server wearing a desktop's clothes. That is a different article, and the nice thing about tonight's £278 is that it commits us to absolutely none of it. The door is open. The hinges were nearly free.
The parity monitor reported in as this piece was being finished: 616 samples across ten and a half hours, spanning the 3 a.m. dream cycle, the 4 a.m. backup and the morning briefing, and the check it was sent to incriminate is invisible from inside the machine - CPU benchmarks flat in every thirty-minute bucket, IO pressure a row of zeros, every blip in the worst-five list lining up with a cron firing rather than a parity read. The most skeptical seat on the council had said the whole case rested on an unmeasured degradation claim. The measurement was free, and it killed the claim. That is two pillars of the original plan now dead by data, and the box getting built on what survives: the shutdown windows, the shared failure domain, the night shift.
| Held up? | What the evidence said | |
|---|---|---|
| The machine is starved | 77 seconds of stall across 78 days of uptime; the swap was a souvenir | |
| RAM is cheap | 64GB of DDR5: about £100 in early 2025, £640 to £990 now | |
| A Mac is imminent | the M5 mini has slipped towards the end of 2026, possibly beyond | |
| The degradation claim | 616 samples across ten and a half hours: the parity check is invisible from inside the machine | |
| What survives | the shutdown windows, the shared failure domain, the night shift |
A table of the claims the original plan rested on, whether each held up, and the evidence. The machine is starved: no; 77 seconds of stall across 78 days of uptime, so the swap was a souvenir. RAM is cheap: no; 64GB of DDR5 cost about £100 in early 2025 and now runs £640 to £990. A Mac is imminent: no; the M5 mini has slipped towards the end of 2026, possibly beyond. The degradation claim: no; 616 samples across ten and a half hours found the parity check invisible from inside the machine. What survives: yes; the shutdown windows, the shared failure domain, the night shift.
The original question - does the agent stack deserve its own computer? - turned out to have the same answer as most questions about whether one should build a small Linux machine, which is that the question was never really the question. The performance case was the costume this project wore to get past the committee. The swap was a souvenir, the degradation was a rumour, and underneath it all, the entire time, was the plain fact that we are nerds who like playing with toys, and a drawer of good parts was never going to stay a drawer. The measurements did not justify the box, and nothing needed to. What they did was make sure it got built with eyes open, for £278. As excuses to play go, that is about as respectable as they come.
That's the take.
Pressure stall information lives in
/proc/pressure/memoryand answers the only question that matters: not "is anything in swap" but "did anyone ever stand in a queue about it." Machines, like attics, fill with the residue of old emergencies. The residue is not the emergency. ↩Nine models, nine vendors, votes cast blind before anyone sees anyone else's reasoning. The composition rule still does the heavy lifting: reviewers who drink from the same training distribution produce one opinion wearing several name badges. The room's job is to be unable to agree by accident. ↩
Microbenchmark caveats, stated plainly: these are llama.cpp single-stream numbers at short depth, the CUDA figures with flash attention on (without it the 4090 reads at a mere 11,993 - the asymmetry survives either way); throughput on both platforms roughly halves by 32K context, and production NVIDIA stacks (vLLM, TensorRT-class, batched FP8 prefill) would widen the gap further, not narrow it. The comparison is conservative in NVIDIA's favour. ↩
The 30B-class figures are derived from measured anchors rather than read off a single table - an RTX 5090 measures ~2,900 tok/s prefill on a 32B at 4K depth, and the Apple-side scaling converges from two independent routes on roughly 150-220 tok/s for an M4 Max. Sources at the bottom; quibbles about ±20% are welcome and do not change which machine finishes reading before you have made tea. ↩
Fairness compels the rest of the steelman: an M3 Ultra runs colossal models under 200 watts and idles at nine, llama.cpp's prefix cache means an append-only loop rarely pays full price twice, and SSD-persisted KV caches can cut a warm start from ninety seconds to three. All true, all real engineering. None of it helps on the cold read, and agent loops are a machine for manufacturing cold reads. ↩
The Founders Edition, it should be noted for fellow small-case sufferers, is the only two-slot 5090 in existence; everything the partners make is a three-and-a-half-slot cathedral. Sometimes the engineering constraint and the aesthetic preference arrive at the same answer, and one should accept these moments gracefully. ↩
Sources (all verified June 2026; an adversarial second pass checked every load-bearing number against its primary source):
- Manus - Context Engineering for AI Agents - the production ~100:1 input:output ratio (self-reported, July 2025)
- Agentic AI Workload Characteristics (arXiv 2605.26297, May 2026) - instrumented agent traces: 53.9-559.8x input:output, 84.6-99.5% KV-cache hit rates, decode-dominance under warm caches
- DeepSeek V3/R1 inference system overview - the 3.6:1 chat-traffic baseline
- llama.cpp Apple Silicon benchmark table and CUDA scoreboard - the same-harness 7B prefill/generation numbers
- LocalScore 14B leaderboard - same-harness time-to-first-token comparisons
- RTX 5090 LLM benchmarks (Qwen3-32B) - 30B-class prefill at depth
- llama.cpp gpt-oss benchmark thread and GPU-Benchmarks-on-LLM-Inference - MoE-vs-dense prefill on unified memory
- MLX on Apple Silicon (March 2026) - M5 neural-accelerator TTFT gains, SSD-persisted KV caching