Part 1: Two AI tiers on one Mac Studio, and the routing between them

The economics piece settled whether owning your AI is worth it. Almost everyone who agreed came back with the better question. Fine, how do you run local and cloud together, on one machine, doing real work?

Here is my answer, and it is a partial one. This setup is young, some of it I got wrong twice, and the numbers below are mine rather than a benchmark. What I can say is that it runs, it earns its place, and the interesting part turned out not to be the hardware.

The machine and the traffic split

A Mac Studio, M5 Max, 128 GB of unified memory. It sits on a desk and it also runs a browser and a phone call.

The framing that misled me for a while was local versus cloud, as if every model call were the same call. They are not the same call, and mine fall into three groups with three different homes.

The middle row is not a debate. The top row is where a desk machine stops being a toy and starts being infrastructure, and that is the row this whole post is about. I am still deciding where the last one lands, except that deciding it locally is not on the table.

The 93 GB half

The heavy model is Qwen’s Flash Next, quantised to IQ4_XS: 93.7 GB of weights in three shards on disk, about 5 GB of side files. I read its own header rather than trusting the filename. 48 layers, 512 experts with 10 routed active per token plus a shared one, 24 attention heads but only 2 key-value heads, a 262,144 token window, a gated-delta recurrent block, and full attention arriving only every fourth layer.

Ten of 512 is roughly two percent. That single ratio is why the machine can hold the model at all. The weights are enormous and mostly idle, and generation is bound by how fast you can drag weights past the compute, so a dense model of that total size would either not fit or would crawl. The same hybrid trick, attention mixed with recurrent Gated DeltaNet layers, shows up underneath the small models too, which is a nice confirmation that the architecture choice is doing the work rather than my configuration.

It is served by a Metal build of llama.cpp on 127.0.0.1 behind its own API key, and the flags are the boring part that makes everything else possible. --load-mode dio stops a 93.7 GB load from eating the page cache the desktop needs. The key-value cache is quantised to q8_0 on both keys and values, which is what makes a 262k window survivable. --spec-type draft-mtp --spec-draft-n-max 3 points at a 4.1 GB multi-token-prediction draft so the big model verifies guesses instead of generating them. The vision projector is a separate 907 MB file that loads when images are in play and not otherwise. --models-max 1, because a second resident model is a memory incident I have already had once.

The fast half, and what Kev actually is

The decision tier is Kev, an Apache-2.0 family of small decision models fine-tuned on Qwen bases, and the single biggest adjustment in my head was that Kev does not chat. It takes a document plus a set of typed questions and returns probability distributions in one forward pass, without generating any text.

Three question types. noul for yes or no, choice for a distribution over options you supply, score for a rating placed against an ordered scale. A pointer head scores each option’s closing token against the question’s deciding token, and a softmax turns those scores into probabilities. No decoding loop, which is the whole reason a decision costs milliseconds rather than a hundred of them.

Two details in there matter more than they look at first.

Questions share the document but cannot read each other, because on these bases each question runs as its own row against a state computed once. On attention-only models that isolation comes from a mask; on the Qwen3.5 and 3.8 hybrids the recurrent layers ignore masks, so the server gets exactness from structure instead. And each checkpoint ships with a temperature fitted so its probabilities are calibrated, which is why my 4B serves at 2.41 and the 27B at 1.32. That number is not a creativity dial. It is the reason I can set a threshold like above 0.9 skip review and have it mean something.

I put both sizes through the same battery on this machine. The two agreed on every multiple-choice answer, which was not obvious in advance, and then split on the questions that asked for a ranking.

The confidence column is the finding. The 4B spread itself flat across the rating scale and reported near zero confidence, which is it telling me honestly that it has no opinion. The 27B gave a real ranking with actual confidence, and its ordering is right: the single unified memory pool and 4-bit quantisation as the load-bearing elements, hybrid attention and on-demand loading as helpful but replaceable. That is why the 27B sits in front of batch gating and ranking, where being wrong is expensive, while the 4B sits inside loops where every next action waits on the answer.

Asking eight questions in one call cost less than asking one (156 ms against 211 ms), because the expensive part is reading the document once, not the questions. That is a routing rule I would not have found without measuring, and it matches what the project’s own serving numbers describe.

Two things I did wrong there, since they are the kind of wrong that costs people real time. I asked noul an open-ended question and got back 0.625 from one model and 0.264 from the other. Two numbers, no sentence, no meaning. It wants a yes or a no, and nothing in the response complains when you get that wrong. And I read the temperature as a problem to fix before I understood it was calibration.

The constraint is the 128, and it is not evenly spent

Disk is cheap. Memory is the budget, and the two halves do not want to be resident at the same time.

Nothing stays loaded. The 4B and the 27B each sit behind a small supervisor that costs tens of megabytes: it spawns the real backend on the first request, waits for the port and the model listing to actually answer before forwarding, and terminates the backend after ten idle minutes. The always-on part is the cheap part and the expensive part is on demand.

I learned the limit the hard way. With Flash Next loaded at 75 GB of resident memory, I started a cold load of the 27B on top of it, which needs 51 GB. Swap went to 29.6 GB of 30.7 GB, memory free dropped to fourteen percent, and the loader crawled along at twelve percent CPU instead of the nine seconds I had measured earlier. I killed it and unloaded both. That is the honest version of this setup: 128 GB is enough for the big model plus a small one, and not enough for the big model plus a 27B, and the supervisor is a queue for memory rather than a way to dodge it.

The memory numbers also moved, which is worth knowing before you size anything from a single reading. The 4B peaked at 7.5 GB and settled to 3.65 GB, because the weights are file-backed and the system takes the pages back when nothing touches them. The 27B measured 48.4 GB in one reading and 41.0 GB in another. Peak during load is the number that matters, not steady state, and Kev’s own documentation explains why: adapters get merged into the base at load time, which briefly holds a second copy, while full-weight checkpoints like the 27B load as saved with nothing merged.

There is a row in that documentation nobody had filled in. Kev-27B is described as needing an 80 GB datacentre GPU, and the Apple Silicon path is marked expected to fit a 96 to 128 GB Mac but not yet measured there. My box is a 128 GB M5 Max, and it did run: 41 to 48.4 GB resident depending on when I sampled, answers in about a second, probabilities present and thresholdable. I never got a clean peak number, because the load I started in order to capture one is the one I had to abort. The published reference for the same model on CUDA is 65.5 GB resident and 17.6 s to load, so the Mac path sat under the GPU footprint and loaded faster than it, while giving up the throughput that the GPU is there for. One machine and one configuration, so treat that as corroboration rather than a spec, and keep the project’s own caveat in view: on a Mac answers take hundreds of milliseconds rather than tens.

Hermes is what makes it one system

Three model servers on one box is not a setup, it is three hobbies. What turned it into a system is Hermes pointed at all of them as providers, the big local model, the 4B and the 27B, local whisper for transcription, and a hosted frontier model for the work that needs one, behind one routing layer with memory and skills attached.

The piece I keep coming back to is not in any benchmark. The preset file for the heavy model carries a line I did not hand-write:

# hermes-decision: {"model_id": "Qwen3.8-Flash-Next-UD-IQ4_XS", "window": 262144, "spilled": false, "refusal": null}

Which model was chosen for the job, the window it fitted into, whether memory spilled, whether it refused. A small classifier picking the brain per task and writing its reasoning down beside the config it produced. Writing it down is the half people skip. Without a record you cannot tell whether local was good enough, you only have a feeling, and feelings are how you end up serving 93.7 GB of weights for work a 4B answers in 156 milliseconds. With a record you can read a week of routing and find the misses.

The same applies to skills. The measured latencies, the cold-start costs, which flags matter, what each question type expects, all of it lives in skills that load when the task is relevant, instead of in my head, where I had two of those wrong.

What the cloud still owns

A lot, and the honest number for how much comes from Kev’s own documentation rather than my impressions.

At a five percent error budget, those models can automate between 0.52 and 0.69 of decisions on sources they have not seen, against 0.70 for the hosted Jev they are built to replace. So a third to a half of real decisions still need to escalate, and calibration cannot fix that because it is a single fitted temperature, which cannot reorder confidences. Knowledge questions are set by the base model: the 27B scores 0.675 on MMLU-Pro against 0.840 for Jev, and no amount of decision fine-tuning changes what the base never learned. Validated context is 8,192 tokens on the 4B and 65,536 on the 27B, which is a routing rule with a documented basis: send the long document to the bigger model because of its measured length, not because it sounds more impressive.

Then there is the instructive regression. Kev-27B v2 is the more accurate model and it got worse at calibration on long contracts, expected calibration error 0.053 against 0.007 for v1, more confident and less trustworthy at the same time. Newest and strongest is not what you want in a gate. I now keep the older numbers next to the model I am using.

What stayed in the cloud this week was judgement with no clock on it: reading a machine’s file tree and deciding which folder was actually the site for a given domain, noticing that five of fifty-two repos had no remote and so were the only unrecoverable data on it, spotting forty-three commits that had never been pushed, working out that a credentials CSV had no business being in a project tree. What was local was the touching: files crossing a wire, byte totals compared, accounts queried for key pairs. A machine doing network work needs hands, and hands do not belong in a chat completion.

What is still rough

I have no labelled set of my own decisions, so I am reading calibrated probabilities from models calibrated on public data, which is exactly what their documentation warns will not transfer. My accuracy comparisons are samples of dozens, not hundreds: one hundred social posts on a single topic, twelve option permutations on a single question, both anecdote with a decimal point. The permute result looked clean, same answer in all twelve orders, with the probability of the chosen option moving between 0.9231 and 0.9573, and the project states plainly that option order can change an answer and question isolation does not prevent it. Twelve runs of one question is not a refutation of that.

I also still route partly by feel, which is the thing the decision record exists to remove. Next is a week of logging every routing call against what I actually wanted, then setting thresholds on my own data rather than borrowing fitted temperatures.

That is the shape of it. Decisions at volume go local, and you pick the size by whether the model has a distribution worth reading rather than by how hard the question looks. Raw audio and private narrow jobs go local, because there is no trade-off to manage there. Judgement, range, and long chains where being wrong early is expensive go to the strongest model you can buy. Write down which one you chose, and check the memory before you check the model list.

The hardware took an afternoon. Learning what each tier is actually good at is still going.