Let's run a 320B model on 16GB Mac M4

Mehdi Roshan Fekr ·

Sep 2, 2026

Hardware: MacBook Pro M4, 16 GB unified memory, 10 cores · Storage: 1 TB SanDisk portable SSD · Engine: llama.cpp (Unsloth fork), one line of source changed

1. The reading list that started it

In March 2026, Dan Woods published "Autoresearching Apple's 'LLM in a Flash' to run Qwen 397B locally" — the Flash-MoE project. He took Qwen3.5-397B-A17B, the 397-billion-parameter flagship MoE, and ran it on a MacBook Pro M3 Max with 48 GB of unified memory. The trick: only ~5.5 GB stayed resident in RAM while the remaining ~209 GB (4-bit quant) streamed on demand from the internal NVMe at ~17 GB/s. The model ran at ~5.7 tok/s. Simon Willison covered it, Hacker News argued about it, and a newsletter headlined it perfectly: "Your SSD Is the New GPU."

I collected the links, read them twice, and kept the details in my findings notes:

The Hacker News thread is worth a skim too — mostly people arguing that "48 GB is not 16 GB," which is, roughly, the debate this article is about.

I always found these kinds of experiments appealing, but I didn't want to replicate the existing ones — I was looking for a new type of experiment. I read the articles with real enthusiasm and was fascinated by the art of thinking these people brought to the problem. And I couldn't shake the feeling that there was room to push a little further — to run at least one additional experiment in a different, yet similar and more challenging environment. And since we developers love engineering and experimenting, this idea kept circling in my head for a while.

2. The Moment of Truth

Right around the time I was turning these articles over in my head, I decided the other change this experiment needed was to put everything on a portable device. My MacBook had 256 GB of storage — meaning even the model itself could hardly be stored on the computer, let alone run from it. So the experiment would live on something portable. And lucky for me, right around the time I was figuring this out, Z.ai introduced GLM-5.3-Flash — GLM being my favorite model family — and its really interesting numbers made me a lot more interested in moving the experiment onto it:

Dan had 48 GB of memory and an internal drive nearly twenty times faster than mine. I had 16 GB and a portable SanDisk SSD that I use to carry files between machines. Dan built a custom 5,000-line inference engine in 24 hours with an AI agent doing the research.

I asked the smaller question: can vanilla llama.cpp — with quantized GLM-5.3-Flash and no custom engine at all — do this on a machine that can't even hold the quant?

3. The machine that shouldn't be able to do this

Let's be precise about the hardware, because the hardware is the story. When I bought this MacBook Pro I was thinking about normal things — compiles, browsers, Docker. I chose the 256 GB internal drive, because 16 GB machines with big drives are expensive, and I never imagined I would one day want a 200 GB file to live anywhere near it.

So everything in this project — the engine, the patch, the client, and the model itself — lives on a 1 TB SanDisk portable SSD, carried over USB:

Look at that working-set line again. The per-token working set is ~15.1 GB against ~15 GB of RAM. Not comfortably under. Not slightly over. Exactly at the knife's edge — the quantized model just barely decomposes into a part that can live in memory (the routing brain) and a part that can be swapped in from an SSD (the expert tissue). The experiment exists precisely because that line exists.

4. "llama.cpp doesn't know this model exists"

Vanilla llama.cpp, as of this writing, does not support the glm5next architecture GLM-5.3-Flash uses — there are three open PRs to add it (#27752, #27754, #27773), none merged. The working path is the Unsloth fork of llama.cpp, which carries the architecture on its glm5next/upstream branch (itself proposed as unslothai/llama.cpp#61). So step one was cloning the fork and building llama-server from source:

And then the loader died on the very first run. The architecture was only half-integrated: the enum knew about glm5next, but the model-loader path never properly set up its mappings — and where the fork did wire it in, it was wired wrong. The line that killed the run was at src/llama-model.cpp:1684:

// the fork's version — hardcoded prefetch:
ml.init_mappings(true, use_mlock ? &pimpl->mlock_mmaps : nullptr);

The first argument of init_mappings is prefetch. Hardcoded true means: on every load, issue POSIX_MADV_WILLNEED across the entire 199.7 GB file before a single tensor is allocated. Upstream llama.cpp ties prefetch to use_mlock — a deliberate, tiny, opt-in pinning. The fix is exactly one line, and it remains my only source change to the whole engine:

// patched — the one-line diff:
ml.init_mappings(use_mlock, use_mlock ? &pimpl->mlock_mmaps : nullptr);

Rebuilt and the server came up:

# 1. get the fork (the branch that carries glm5next)
git clone --branch glm5next/upstream https://github.com/unslothai/llama.cpp
#    this article was built on base commit d07e71e

# 2. apply the one-line patch to src/llama-model.cpp:1684 (above)

# 3. configure once, build
cmake -B build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=OFF
cmake --build build --config Release -j --target llama-server

That is the entire change surface of this project: one fork branch, one line of C++, one rebuild.

5. The disclaimer (and the irony)

Before we go further, full disclosure: I am not a C or C++ programmer. I studied and played around with both during my computer-science years — programming languages are a genuine love of mine — but nowhere near close to being able to handle something like this. I cannot read ggml internals for pleasure, and I did not find init_mappings by squinting at a stack trace. The diagnosis and the patch were AI pair-programmed, end to end: I described the crash, the AI read the source, found the half-integrated architecture, wrote the one-liner, and explained the mlock semantics to me like a patient colleague. What I brought to the table was the direction — from my knowledge of how files, memory, and compute units work, I knew what this experiment should look like and where it should go; the AI turned that direction into working code.

And here is the irony I cannot resist putting in a headline somewhere: the assistant was GLM itself — accessed through its API. The same model we were fighting to run locally on 16 GB of RAM wrote the code that made it run. The API served the 320B model from a datacenter; the patch let a 16 GB Mac serve it from a USB SSD. AI debugging its own runtime.

6. Attempt #1: hours of silence, then killed

The first full experiment was, frankly, heartbreaking. Everything was wired up: the model on the SSD, the patched server, the streaming client. I started the run and watched — silent disk streaming at 100%, minutes at a time. RSS climbing. Then more silence. Eventually macOS solved the problem for me: zsh: killed. No crash report, no log line, no goodbye. (Lesson learned the hard way: jetsam SIGKILLs print nothing — the reason only lives in log show, under memorystatus.)

A few hours of that cycle and I finally ran the diagnostics properly — iostat on the SSD, sample on the process, vmmap for residency — and the answer was embarrassing: I had wired the SSD into the wrong port. A USB 2 port. The drive itself is USB 10 Gbps-capable; the port was feeding it ~30 MB/s.

The arithmetic of that mistake is brutal. The model is 199.7 GB:

199.7 GB ÷ 30 MB/s ≈ 1.85 hours — for ONE full pass over the file

At 30 MB/s, the loader's eager prefetch pass alone was a ~2-hour job, and the page cache never had a chance to hold the working set anyway. The weights were never going to make it into memory — not because the approach was wrong, but because the pipe was 100× too thin. And with the working set re-read per token, even a "successful" run at that speed was physically hopeless.

Moved the SSD to a proper port and re-ran the cold read: ~915 MB/s — over 25× the throughput, same drive, same cable, same everything else.

The bottleneck was never the theory. It was the plumbing.

7. Attempt #2: mmap and hope (and why it thrashed)

With the right port, the obvious next move is the vanilla one: let llama.cpp mmap the whole 199.7 GB and let the macOS page cache page weights in and out. On paper this is the cleanest design — zero copies, the kernel is the streaming engine, and it's exactly how the 17.3-tok/s Qwen-35B Mac-mini trick works.

On a 16 GB machine with a 200 GB file, it thrashed. Three separate mechanisms were stacked against me, each invisible in the logs:

1. The eager prefetch (the one-line patch from §4). With init_mappings(true, …), the loader WILLNEED'd the entire file on every load — ~7 minutes of pure I/O even at 466 MB/s, page-cache pressure through the roof, footprint of the process attributed at 48.7 GB. Then jetsam.
2. The kernel's LRU is layer-blind. The page cache has no idea that layer N's expert weights are dead the moment layer N+1 starts. It kept evicting the wrong things and re-reading them.
3. The default flags assume the model fits. The --fit memory fitter sized buffers optimistically, the warmup pass touched every weight (pulling all 199.7 GB through the pipe before serving anything), and memory fitting kept trying to make room for a model that would never fit.

And the pivot — the one that made the whole thing work — came from looking at the MoE structure itself. GLM-5.3-Flash is not one big model; it's a ~9.84 GB "routing part" that is needed on every token (attention, router, shared expert, embeddings, output head) plus 288 ~0.66 GB expert modules of which only 8 fire per token. I didn't need the experts in memory — I needed the routing part resident, and the experts swappable. Keep the brain in RAM; stream the tissue from the SSD. The OS page cache is exactly the tier for that — but only after every silent copy/pin mechanism is tamed.

8. The llama.cpp surgery

This is the part I promised to elaborate, because nothing about it was obvious and everything about it was load-bearing. With the prefetch fixed, the config still died — silently copying the whole model into RAM — until three separate mechanisms were found and tamed:

8.1 The eager prefetch (fixed in source)

§4's one-line patch. The fork hardcoded full-file WILLNEED; upstream ties prefetch to use_mlock. Effect: load went from a ~7-minute full-file read (and jetsam kill at 48.7 GB attributed footprint) to a 20–28 s, zero-copy, metadata-only mmap.

8.2 --tensor-read-lazy — the feature that did nothing

The fork ships an on-demand tensor feature: --tensor-read-lazy on|auto|off, default auto. Sounds perfect for streaming, right? I checked the logs for lazy read enabled lines: zero. The reason is a size cutoff: auto only marks tensors larger than 4 GiB as lazy. GLM-Flash's expert tensors are 1.3–1.7 GB each — none qualified. And on only marks tensors the architecture explicitly flags with TENSOR_READ_LAZY, which glm5next flags none of. Conclusion: leave it off; plain mmap + page cache already does on-demand paging. The lesson generalizes: when a feature does nothing, check its cutoffs before assuming it's working.

8.3 The repack buffer — the silent full-model copy (fixed by flag)

The nastiest one. With -ngl 0 (CPU inference), the CPU backend wraps mmap'd weights in a repack buffer: each quantized tensor is read out of the mmap and memcpy'd into a freshly malloc'd buffer in a faster-compute layout. That is a full 199.7 GB copy into RAM — invisible in the logs, fatal on a 16 GB machine. I found it by sampling the live process (sample <pid> 2) and reading the stack: load_all_data → ..._set_tensor → repack_q4_K_to_q4_K_8_bl. Repack is a compute-speed optimization — and on a storage-streaming setup it's poison: it converts "stream from SSD" into "copy 200 GB into RAM."

Fix: --no-repack — weights stay zero-copy in the mmap, and the page cache becomes the tier.

9. The final recipe

Every flag in the final command earns its place:

/Volumes/PortableSSD/start-server.sh --detach
# which runs:
llama-server \
  -m .../GLM-5.3-Flash-UD-Q4_K_XL-00001-of-00006.gguf \
  -ngl 0 --no-repack --fit off --no-warmup \
  -c 1024 -t 10 --jinja \
  --host 127.0.0.1 --port 8080

On the good hardware path this works beautifully. On load: ~11 GB resident for a 199.7 GB model — 82% of memory still free, zero swap, no jetsam. The chat client (stream_chat.py) talks to the OpenAI-compatible endpoint and prints tokens as they arrive.

10. Results — the honest numbers

First, the disclaimer I promised: by no definition is this a usable setup. You will not chat with this thing. If you want a usable number, Dan's 5.7 tok/s on 48 GB and internal NVMe is what "usable" looks like; mine is not that, and was never going to be.

But that was never the point. The point is modularity: these numbers show that a 320B model can be decomposed — into a ~10 GB resident routing brain and ~190 GB of swappable expert modules streamed over a portable SSD — on a machine whose RAM is 13× smaller than the model. The architecture of GLM-5.3-Flash turns out to be that modular. Here is the matrix, both kinds of day:

("Bad day" = the slow USB converter or a machine busy with other work — when the page cache fights the rest of the OS for the same 15 GB.)

Where the variance comes from: each token routes to 8 of 288 experts per layer, and expert slices fault in from the SSD unless they're already cached. The ~9.84 GB always-active part stays hot in page cache; the routed experts churn. The expert hit rate decides whether a token costs 6 seconds or 12–23. The bandwidth equation predicted all of it:

tok/s ≈ storage bandwidth ÷ bytes re-read per token

Take that equation seriously and you can daydream about better hardware. These are purely theoretical projections — back-of-the-envelope math, nothing measured, and the cache hit-rate terms are the least certain part:

11. Rules for running an LLM between an SSD and RAM

Hard-won, condensed — with the commands where they matter:

1. llama-server's HTTP loop starts before the model loads — /health returning 503 means loading, not hanging:

   curl http://127.0.0.1:8080/health   # 503 "Loading model" = still streaming, keep waiting
   

2. Trust nothing about mmap. Prefetch, lazy-read cutoffs, repack buffers — three separate mechanisms can each silently copy or pin the whole file. Diagnose the live process, not the logs:

   sample <pid> 2 -file /tmp/s.txt
   # look for: load_all_data → ..._set_tensor → repack_q4_K…  = a full copy is happening
   

3. --no-repack is mandatory for zero-copy CPU inference of quantized models from slow storage (§8.3).

4. Check feature cutoffs. --tensor-read-lazy auto has a 4 GiB cutoff; if your tensors are smaller, the feature is inert. Verify it actually did something:

   grep "lazy read enabled" ~/llama-flash-server.log || echo "feature did nothing"
   

5. Never let Metal allocate on a 16 GB machine with a 200 GB model. -ngl 0 keeps the GPU out of the memory equation.

6. jetsam kills are silent. zsh: killed, no crash report — the reason lives in the system log:

   log show --last 10m --predicate 'eventMessage CONTAINS "memorystatus"'
   # → killing largest compressed process llama-server [PID] 48677 MB
   

7. mmap is the streaming engine. With zero-copy mappings, the OS page cache is the tier: the always-active ~10 GB stays hot, experts fault per-need and evict. No userspace cache required.

8. Time-to-first-token ≈ working set ÷ bandwidth. Every improvement lives in that fraction — and the wrong USB port is a 100× tax on it.

9. Detach long server processes — macOS has no setsid, so an aborted terminal will kill your run. Detach via a new session instead:

   python3 -c "import subprocess,sys; subprocess.Popen(sys.argv[1:], start_new_session=True)" ./start-server.sh
   

And the launcher itself — the whole thing, since it's short and it's the reproducible form of §9:

#!/bin/bash
# start-server.sh — the verified streaming config from §9
set -euo pipefail

DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
BIN="$DIR/llama.cpp-glm5next/build/bin/llama-server"
MODEL="$DIR/models/GLM-5.3-Flash-UD-Q4_K_XL/UD-Q4_K_XL/GLM-5.3-Flash-UD-Q4_K_XL-00001-of-00006.gguf"

[ -x "$BIN" ]   || { echo "ERROR: llama-server not found at $BIN";   exit 1; }
[ -f "$MODEL" ] || { echo "ERROR: model GGUF not found at $MODEL"; exit 1; }

exec "$BIN" -m "$MODEL" --alias glm-5.3-flash \
    -ngl 0 --no-repack --fit off --no-warmup \
    -c 1024 -t 10 --jinja \
    --host 127.0.0.1 --port 8080 "$@"

(The real script adds a --detach mode that wraps the same command in rule 9's start_new_session trick and logs to ~/llama-flash-server.log.)

12. If you want to take this further

That was my run. If you want to pick this up and push it toward usable, here's where I'd start — in order of leverage:

1. Fix the bandwidth term first. Everything in §10's daydream table lives or dies on this number. A Thunderbolt 5 NVMe enclosure (~3 GB/s) is the single biggest upgrade; even my ~915 MB/s portable drive left most of its own pipe on the table.
2. Give the cache more room. More RAM isn't about "fitting the model" — it's about the expert hit rate. A 64 GB machine grows the expert cache from ~5 GB to ~50 GB, and MoE routing has enough locality that hit rates climb fast (Dan Woods saw ~71% hits with just 5.5 GB cached).
3. Try speculative decoding. GLM-5.3-Flash ships with a NextN/MTP draft layer — keep that resident in RAM and let it draft while the 320B verifies. Multi-token acceptance multiplies tok/s without touching the bandwidth term.
4. Write the layer-eviction patch. Right now, streamed expert weights sit in page cache until the OS evicts them. Dropping layer N's experts the moment layer N+1 starts would roughly halve the resident set. llama.cpp has no hook for this today — if you build it, it's a genuine contribution.
5. Upstream the one-liner. The init_mappings fix at src/llama-model.cpp:1684 is a private patch. Turn it into a PR against the Unsloth fork — or against the mainline glm5next PRs (#27752, #27754, #27773) — so the next person doesn't inherit a rebase time-bomb.
6. Measure, don't trust. This experiment only worked because sample, iostat, and log show said what the logs wouldn't. Whatever you change, measure the page-cache hit rate and the effective streaming rate. The equation is unforgiving, but honest.

From mechanism proof to production

That experiment ran on the smallest machine possible — on purpose. If you want this in production, with machines that stream at 17 GB/s instead of 30 MB/s, that's what we do at Qcentic:

• Self-hosting, properly provisioned. You provide the GPUs — we build you a production vLLM setup (tensor-parallel serving, continuous batching, tool calling, reasoning parsing) and wire it into our own serving pipelines. Same model families as this article; machines that hold them.
• Agentic flows & AI consulting. We consult on agentic architecture, evaluation, and AI setups end to end — and we bring our own Actana Core and harness (built on top of the open-source pi coding-agent harness) for multi-agent, multi-harness software delivery.
• Open source: Actana Control. Our control plane for orchestrating agent Sessions is open source at control.actana.ai — built by our own software factory: a heavily test-driven, static-analysis-driven, multi-agent / multi-harness process that uses Actana Control itself to build Actana Control.

More open source and more articles soon. If your machines — or ours — should be running models like this, talk to us at Qcentic.

The entire kit — the patched engine, the portable launcher, the streaming client, and the notes this article is based on — fits on the same 1 TB SSD that carries the model.