Get Started
← Advanced builds · Mission 3 of 30

Run an AI model too big for any one of your computers

A 70-billion-parameter model needs about 42GB of memory; your gaming PC has 24GB and your laptop 16GB. llama.cpp's RPC backend splits the model's layers across both, so a model that fits nowhere runs across everything, with a ChatGPT-style web UI on top. It buys memory, not speed: expect slower answers than the same model would give on one machine big enough to hold it.

150 minAdvancedTools: Free. Uses computers you already own, plus Ethernet cablesStep 1 free
A plan before you downloada split planner that says whether a model fits across your machines and how to divide it
A working rehearsal on one PCtwo RPC servers, real layer assignments and real before-and-after speed
Workers on your LAN that share their GPU or memory safely, bound to the LAN only
  • A 70B-class model running across machines, with llama.cpp's chat UI and an OpenAI-compatible API
  • A kit: split planner, worker start scripts for Windows and Mac/Linux, and a network checklist
Before you start
  • Two or more computers on the same network, ideally wired with Ethernet. Any mix of Windows, Mac and Linux
  • Enough combined memory for the model you want: the planner in step 2 checks
  • Comfortable in a terminal on each machine. Start with Run AI on your PC if you've never run a local model
  • Disk space on the main host for the model file (42.5GB for the 70B example)
01/07How one model runs on two computers0 done
STEP 01 of 07~10 min

How one model runs on two computers

Know what llama.cpp's RPC backend splits, what it pools, and what it can't do.

  1. The problemA 70-billion-parameter model at 4-bit is about 42GB. Your gaming PC has a 24GB graphics card, your laptop has 16GB of RAM, a friend's Mac has 32GB. None of them can hold it. Together they can.

  2. The trickA language model is a stack of layers, and each token flows through them in order. llama.cpp's RPC backend lets one machine (the main host) treat GPUs and CPUs on other machines as if they were extra local devices. It puts the first chunk of layers on one device, the next chunk on another, and passes the small in-between results across the network for every token. Each worker runs ggml-rpc-server; the main host runs llama-server with --rpc listing the workers.

  3. By default llama.cpp shares the weights and the KV cache across every device in proportion to its free memory, and --tensor-split overrides that (RPC README ↗).

    • It pools memory. That's the whole point: models that fit nowhere else now run.
    • It doesn't add speed. The layers run one after another, so the machines take turns. Generation speed is set by the slowest device plus a network round trip per hop. One big GPU always beats two small ones; two small ones beat a model that doesn't fit at all.
    • It has no security. The llama.cpp docs call the RPC backend a fragile, insecure proof of concept and say never to run it on an open network. Your own LAN only.

  4. The kitDownload the kit: split planner, worker start scripts for Windows and Mac/Linux, and a network checklist.

✓
CheckpointYou can say which machine will be the main host, which will be workers, and that the goal is memory, not speed.
If it breaksWhich machine should be the main host? + 1 more
Which machine should be the main host?
The one with the fastest GPU and the most free disk. It holds a share of the layers itself, downloads the model file and serves the chat UI, so it does the most work.
Can I mix Windows, Mac and Linux?
Yes. The RPC protocol is the same on every platform as long as every machine runs the same llama.cpp build number. Each machine uses the build for its own GPU (CUDA, Vulkan, Metal or CPU).
Still stuck? Ask the community
Next mission · Advanced buildsTurn three old laptops into a Kubernetes cluster
→