Project case study
Home Token Factory
wipA home inference fleet, roughly 1.68TB of unified memory. Two Mac Studio M3 Ultra (512GB each), an M4 Max agent box, a 4-node DGX Spark GPU cluster on a 200Gbps ring, a Mac mini head node, NAS, and a router. Local inference, no cloud.
The home token factory is the inference fleet everything else runs on: about 1.68TB of unified memory across Apple Silicon and NVIDIA, all on one home LAN.
- Heavy inference: two Mac Studio M3 Ultra, 512GB unified memory each, 4TB SSD. They serve the big models and export the model store over NFS.
- Agent box: one Mac Studio M4 Max, 128GB, 4TB. Pinned Ollama models, the always-on agent fleet, and cron.
- GPU cluster: four NVIDIA DGX Spark (GB10 Grace Blackwell, 128GB each, 512GB pooled), wired in a ConnectX-7 200Gbps DAC ring with no switch, running vLLM at TP=4.
- Head node: a Mac mini M4 running Elias, the orchestrator and gateway. Plus a QNAP NAS for model and backup storage and the LAN router tying it together.
The DGX Spark saga
Getting the 4-node ring stable was the hard part, and none of it was in the docs. I found it by probing. The 2.29.2 NCCL Docker image has a known bug, so I built a custom NCCL 2.28.9. The ConnectX-7 interface names do not match the physical ports. RDMA only works at L2, so the ring runs with NCCL_IB_DISABLE=1. Spark A re-exports the NFS model store to B, C, and D over the DAC ring.
In progress: making the multi-node ring stable enough to be the primary local inference for the agent fleet.