What did the GPU wake up?
In the 100 Gb/s checks, Mac WRITE completions above 16 µs fell from 195 per 1,000 with the Metal helper off to 10–15 with it on. The cause has not been measured.
Read the GPU investigation ↗00 / FOR THE ENGINEERS
Show the run.
Three questions worth investigating, followed by the conditions behind the headline results. The recorded run and today's public source are identified separately.
Where can the buffer live?
Mapped host allocations passed Metal/CUDA byte checks, while direct RDMA registration of an existing cudaMalloc allocation failed on the tested Spark.
Can AMD join the path?
Strix Halo RDMA and a Vulkan compute client are the next tests. There is no result to plot yet; a system is being sought for the experiment.
See the Strix Halo plan ↓Every number has a boundary.
Open a result for its test setup, public reproduction recipe, and limits. The downloadable CSV contains published summaries, not per-sample traces.
013.680 µsSpark → Mac · 4 KiB RDMA WRITE0.1.17 · 3,000 samples
Recorded conditions
- M3 Ultra Studio, macOS 27 build 26A428, Helios 5S with ConnectX-5 Ex, one DGX Spark with ConnectX-7.
- 40 Gb/s negotiated link; 4 KiB host buffer; queue depth 1; RDMA path MTU 1024; BlueFlame-64 and a continuous Metal keepalive.
- Three runs of 1,000 samples per operation after 100 warmups, pooled without dropping outliers.
Submission to application-observed completion on the initiator. This is host-memory latency, not one-way wire latency, GPU-to-GPU latency, or a measured 0.1.18 speedup.
Current-source reproduction recipe
python3 tools/native_cross_host.py \
--mac-host "$MAC_SSH" --peer-host "$PEER_SSH" \
--mac-client "$MAC_CLIENT" --peer-client "$PEER_CLIENT" \
--mac-provider /usr/local/lib/rdma/libmcdma-rdmav34.so \
--mac-checker "$MAC_CHECKER" \
--mac-interface "$MAC_IF" --peer-interface "$PEER_IF" \
--mac-device "$MAC_RDMA_DEVICE" --peer-device "$PEER_RDMA_DEVICE" \
--peer-gid-index "$PEER_GID_INDEX" --path-mtu 1024 \
--payload-bytes 4096 --mac-cq-map 2 --mac-user-post 1 --mac-user-bf 64 \
--mac-latency --peer-latency --output results/latency-4096-mtu1024-gpu-active-bf64.jsonRun the documented four-way byte checks first, set the variables from the install guide, and start fabric-keepalive 0 small on the Studio. This recipe targets the current public source; it is not a replay of the historical 0.1.17 binary.
0250.6 Gbit/sSpark → Studio · Studio-initiated READ0.1.18 · 96 GiB
Recorded conditions
- 100 Gb/s link; PCIe Gen4 x4 Thunderbolt tunnel; one RC queue pair at depth 1; a reused 4 MiB registered host-memory slot.
- 96 GiB transferred in 16.302 s in one longer run after the permission and PCIe-counter update.
- Build identity: kernel UUID
0B3074D1-560B-3FBC-954B-B9F327965DC7; provider SHA-25618fb492ee38a2663a2d8187b04b412c81c5e7bda87c122d332c0918457ef0eca.
The run checked sampled payload windows and guards, not every byte. Timing excludes setup, registration, initialization, and verification; the initiator busy-polled about one CPU core.
Published bandwidth recipe
python3 benchmarks/run_bw.py \
--mac-host "$MAC_SSH" --peer-host "$PEER_SSH" \
--mac-bw "$MAC_BW" --peer-bw "$PEER_BW" \
--mac-provider /usr/local/lib/rdma/libmcdma-rdmav34.so \
--mac-checker "$MAC_CHECKER" \
--mac-interface "$MAC_IF" --peer-interface "$PEER_IF" \
--mac-device "$MAC_RDMA_DEVICE" --peer-device "$PEER_RDMA_DEVICE" \
--peer-gid-index "$PEER_GID_INDEX" \
--ops read,write --initiators mac,peer --sizes 4194304 \
--depths 1 --qps 1 --total 8589934592 --repeats 3 --warmup 1 \
--mtu 4096 --finish flag --verify-bytes 1048576 --timeout 120 \
--mac-cq-map 0 --mac-user-post 0 --mac-user-bf 0 \
--output results/bandwidth-4mThis published recipe repeats 8 GiB trials to reproduce the bandwidth shape. It is not the original command for the single 96 GiB run.
031.8× faster reply15,402 prompt tokens · 128-token responseFirst split run
What ran
- Qwen3-4B-Instruct-2507, MXFP4; vLLM prefill on one Spark, a 2,166 MiB KV cache pulled over RDMA, MLX decode on the M3 Ultra Studio.
- At this prompt length, the published stage-sum reply time was 5.55 s split, 10.03 s Studio-only, and 5.72 s Spark-only.
- The RDMA pull took 0.48 s; Studio decode ran at 109 tokens/s. The run used lab driver 0.1.17 and file staging on both machines.
Reply totals add separately measured stages and exclude SSH setup. There was one prompt family and one sweep, with no independently timed full client request or matched TCP hand-off.
Reproduction status
The experiment scripts, tool hashes, and checkpoint hashes remain outside the public repository, so there is no honest copy-and-run command for this historical result. The report publishes the table, stage timings, answer checks, and limitations.
Follow the hand-off
Step through the reported path from Spark to Studio.
01 / DGX SparkPrefill on CUDAvLLM builds the KV cache for a 15,402-token prompt.
02 / RoCE v2RDMA pulls 2,166 MiBThe reported cache transfer took 0.48 s.
03 / Mac StudioDecode on MetalMLX generated at 109 tokens/s in this run.
04 / Reported total5.55 s splitStage sum for a 128-token reply; Studio-only was 10.03 s.
Illustrated from published stage measurements. This is not a recording or an independently timed end-to-end request.
Raw lab traces and the original inference scripts are not in the public repository. The CSV above transcribes published summary values only; it is not raw measurement data.
01 / THE IDEA
Why split the work?
A Mac Studio and a DGX Spark are each good at one half of running a model.
The Spark's Blackwell GPU and CUDA stack get through a long prompt fast. The M3 Ultra, with 256 GB of fast unified memory, generates the reply faster. On the same desk they still behave like two separate computers with a network socket between them.
We wanted them to work as one machine: prefill on the Spark, decode on the Mac, and a KV-cache hand-off quick enough to pay for itself. That needs RDMA, where one machine's network card reads or writes registered memory on the other machine directly.
So we wrote the driver. MCDMA runs a Mellanox ConnectX-5 Ex inside a Thunderbolt 5 PCIe enclosure, plugs it into Apple's own RDMA stack on macOS 27, and speaks RoCE v2 to the ConnectX-7 in each Spark. Ordinary verbs code on either machine can post a READ or WRITE to the other.
The first split request beat both machines working alone at every prompt length up to 15,402 tokens. It did that on first-version plumbing, with a file on each side and an extra copy.
Each machine wins one half
Prefill, time to first token lower is better
DGX Spark2.69 s
Mac Studio8.81 s
Decode speed higher is better
Mac Studio109 t/s
DGX Spark42 t/s
02 / MEASURED 15 AND 17 SEPTEMBER 2026
The link, under a microscope.
Byte-verified runs between a Mac Studio and DGX Spark. The test conditions sit under every figure, the same way they do in the repo.
Completion latency
4 KiB at queue depth 1. The arrow names the initiator, and READ pulls the bytes back from the peer. Pick a cell.
WRITEREAD
Mac → SparkMac initiates
Spark → MacSpark initiates
Spark → Mac · RDMA WRITE
4 KiB goes out to the Mac and the acknowledgement comes back. One loop is the median slowed 200,000×: 0.74 s.
- Median
- 3.680
- p95
- 4.320
- p99
- 4.384
- Max
- 21.312
- Samples
- 3,000
Lab driver 0.1.17 · macOS 27 build 26A428 · M3 Ultra Mac Studio, 256 GB · one DGX Spark · 40 Gb/s link, RDMA path MTU 1024 · userspace posting with BlueFlame-64 · Metal keepalive running · 3 runs × 1,000 samples per operation after 100 warmups, pooled · registered host memory, submission to observed completion · all 12,000 samples kept, outliers included
Re-cabled to 100GBASE-CR4 later that day, single runs came in 0.1 to 0.4 µs lower. The headline stays on the pooled 40 Gb/s result because it has three times the samples.
Keeping the GPU busy cut latency
A tiny Metal kernel running on the Mac's GPU lowered RDMA completion times in both directions, even though the transfers used ordinary host memory.
Mac-initiated 4 KiB writes over 16 µs per 1,000
Keepalive off195
Keepalive on10 to 15
Our working theory is a power state tied to GPU activity. We haven't measured the mechanism, and the keepalive costs GPU time and power. If you know what's going on, we'd like to hear from you.
Sustained bandwidth
Driver 0.1.18 · one queue pair · a reused 4 MiB host buffer · sampled byte and guard checks after every trial.
Into Mac memorySpark → Studio · Studio-initiated READ · 96 GiB in 16.302 s50.6Gbit/s
Out of Mac memoryStudio → Spark · Studio-initiated WRITE · 64 GiB in 18.676 s29.4Gbit/s
The 100 Gb/s port isn't the limit. The Thunderbolt 5 PCIe tunnel is, at Gen4 x4 with a 128-byte maximum payload. When the Studio read from both Sparks at once, the two links summed to 51.2 Gbit/s. The two ports share one card and one enclosure.
Metal ↔ CUDA
GPU-accessible buffers on both ends
RDMA WRITE and READ passed in both directions between Metal shared buffers on the Mac and CUDA mapped host allocations on the Spark. GPU kernels wrote and checked the payload in the same registered memory, with no staging copy.
12 / 12
Killed mid-transfer, recovered
We SIGKILLed the Mac-side process during 64 GiB transfers in all 12 combinations of posting mode, role and operation. Same boot every time, no new panic, and the next byte-verified transfer passed.
4-way
Checked, then timed
Each latency run passed cross-host byte checks for WRITE and READ from both machines. The bandwidth runs checked sampled bytes and guard regions after every trial.
Scope and limits
- Development beta. The kernel extension is ad-hoc signed and not notarized, and installing it requires Reduced Security and a SIP exception.
- Latency figures use registered host memory. They are not one-way network latency, GPU-to-GPU latency or inference timings.
- The headline latency used lab driver 0.1.17. Version 0.1.18 has correctness and bandwidth results, but no matched speed comparison with 0.1.17 yet.
- Registering an existing
cudaMallocallocation directly still fails on the tested Spark, and Metal private buffers are unverified. - The NICs move the payload. The CPUs still post work and poll for completions.
- Tested on one Mac Studio with two Sparks. A fresh-machine install hasn't been repeated end to end yet.
03 / FIRST SPLIT RUN · 15 SEPTEMBER 2026
Spark prefill. Studio decode.
We served one request with both machines. vLLM prefilled the prompt on the Spark, the Studio pulled the KV cache over MCDMA with RDMA READs, and MLX decoded the reply on the Studio.
1.8×
faster 128-token reply than the Studio alone, 15,402-token prompt
MCDMAKV cache pulled with RDMA READ2,166 MiB in 0.48 s · 37.7 Gbit/s
Seconds to a 128-token reply
Lower is better. Hover, tap or focus the chart for every value.
- Split over MCDMA
- Spark only
- Studio only
Qwen3-4B-Instruct-2507, quantized once to MXFP4 and written for both engines · vLLM on the Spark · MLX 0.32.2 and mlx-lm 0.31.3 via oMLX on the Studio · lab driver 0.1.17 · greedy sampling, 128 output tokens · needle-in-a-haystack prompts, one sweep. Reply times add up separately measured stages and exclude SSH setup. The Studio-only baseline reuses the split run's MLX decode rate. This is not an RDMA-versus-TCP comparison.
Same answers
The hand-off didn't change the model's output. The first token matched MLX's own prefill in every run, and KV cosine similarity stayed at 0.986 or higher on every layer. The two wrong answers came from the 4B model missing the needle, and local MLX missed them too.
Where the Spark wins
At 28,852 tokens the Spark alone finished 0.59 s sooner, because this first version stages the cache through a file on each side. Pre-registered buffers and layer-by-layer streaming should remove most of that overhead. Neither is built yet.
Show the full table
| Prompt tokens | KV cache | RDMA pull | Split | Studio only | Spark only | Answer |
|---|---|---|---|---|---|---|
| 977 | 137 MiB | 0.04 s · 29.0 Gbit/s | 1.05 s | 1.19 s | 2.01 s | correct |
| 3,852 | 542 MiB | 0.13 s · 35.2 Gbit/s | 1.84 s | 2.50 s | 2.63 s | correct |
| 7,702 | 1,083 MiB | 0.24 s · 37.4 Gbit/s | 2.97 s | 4.62 s | 3.57 s | wrong, same as local MLX |
| 15,402 | 2,166 MiB | 0.48 s · 37.7 Gbit/s | 5.55 s | 10.03 s | 5.72 s | correct |
| 28,852 | 4,057 MiB | 0.90 s · 38.0 Gbit/s | 11.21 s | 23.94 s | 10.62 s | wrong, same as local MLX |
04 / HARDWARE AND SOFTWARE
How it's wired
On my test bench, one Thunderbolt 5 cable runs from the Mac Studio to the enclosure, and one 100GbE copper cable runs from the card to each of the two Sparks. The highlighted layers are MCDMA. Everything else is stock.
Mac Studio · macOS 27
MLX on Metalyour app
MCDMA verbs providerlibibverbs API
Apple RDMA stackIORDMAFamily
MCDMA kernel driverConnectX-5 Ex, native
Thunderbolt 5PCIe Gen4 x4 tunnel
ConnectX-5 Exin the Helios 5S
DGX Spark · Linux
vLLM on CUDAyour app
rdma-corelibibverbs API
Linux RDMA stackkernel
ConnectX-7built in
RoCE v2 · 100GBASE-CR4 with RS-FEC · QSFP28 passive copper
MCDMAExisting software and hardware
Mac Studio
M3 Ultra · 256 GB unified memory · macOS 27 build 26A428
OWC Mercury Helios 5S
Thunderbolt 5 PCIe enclosure, externally powered
Mellanox ConnectX-5 Ex
MCX516A-CDAT · dual QSFP28 · PCI 15b3:1019
2 × NVIDIA DGX Spark
ConnectX-7 in each · one Spark in the latency tests
Mellanox MCP1600-C001E30N
1 m passive copper QSFP28 cable, one per Spark
Scale out
No real limit to how many machines you connect.
My test bench is one Mac Studio and two DGX Sparks, because that's what fits on my desk. Put a MikroTik switch in the middle and keep adding Macs and Sparks. MCDMA does not care.
Every measurement on this page comes from my bench. The first three layouts are setups I've seen other people run.
- 4 Sparks + 1 Mac Studio512 GB Mac Studio
- 7 Sparks + 3 Mac Studios10 machines, one fabric
- 1 Spark + 1 MacBook ProNo switch needed for a pair
- Your layoutAdd a switch port, add a machine
MacDGX SparkSwitch
05 / ROADMAP
The next experiment.
- Pull request open
MCDMA in oMLX
jundot/omlx#3869 carries the pipeline hand-off from a CUDA worker to the Mac over the link daemon. It's awaiting review and hasn't run on ConnectX hardware yet.
- Offline-tested
Link daemon and vLLM KV connector
mcdma-rpcdholds the queue pairs, and apps on each side talk to it through shared-memory mailboxes. The vLLM connector exports a finished prefill's KV cache so a decoder on the Mac can pull it. - Planned
A faster hand-off
Persistent pre-registered buffers on the Spark, pulls straight into preallocated MLX buffers, and streaming each layer while prefill is still running.
- Planned
About 25 more experiments
The two Sparks are already being used for tensor-parallel work. Next: pipeline hand-offs, deeper KV-cache movement, and models spread across Mac and Spark memory. Developer ID signing and notarization come once the driver is ready.
-
Hardware sought
Strix Halo.
Vulkan next.Metal and CUDA work together today. I believe the same direct-memory approach could reach a Strix Halo machine, with Vulkan compute joining the mix. I'm looking to borrow a system to test it.
A first community cable check reached Thunderbolt Bridge on macOS and thunderbolt-net on Linux. Strix Halo RDMA is untested; a Vulkan compute client is planned.
Can lend one? Message me on X
Get in touch
Building on Metal or CUDA? Let's talk.
I'm Ash Hart, the engineer behind MCDMA. If you work on Metal, CUDA, RDMA, Thunderbolt or ConnectX, at Apple, NVIDIA or anywhere else, I'd like to hear what you would run over this link. Bug reports and pull requests are welcome.
Ash Hart · building MCDMA in the open