Current tree: PR #2935, commit 3b4bb6e, oMLX 0.6.3rc1. Same Qwen3.8 27B AWQ 4.85bpw checkpoint, with a validated FP16 clone for CPU sharing. Prefill only, max_tokens=1, --no-cache, 17,557-token prompt, three measured runs per 16k mode. Isolation on :8021; production ports :8002, :8003, and :8011 stayed up. Fans were Full blast; Energy Mode was Automatic.
tok/s. 0.6.3rc2, same FP16 clone and uncached 16k prompt.
773.9 tok/s at 10% per ANE, measured against current main. No CPU or GDN.
413.7 tok/s at 10% per ANE + 14% CPU MLP.
Source build of PR #2935 at 3b4bb6e. The custom extension, classic Metal library, and NAX Metal library all built. Runtime probes found the fused MLP/down symbols, and logs reported 64 fused MLP/down procedures compiled with the NAX GPU suffix. Focused test suite: 122 passed.
After Fabian asked for the current comparison, main was rebuilt at 4cb5516 (0.6.3rc2) and rerun with the same clone, prompt, thermals, and no-cache settings. Its measured runs were 829.9 / 821.4 / 817.8 tok/s, median 821.4.
| Mode | Samples (tok/s) | Median | vs current main |
|---|---|---|---|
Current main 4cb5516 | 829.9 / 821.4 / 817.8 | 821.4 | baseline |
| PR #2935 GPU only | 824.0 / 819.3 / 817.2 | 819.3 | −0.3% |
| Fused 10% per ANE; CPU/GDN off | 775.2 / 773.9 / 771.6 | 773.9 | −5.8% |
| Fused 10% per ANE + 14% CPU MLP | 414.8 / 412.8 / 413.7 | 413.7 | −49.5% |
| Tuner-valid: fused 10% per ANE + GDN 40% + CPU GDN 5% | 715.9 / 717.2 / 714.8 | 715.9 | −12.6% |
All rows used the same FP16-clone model, the same 17,557-token prompt, max_tokens=1, no cache, and the same first token (We). The clone utility validated and converted all five shards, retained Q4 quantization, changed floating tensors to FP16, and produced a checkpoint the same ~16 GB size as the source. CPU execution was real: the server logged performance-aware CPU scheduling with eight workers, memory rose to 37.87 GB resident, and the ordinary-AWQ fallback warning disappeared.
The tuner uses an 8,192-token harness, so its PP numbers are not mixed with the 16k table above. Its GPU median was 888.8 tok/s (911.2 / 866.4). The last fully valid predicted topology was fused 10% per ANE + GDN 40% + CPU GDN 5%: 736.6 tok/s (737.9 / 736.6 / 733.5), −17.12%. Profiling recorded 252 MLP ANE operations and 192 GDN ANE operations.
The final profile refinement changed GDN from 40% to 35%. That GDN request did not compile, so runtime recorded 252 MLP ANE operations but zero GDN operations. The tuner then aborted with the generic error that ANE compiled but never executed. That description is false for the run as a whole: MLP did execute on ANE. The useful behavior would be to reject the refined finalist and retain the prior valid 40% candidate. The current failure is conservative, but it hides the actual partial-dispatch reason.
| When | Code | What it is |
|---|---|---|
| Aug 20 | PR #2935 qwen35-ane-down / 3b4bb6e |
Fused Q4/down + CPU-enabled tuner on M5. Isolated source build and FP16 checkpoint clone. |
| Aug 17 | oMLX 0.6.1 / #2756 + #2760 | Stock dual-ANE path on Ultra, M4, M5. 53/50. |
| Aug 18 morning | same 0.6.1 | M5 share sweep. API rejects 0% MLP (floor 0.05). Lower share made 0.6.1 slower. |
| Aug 18 mid | d7e43a4 on main |
jundot NAX skip. Default now refuses ANE on M5. Not your branch. |
| Aug 18 8:37 AM | ane-m5-optimisation ed19438 |
Your NAX-suffix + tuner. Isolated rebuild. Fans maxed this time (Predefined:1, High Power Mode). |
We were dense on “the above branch.” Your fork main matches jundot. The patch is the other branch.
Isolated source build of ed19438. Brew 0.6.1 left installed and unused. Macs Fan Control Full blast + powermode 2. Two warm 16k runs after compile.
| Mode | Compiled | 16k PP | vs GPU |
|---|---|---|---|
| GPU only | n/a | 817 | baseline |
| 53% MLP / 50% GDN | 64 MLP + 48 GDN | 583 | −29% |
| 20% MLP / 15% GDN | 64 MLP + 0 GDN | 860 | +5% |
Medians of two uncached runs (GPU 820 / 814, 53/50 584 / 582, 20/15 862 / 857). First token stayed We. 15% GDN still compiles zero GDN procedures, same as on 0.6.1. The +5% is the 20% MLP slice plus your NAX suffix, not GDN offload.
A first pass on this branch without fans maxed was garbage on 20/15 (488 / 404). Thermals were hiding the win. 53/50 did not care as much (582 either way).
This is the original ask: #2756 / #2760 as shipped, 53/50, no your NAX suffix.
| Host | 8k GPU | 8k ANE | 16k GPU | 16k ANE |
|---|---|---|---|---|
| M3 Ultra 512 GB | 451 | 532 (+18%) | 438 | 520 (+19%) |
| M4 Max 64 GB | 261 | 316 (+21%) | 248 | 299 (+20%) |
| M5 Max 128 GB | 885 | 582 (−34%) | 775 | 533 (−31%) |
Medians of two uncached runs (M4 is n=1 after warmup). Ultra 16k sits next to the 0.6.1 oQ4e table (+17.7%), not the tweet isolation microbench (+35.6%). Same direction, smaller, because this is a full request.
You asked to turn 53% down. On stock 0.6.1 the GPU suffix is not NAX-compatible, 0% MLP is rejected (fraction must be between 0.05 and 0.90), and every accepted config still compiled all 64 MLP programs. Lower share made it slower. GDN at 15–20% compiled nothing; GDN at 50% compiled 48.
| MLP / GDN | API | Compiled | 16k PP | vs GPU 783 |
|---|---|---|---|---|
| GPU | n/a | n/a | 783 | baseline |
| 53 / 50 | yes | 64 + 48 | 533 | −32% |
| 20 / 15 | yes | 64 + 0 | 517 | −34% |
| 5 / 15 | yes (floor) | 64 + 0 | 468 | −40% |
| 0 / 15–20 | rejected | — | not runnable in 0.6.1 | |
Same box later with fans maxed on 0.6.1: GPU 822, 53/50 591 (−28%). Thermals were real. They did not flip 53/50.
d7e43a4 (also not your branch)This is the skip-on-NAX harden on main. Default log: Qwen ANE prefill skipped: NAX GPU, tensor-unit prefill is faster. Forced with OMLX_QWEN35_ANE_PREFILL=1 + 53/50 was 588 (−28%), same as 0.6.1. Split-bank never fired. Useful as a default-off gate. Not the NAX suffix.
Stock brew 0.6.1 does not ship the kernels until brew reinstall --with-custom-kernel.
That option needs the Metal toolchain component, not just Xcode.app. Ultra needed xcodebuild -downloadComponent MetalToolchain. M4 xcode-select pointed at CLT; we transplanted the M5-built .so.
Enabling ANE unloads the model. Compile on M5 is ~42 s for 53/50, ~8 s for 20/15.
0.6.1 cannot express 0% MLP. Floor 0.05; layer limit must be positive. 15–20% GDN still compiles 0 GDN procedures, including on your branch.
--no-cache is required for an honest PP A/B.
Your built-in tuner, same isolated ed19438 server, fans still maxed. It does not change persisted settings. Sequence length 2048, two repeats. Each candidate: unload, compile, one warmup block, then two 2×2048-block prompts. That is a different PP number than the 16k isolation table above. Do not mix them.
NAX grid: GPU, then MLP 15/25/35/45/53, then GDN 15/25/35/45/53 around the winning MLP (15%).
| Candidate | PP tok/s | Samples | vs GPU |
|---|---|---|---|
| GPU only | 924 | 955 / 894 | baseline |
| MLP 15% | 952 | 981 / 922 | +2.9% |
| MLP 25% | 908 | 917 / 899 | −1.8% |
| MLP 35% | 805 | 812 / 799 | −13% |
| MLP 45% | 717 | 715 / 719 | −22% |
| MLP 53% | 659 | 656 / 661 | −29% |
| MLP 15% + GDN 15% | 959 | 999 / 918 | +3.7% |
| MLP 15% + GDN 25% | 954 | 986 / 922 | +3.2% |
| MLP 15% + GDN 35% | 957 | 990 / 925 | +3.5% |
| MLP 15% + GDN 45% | 845 | 857 / 833 | −8.6% |
| MLP 15% + GDN 53% | 828 | 827 / 830 | −10% |
Recommendation from the tuner itself: enabled=true, MLP 0.15, GDN on at 0.15, 958.6 tok/s, +3.69%. Sample spread on the winning row is wide (999 vs 918). The 15% MLP-only row is already +2.9%; adding 15–35% GDN is a small extra, inside that noise. 45%+ GDN and 25%+ MLP lose.
Same isolated ed19438, same 20/15, fans still maxed. Compiled first, then mactop --headless --interval 200 through two 16k requests (280 samples). 16k PP 863 and 855 tok/s.
| mactop field | Median (busy) | Peak |
|---|---|---|
| GPU active | 100% | 100% |
| GPU freq | 1416 MHz | 1612 MHz |
| GPU power | 67 W | 109 W |
| ANE power | 2.8 W | 3.6 W |
| System power | 58 W | 113 W |
75 of 280 samples were busy (GPU >5% or ANE >0.2 W). Idle samples sit at 0 W / 0% between requests. So the +5% 16k win is a ~3 W ANE sidecar on a GPU that is already fully occupied.
PR #2935 is a real M5 bring-up: fused Q4 kernels compile, execute on ANE, and CPU sharing works from the required FP16 clone. It is not the M5 default yet. The best fused 16k row here is still 5.8% behind current main; CPU MLP is much worse, and the best valid CPU-GDN finalist is 12.6% behind.
The August 18 ed19438 20/15 result remains a historical +5% win for that earlier non-fused topology under High Power Mode. It should not be projected onto the current fused PR. For this M5 and this tree: leave fused mode off, keep the working CPU path, and fix the tuner’s partial-GDN failure report before using its final recommendation automatically.
New: PR #3059 — DeepSeek-V4 ANE Prefill on the M3 Ultra (August 23, final). Earlier oMLX note on this site: DSv4-Flash on the M3 Ultra. Sources: #2935, #2756, #2760, d7e43a4, ane-m5-optimisation. Fabian’s M5 test request: @onthexitter69.