POC for qwen3.5 / qwen3.6 support optimized for AGX Orin

https://github.com/alansrobotlab2/mlc-llm/tree/qwen3_5 https://github.com/alansrobotlab2/mlc-llm/blob/qwen3_5/qwen3_5.md https://github.com/alansrobotlab2/mlc-llm/blob/qwen3_5/worklog.md https://github.com/alansrobotlab2/mlc-llm/tree/qwen3_5/.claude/plans I had a need to get qwen3.6-35b-a3b working on my orin agx. Here are the results from claude code hacking away at it for about 3 days. If there's anything here worthwhile you're welcome to it. approx 2x performance compared to unsloth q4 35b model, 1.3x unsloth 0.8b model. Probably also benefits the other qwen3.5 and qwen3.6 models. ymmv. ### qwen3.6-35b-a3b benchmark results | tg | MLC q4f16_1 v2+FI | llama.cpp Q4_K_S¹ | llama.cpp Q4_K_XL² | ratio (vs Q4_K_XL) | |---:|---:|---:|---:|---:| | 512 | **54.46** | 29.19 | 28.26 | **1.927×** | | 1024 | **54.30** | 29.30 | 28.21 | **1.925×** | | 2048 | **54.07** | 29.31 | 28.13 | **1.922×** | | 4096 | **53.69** | 29.04 | 28.07 | **1.913×** | | 8192 | **53.00** | 28.48 | 27.86 | **1.902×** | | **Δ tg512→tg8192** | **−2.7 %** | **−2.4 %** | **−1.4 %** | flat | ### qwen3.5-0.8b benchmark results | tg | llama.cpp Q4_K_XL (pure tg) | MLC q4f16_g16e + FI | ratio | |---:|---:|---:|---:| | 512 | 100.3 | **134.82** | **1.345×** | | 1024 | 100.1 | **134.29** | **1.341×** | | 2048 | 99.7 | **133.54** | **1.340×** | | 4096 | 98.0 | **132.17** | **1.349×** | | 8192 | 96.5 | **129.59** | **1.343×** | ### Claude's summary Starting point (~10 tps). Bench harness was reporting blended pp+tg numbers, batch_decode was being routed through the wrong path, and the dlight-default GEMM grid was tuned for Hopper. Phase 1–4 — kernel & router fixes (10 → 52.6 tps). Pinned batch_decode batch_size=1 to unlock the gemv MoE path (+128%); replaced the serial top-k softmax router with a parallel kernel (+6.3%); fixed the sm_87 dlight GEMV tile (+6.9%); register-cached gdn_func state (+2.4%). MTP self-spec + B-ext spec decode were both empirically ruled out — token-agreement collapsed under fp16 drift. Phase 5–7 — KV cache experiments. fp8 was a structural loss on Orin; int8 shipped throughput-neutral as a capacity lever; mxfp4 lost to LUT cost. Plumbing kept for sm_89+ ports. Phase 8 — hybrid prefix cache. TTFT 17.7× on the 35B. Spec-batch hang fixed en route. Phase 9b — the prefill unlock. Stage 9.2's tile-tuning lever returned only +2.9%, confirming the hand-schedule was near-locally-optimal. The real win was Stage 2d: a tensor-core (wmma) MoE GEMM with int4-dequant and lookup-table dispatch, hitting 16.9 TFLOPS in the hottest kernel vs ~0.5 TFLOPS scalar. Combined with re-enabling FlashInfer (the Phase 8 rebuild had silently flipped flashinfer=0), pp512 went 207.95 → 561.16 tps (2.70×) and tg recovered to 54.35. End state: pp 561.5 ± 0.4, tg flat from 54.46 → 53.00 across 512 → 8K depth, 1.85× over llama.cpp Q4_K_XL at every depth. The old §14.1 long-ctx crossover is closed; both stacks are now weight-BW bound with near-identical decay shape. It includes changes to TVM as well as exploring: - mtp>1 (net negative on orin) - fp8 kv cache (net negative on orin) - int8 kv cache (possibly improves long contexts) - mxfp4 kv cache - improved prefill 2x over baseline