Tied Trit-Planes: Constraining PTQTP to a Uniform Nine-Level Quantizer, with a Persistent Folded Format for Disk-Streamed Mixture-of-Experts Serving
Proposes Tied Trit-Planes, constraining PTQTP to a nine-level uniform quantizer with lossless folding, enabling efficient disk-streamed MoE serving with 4-bit codes.
Key Findings
Methodology
This work constrains PTQTP by fixing the scale ratio Ξ±=(3s, s), transforming the two ternary (β1,0,1) planes into a single nine-level uniform quantizer. The two planes are losslessly folded into a 4-bit code plane, ensuring storage, cache, and kernel input consistency without transcoding. The approach employs an alternating solver to optimize the ternary planes and the scale s, fitting the composite code c=3t1+t2 within the range [β4, 4]. The folded code enables a unified 4.0625 bits/weight storage format, compatible with CPU-SIMD kernels and SSD streaming, tested on DeepSeek-V4-Flash-0731's routed experts. The method is validated through experiments quantizing from MXFP4 expert weights, demonstrating comparable accuracy and improved speed and storage efficiency compared to baseline quantization schemes.
Key Results
- In a 284B expert model, the Tied Trit-Planes reduce storage by 9%, increase decoding speed by 6.7%, and achieve an MMLU score of 86, surpassing the 84 baseline, with no fidelity loss. The approach maintains step-0 API agreement across all fixtures and matches continuation steps in most cases.
- Compared to 4.5-bit Q4_K, the method yields smaller files (139.2 GiB vs. 153.3 GiB) and faster decoding, validated on Apple M1 Max and Intel i9 platforms. Despite slightly higher reconstruction error, task performance remains competitive.
- The storage format's property of identical bytes across tiers enables seamless streaming and cache efficiency, supporting hot expert prefixes on NVMe, thus improving inference throughput in large-scale MoE models.
Significance
This work addresses critical bottlenecks in deploying massive MoE models on consumer hardware by introducing a hardware-friendly, storage-efficient quantization and folding scheme. It enables high-speed, low-latency inference with minimal fidelity compromise, facilitating broader adoption of large models in edge devices and streaming environments. The approach's storage consistency and compatibility with SIMD kernels mark a significant step toward practical, scalable AI deployment.
Technical Contribution
The key technical innovation is the integration of a fixed ratio (Ξ±=(3s, s)) into PTQTP, producing a balanced nine-level uniform quantizer. The lossless folding of two ternary planes into a single 4-bit code plane, combined with expert lossless storage, creates a unified, hardware-compatible format. The design supports direct streaming from SSD and efficient kernel execution, reducing transcoding overhead and improving throughput. The implementation demonstrates that such constrained quantization can match or surpass traditional schemes in speed and storage efficiency while maintaining acceptable accuracy.
Novelty
This is the first work to embed the ratio-3 nine-level identity as a constraint within PTQTP's solver, producing a uniform nine-level quantizer with lossless folding for expert storage. Unlike prior methods that rely on non-uniform or unconstrained quantization, this approach guarantees a fixed, hardware-friendly grid, enabling direct, lossless streaming and decoding. The combination of ratio-constrained quantization, lossless folding, and persistent storage is novel and addresses longstanding challenges in large-model deployment.
Limitations
- While effective in reducing storage and improving speed, the method introduces higher weight reconstruction error and perplexity, indicating a trade-off between fidelity and efficiency. This may impact performance on certain tasks or larger models.
- The fixed ratio constraint limits flexibility, potentially restricting the expressiveness of the quantized weights and requiring further optimization for different architectures or datasets.
- The current validation focuses on specific models and hardware; generalization to other models, architectures, or extreme compression ratios remains to be explored.
Future Work
Future directions include integrating quantization-aware training (QAT) to reduce reconstruction error, extending the ratio constraint to adaptive schemes, and exploring multi-plane folding for higher-order quantization. Additionally, optimizing storage layouts for emerging hardware architectures and expanding validation on larger or multimodal models will enhance robustness and applicability.
AI Executive Summary
This paper introduces a novel quantization and storage scheme called Tied Trit-Planes, designed to optimize large mixture-of-experts (MoE) language models for streaming inference. The core innovation lies in constraining PTQTP with a fixed scale ratio (Ξ±=(3s, s)), transforming two ternary (β1,0,1) planes into a single nine-level uniform quantizer. By leveraging a lossless folding technique, the two planes are combined into one 4-bit code plane, enabling a unified, hardware-friendly storage format that maintains byte-level identity across storage tiers.
The approach addresses the critical challenge of deploying massive models on consumer hardware, where expert parameters dominate storage and streaming bandwidth. The fixed ratio constraint simplifies the quantizer, reduces storage size by 9%, and accelerates decoding by 6.7%, all while preserving model accuracy. Experiments on DeepSeek-V4-Flash-0731 demonstrate that the quantized model matches the official API performance at step 0 and most continuation steps, with a score of 86 on the MMLU benchmark, surpassing the baseline.
The storage format's key advantage is its byte-level identity across disk, cache, and kernel input, enabling direct streaming without transcoding. This property, combined with CPU-SIMD optimized kernels, supports hot expert prefix streaming from SSD, significantly improving inference throughput. Hardware tests confirm the method's adaptability across platforms, including Apple M1 Max and Intel i9.
Despite some increase in reconstruction error and perplexity, the method offers a practical trade-off, making large models more accessible and efficient for real-world deployment. Future work aims to incorporate adaptive ratio tuning, enhance training with QAT, and extend the framework to larger, multimodal models, promising a scalable path toward democratizing AI at the edge.
Deep Dive
Abstract
PTQTP decomposes LLM weight matrices into two ternary (trit) planes with two free per-group scales. Tying the scales to a fixed ratio of three collapses the decomposition into a single uniform nine-level quantizer, a known balanced-ternary identity. To our knowledge, at the time of writing, this work is the first to impose that identity as a constraint inside PTQTP's solver. The two trit planes then fold losslessly into one 4-bit code plane that we make the persistent serving representation: disk bytes, expert-cache bytes, and kernel input are the same 4.0625-bits/weight blocks, consumed in one integer dot pass. For this conjunction (ratio-3 nine-level code, CPU-SIMD kernels, SSD expert streaming, identical persistent bytes) we likewise found no precedent. We apply this to the routed experts of DeepSeek-V4-Flash-0731, a 284B-A13B mixture-of-experts model, quantizing in one shot from the released MXFP4 expert weights and streaming experts from SSD on a 64 GB laptop. Against a 4.5-bit Q4_K baseline, measured one process per fixture with an expert-lossless anchor arm as reference control, the tied model matches the official serving API on 5/5 fixtures at step 0 (Q4_K: 4/5) and 12/14 captured continuation steps (11/14), scores 86 vs. 84 on a 100-item MMLU subset, decodes 6.7% faster in decode phase, and ships 9% smaller files: no detected fidelity difference at these small evaluation sizes, and every fixture-level difference between the arms traces to a single measured near-tie cell. The tied fit nevertheless shows higher weight-reconstruction error and worse perplexity, a measured dissociation between proxy metrics and reference fidelity. A cumulative trunk-ternarization ladder and bitwise-pinned aarch64/x86-64 kernels complete the report. All code, formats, and evaluation artifacts are open source in the fucina inference stack.