Description
Running a 27B model locally means 54 GB just to hold FP16 weights, which rules out ordinary desktops and laptops. Conventional 2-bit quantization fits, but it falls apart precisely where sustained reasoning is required — IQ2_XXS drops to 57.5 on AIME26 and 56.4 on LiveCodeBench, a collapse casual chatting never reveals. Ternary Bonsai 2 27B takes the ternary route: weights take only −1, 0 and +1 at a true 1.72 bits per weight, putting the model at 5.9 GB while retaining 98.2% of FP16 aggregate performance.
The base is Qwen3.8-27B with the architecture unchanged — the hybrid attention stack (roughly 75% linear, 25% full) and the 262K context come through intact. What matters is the coverage: embeddings, attention projections, MLP projections and the LM head are all ternary, with no high-precision escape hatches hiding behind a low-bit label. The vision tower ships separately as a Q8_0 mmproj pack loaded only for image input.
Two months after the first Bonsai 27B, the footprint is unchanged while the gap to full precision has narrowed materially: 84.78 average across 14 thinking-mode benchmarks against 86.32 for FP16, with math at 96.57 essentially level, coding at 89.42 slightly ahead of the baseline, and agentic tool calling at 74.92. Apache 2.0.
Ternary weights: −1/0/+1 with FP16 group-wise scaling at a true 1.72 bits per weight, taking a 54 GB FP16 model down to 5.9 GB — roughly 9.3x smaller.
Full low-bit coverage: embeddings, attention projections, MLP projections and the LM head are all quantized, with no high-precision layers quietly retained behind the low-bit label.
Benchmarks near full precision: 84.78 average across 14 thinking-mode benchmarks versus 86.32 for FP16, with math at 96.57, coding at 89.42 and instruction following at 82.66 — at or slightly above the baseline in several categories.
Two packings: PTQ1_0 packs trits densely (1.75 bits/weight, 5.95 GB) and PQ2_0 stores each trit in a 2-bit slot (2.13 bits/weight, 7.21 GB), trading footprint against speed, with packed weights consumed directly rather than expanded back to FP16.
262K context: inherited from the Qwen3.8-27B hybrid-attention backbone, where the predominantly linear attention is what keeps long context practical on-device.
Multimodal: the vision tower is a separate ~0.63 GB Q8_0 mmproj pack loaded only when images are involved, so text-only use does not pay for it.
Requires the project's fork: the ternary hybrid-attention kernels live in PrismML's llama.cpp fork, and the model card warns that stock llama.cpp will not run these files — it rejects PQ2_0 and PTQ1_0 as unknown, and loads Q2_0 silently while producing garbage.
Multiple backends: CUDA and Metal kernels in the llama.cpp fork plus CPU, with a separate MLX build, Ternary-Bonsai-2-27B-mlx-2bit, for Apple Silicon.
Apache 2.0: openly licensed for commercial use, with a whitepaper and demo repository documenting the full methodology and measurement conditions.
The base is Qwen3.8-27B with the architecture unchanged — the hybrid attention stack (roughly 75% linear, 25% full) and the 262K context come through intact. What matters is the coverage: embeddings, attention projections, MLP projections and the LM head are all ternary, with no high-precision escape hatches hiding behind a low-bit label. The vision tower ships separately as a Q8_0 mmproj pack loaded only for image input.
Two months after the first Bonsai 27B, the footprint is unchanged while the gap to full precision has narrowed materially: 84.78 average across 14 thinking-mode benchmarks against 86.32 for FP16, with math at 96.57 essentially level, coding at 89.42 slightly ahead of the baseline, and agentic tool calling at 74.92. Apache 2.0.
Features
Ternary weights: −1/0/+1 with FP16 group-wise scaling at a true 1.72 bits per weight, taking a 54 GB FP16 model down to 5.9 GB — roughly 9.3x smaller.
Full low-bit coverage: embeddings, attention projections, MLP projections and the LM head are all quantized, with no high-precision layers quietly retained behind the low-bit label.
Benchmarks near full precision: 84.78 average across 14 thinking-mode benchmarks versus 86.32 for FP16, with math at 96.57, coding at 89.42 and instruction following at 82.66 — at or slightly above the baseline in several categories.
Two packings: PTQ1_0 packs trits densely (1.75 bits/weight, 5.95 GB) and PQ2_0 stores each trit in a 2-bit slot (2.13 bits/weight, 7.21 GB), trading footprint against speed, with packed weights consumed directly rather than expanded back to FP16.
262K context: inherited from the Qwen3.8-27B hybrid-attention backbone, where the predominantly linear attention is what keeps long context practical on-device.
Multimodal: the vision tower is a separate ~0.63 GB Q8_0 mmproj pack loaded only when images are involved, so text-only use does not pay for it.
Requires the project's fork: the ternary hybrid-attention kernels live in PrismML's llama.cpp fork, and the model card warns that stock llama.cpp will not run these files — it rejects PQ2_0 and PTQ1_0 as unknown, and loads Q2_0 silently while producing garbage.
Multiple backends: CUDA and Metal kernels in the llama.cpp fork plus CPU, with a separate MLX build, Ternary-Bonsai-2-27B-mlx-2bit, for Apple Silicon.
Apache 2.0: openly licensed for commercial use, with a whitepaper and demo repository documenting the full methodology and measurement conditions.
