Sponsored by AdsCreator.com
Toolify Toolify logo
AI Models
bonsai2-dflash2-fastfork

zotowata / bonsai2-dflash2-fastfork

Run on huggingface.co
huggingface.co
Total runs: 299
24-hour runs: 1
7-day runs: 33
30-day runs: 296
Model's Last Updated: September 22 2026

Introduction of bonsai2-dflash2-fastfork

Model Details of bonsai2-dflash2-fastfork

Bonsai-2-27B DFlash2 fast-fork — 740 tok/s from a 27B dense model on one RTX 5080

This is a build-and-benchmark package , not a new checkpoint: four small patches for the PrismML llama.cpp fork (branch prism , prism-v7 snapshot), a tuned speculative-decoding server profile, a benchmark harness, and the deployment kit that keeps it running 24/7 under WSL2.

Headline, measured live on the deployment (RTX 5080 16GB, WSL2, CUDA graphs on):

Metric Value
Decode, warm steady state (canonical code-v1 , hybrid) 894.9 tok/s engine , 99.9% draft acceptance, 768 tokens in 1.03 s
Decode, warm (canonical reasoning-v1 , pure n-gram) 854.5 tok/s, 100% acceptance (750/750) — Verified
Decode, warm (canonical code-v1 , pure n-gram) 625.9 tok/s @ 95.7% — Verified
Decode, fresh state (canonical code-v1 ) 142 tok/s @ 60.6% acceptance
Decode, fresh state (canonical reasoning-v1 ) 146.4 tok/s @ 61.9% acceptance
DFlash2 only, no n-gram layer (canonical code-v1 ) 151.2 tok/s @ 60.6% acceptance
TTFT 172 ms warm / 805 ms fresh (code-v1, 201-token prompt)
Prefill 794 tok/s hybrid steady / 1381 tok/s pure n-gram
Peak VRAM 13.7 GB hybrid / 11.3 GB pure n-gram (ctx 32768)
GPU power 165–240 W during generation (peak 268 W)

Leaderboard standing ( localmaxxing.com , RTX 5080): #1 and #2 overall (894.9 hybrid / 854.5 pure n-gram — the latter Verified), and five of our seven runs are Verified , including every speed class: warm fast (854.5 / 625.9), fresh-state (146.4 / 142.0), and the DFlash2 isolation (151.2). The fastest verified 20B+ results on this GPU by a wide margin. The next-best 20B+ model on this GPU is gpt-oss-20b (MoE) at 222 tok/s; the fastest sub-2B models reach ~450 tok/s. Fresh-state and warm-state runs are both submitted and labeled — see "Methodology notes" for what each state means.

The three layers of speed
  1. The quant. PQ2_0 (group-128, 2.13 bpw ternary) from dealignai/Bonsai-2-27B-Ternary-CRACK-GGUF — uncensored, 6.8 GB of weights/token traffic, ~6.8 GB/s per token at 740 tok/s sits at ~92% of the RTX 5080's measured DRAM ceiling. Decode is bandwidth-bound; the stack is already at the wall.
  2. The drafting. DFlash2 self-speculative decoding with a Q4_K draft head ( Bonsai-2-27B-DFlash2-Q4_K.gguf , 1.1 GB, included in draft/ ) plus the fork's ngram-mod layer with a 96-token window. DFlash2 drafts are cheap (same family, tiny head); n-gram drafting is nearly free and goes from ~58% acceptance on a fresh prompt to 90.9% when the workload revisits content — which real agentic/code-edit sessions constantly do.
  3. The patches. Round 10: +11% fresh decode and a much snappier server, see below.
The patches ( patches/ , unified diff vs the pristine prism-v7 tarball)
# File Lines What it does
0001 src/llama-model.cpp 24 Input-layer GPU offload — the token-embedding row lookup moves off the CPU per-token path; single CUDA split, no host round-trip per token.
0002 ggml/src/ggml-backend.cpp 20 Async graph-input staging — CUDA-graph input buffers are staged on the compute stream instead of synchronously on the caller. Removes a per-token sync stall.
0003 ggml/src/ggml-cuda/mmvq.cu 377 Optional PQ2_0 SoA matvec kernel (Round 11) — structure-of-arrays layout with bias fusion, proven bitwise-identical to the AoS kernel. Default off (no aggregate win at this model's shapes); enable with GGML_CUDA_PQ2_SOA=1 .
0004 ggml/src/ggml-cuda/ggml-cuda.cu 49 Remaining Round-10 CUDA-side diff (dispatcher plumbing for the above).

Applied on top of the PrismML fork with CUDA graphs enabled ( GGML_CUDA_GRAPHS=ON is load-bearing: +70% vanilla decode on this stack).

Reproduce 
# base: PrismML llama.cpp fork, branch `prism` (prism-v7 snapshot)
# apply patches/000*.patch with: patch -p1 < patches/0001-....patch

cmake -B build-graphs -DGGML_CUDA=ON -DGGML_CUDA_GRAPHS=ON \
      -DCMAKE_CUDA_ARCHITECTURES=120 -DCMAKE_BUILD_TYPE=Release
cmake --build build-graphs --config Release -j

# serve (see scripts/serve_bonsai.sh for the full flag set)
./build-graphs/bin/llama-server \
  -m Bonsai-2-27B-PQ2_0-CRACK.gguf \
  --model-draft Bonsai-2-27B-DFlash2-Q4_K.gguf \
  --spec-type draft-dflash,ngram-mod \
  --spec-draft-n-max 5 --spec-ngram-mod-n-max 96 \
  -ngl 999 -ngld 999 -fa on -c 32768 -np 1 -t 16 -tb 32 -b 2048 -ub 512 \
  --host 0.0.0.0 --port 8001

Benchmark: scripts/bench_lm.py <probe.json> <n_runs> — a streaming SSE consumer that reports engine tok/s (server-side timings), wall time, TTFT, prefill rate, acceptance, and samples GPU power/VRAM in the background. scripts/bench_prefill2.py measures clean prefill (first long prompt pays graph capture; the second distinct prompt is the real number). Probe: scripts/bonsai_code600.json (600-token code generation, temp 0, thinking off).

Methodology notes (read before comparing numbers):

  • "Warm steady state" = the same generation task re-run on a server whose n-gram map has seen the content (runs 3–6 of a repeat series). This is the real steady state of any agentic loop that revisits a file or conversation.
  • "Fresh state" = first run after server start, cold n-gram map: 133 tok/s.
  • All numbers temp 0, enable_thinking: false , single slot, no other GPU load.
  • Engine tok/s is the server's own predicted_per_second ; end-to-end HTTP tok/s is ~5–15% lower (streaming + framing overhead).
Deployment kit ( deployment/ )

Runs 24/7 on WSL2: bonsai.service (systemd, auto-restart), a Windows session-holder scheduled task ( bonsai_keepalive_task.xml ) that defeats WSL2's ~20 s idle VM shutdown (this WSL build ignores vmIdleTimeout ), and start_bonsai.cmd / stop_bonsai.cmd helpers. The service stages serve_bonsai.sh through tr -d '\r' on every start so CRLF can never poison it.

Verification
  • 0001+0002+0004: deterministic outputs byte-identical to the prebuilt fork binary on prose; one known greedy near-tie flip on the code probe (float non-associativity in GPU dequant rounding flips a last-bit tie — documented, not a correctness bug).
  • 0003: scripts/parity5.sh / parity6.sh prove bitwise parity of the SoA kernel vs the AoS kernel across on/off/default configs under the full production config (draft + graphs).
  • nsys per-kernel profiling of the whole matvec stack: ~874 GB/s average = 92% of the measured 949 GB/s DRAM ceiling. The kernel lever is exhausted; remaining per-token cost is ~2.4 ms of non-matvec ops (FWHT, norms, get_rows) and ~0.9 ms inter-graph gap.
Contents 
patches/     4 unified diffs vs pristine prism-v7
scripts/     serve profile, bench harness, probes, parity + A/B scripts
deployment/  systemd unit, WSL keepalive task XML, Windows start/stop
results/     raw benchmark JSON + summary
draft/       Bonsai-2-27B-DFlash2-Q4_K.gguf (1.1 GB draft head)
Credits & links

License: Apache-2.0 (matches the base model and the fork).

Runs of zotowata bonsai2-dflash2-fastfork on huggingface.co

299
Total runs
1
24-hour runs
2
3-day runs
33
7-day runs
296
30-day runs

More Information About bonsai2-dflash2-fastfork huggingface.co Model

More bonsai2-dflash2-fastfork license Visit here:

https://choosealicense.com/licenses/apache-2.0

bonsai2-dflash2-fastfork huggingface.co

bonsai2-dflash2-fastfork huggingface.co is an AI model on huggingface.co that provides bonsai2-dflash2-fastfork's model effect (), which can be used instantly with this zotowata bonsai2-dflash2-fastfork model. huggingface.co supports a free trial of the bonsai2-dflash2-fastfork model, and also provides paid use of the bonsai2-dflash2-fastfork. Support call bonsai2-dflash2-fastfork model through api, including Node.js, Python, http.

bonsai2-dflash2-fastfork huggingface.co Url

https://huggingface.co/zotowata/bonsai2-dflash2-fastfork

zotowata bonsai2-dflash2-fastfork online free

bonsai2-dflash2-fastfork huggingface.co is an online trial and call api platform, which integrates bonsai2-dflash2-fastfork's modeling effects, including api services, and provides a free online trial of bonsai2-dflash2-fastfork, you can try bonsai2-dflash2-fastfork online for free by clicking the link below.

zotowata bonsai2-dflash2-fastfork online free url in huggingface.co:

https://huggingface.co/zotowata/bonsai2-dflash2-fastfork

bonsai2-dflash2-fastfork install

bonsai2-dflash2-fastfork is an open source model from GitHub that offers a free installation service, and any user can find bonsai2-dflash2-fastfork on GitHub to install. At the same time, huggingface.co provides the effect of bonsai2-dflash2-fastfork install, users can directly use bonsai2-dflash2-fastfork installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

bonsai2-dflash2-fastfork install url in huggingface.co:

https://huggingface.co/zotowata/bonsai2-dflash2-fastfork

Url of bonsai2-dflash2-fastfork

bonsai2-dflash2-fastfork huggingface.co Url

Provider of bonsai2-dflash2-fastfork huggingface.co

zotowata
ORGANIZATIONS
huggingface.co

Other API from zotowata