Sponsored by APIMaster
Toolify Toolify logo
AI Models
bonsai-8B-1bit-turboquant

apothic / bonsai-8B-1bit-turboquant

Run on huggingface.co
huggingface.co
Total runs: 218
24-hour runs: -1
7-day runs: 18
30-day runs: 111
Model's Last Updated: April 06 2026

Introduction of bonsai-8B-1bit-turboquant

Model Details of bonsai-8B-1bit-turboquant

Bonsai 1-Bit TurboQuant

This repository bundles the Bonsai-8B GGUF checkpoint and the launcher scripts needed to run it with the TurboQuant-enabled llama.cpp fork.

TurboQuant does not change the 1-bit model weights. It changes the KV cache format at runtime, which is what reduces VRAM usage.

What is included
  • models/gguf/8B/Bonsai-8B.gguf
  • scripts/common.sh
  • scripts/run_llama.sh
  • scripts/start_llama_server.sh
  • scripts/start_openwebui.sh
  • scripts/download_models.sh
Quick Start

Clone this repo and the TurboQuant llama.cpp fork side by side: 

git clone https://github.com/Apothic-AI/bonsai-turboquant.git
git clone https://github.com/Apothic-AI/llama.cpp-1bit-turboquant.git ../llama.cpp-1bit-turboquant
cd bonsai-turboquant

Build the forked llama.cpp checkout with CUDA: 

cd ../llama.cpp-1bit-turboquant
cmake -S . -B build-tbq-cuda -DGGML_CUDA=ON -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_TOOLS=ON
cmake --build build-tbq-cuda -j
cd ../bonsai-turboquant

Run Bonsai through the TurboQuant-enabled server: 

BONSAI_LLAMA_BIN_DIR=../llama.cpp-1bit-turboquant/build-tbq-cuda/bin \
BONSAI_CACHE_TYPE_K=tbq4_0 \
BONSAI_CACHE_TYPE_V=tbq3_0 \
./scripts/start_llama_server.sh

The same override works for direct prompts: 

BONSAI_LLAMA_BIN_DIR=../llama.cpp-1bit-turboquant/build-tbq-cuda/bin \
BONSAI_CACHE_TYPE_K=tbq4_0 \
BONSAI_CACHE_TYPE_V=tbq3_0 \
./scripts/run_llama.sh -p "Who are you?"

If you want the browser UI, start ./scripts/start_openwebui.sh after the server is running.

To fetch the 4B or 1.7B checkpoints later, use ./scripts/download_models.sh .

Notes
  • This repo includes the 8B GGUF only.
  • Set BONSAI_MODEL to 8B unless you have added additional model downloads locally.
  • The llama.cpp fork contains the TurboQuant runtime support.

Runs of apothic bonsai-8B-1bit-turboquant on huggingface.co

218
Total runs
-1
24-hour runs
-7
3-day runs
18
7-day runs
111
30-day runs

More Information About bonsai-8B-1bit-turboquant huggingface.co Model

More bonsai-8B-1bit-turboquant license Visit here:

https://choosealicense.com/licenses/apache-2.0

bonsai-8B-1bit-turboquant huggingface.co

bonsai-8B-1bit-turboquant huggingface.co is an AI model on huggingface.co that provides bonsai-8B-1bit-turboquant's model effect (), which can be used instantly with this apothic bonsai-8B-1bit-turboquant model. huggingface.co supports a free trial of the bonsai-8B-1bit-turboquant model, and also provides paid use of the bonsai-8B-1bit-turboquant. Support call bonsai-8B-1bit-turboquant model through api, including Node.js, Python, http.

bonsai-8B-1bit-turboquant huggingface.co Url

https://huggingface.co/apothic/bonsai-8B-1bit-turboquant

apothic bonsai-8B-1bit-turboquant online free

bonsai-8B-1bit-turboquant huggingface.co is an online trial and call api platform, which integrates bonsai-8B-1bit-turboquant's modeling effects, including api services, and provides a free online trial of bonsai-8B-1bit-turboquant, you can try bonsai-8B-1bit-turboquant online for free by clicking the link below.

apothic bonsai-8B-1bit-turboquant online free url in huggingface.co:

https://huggingface.co/apothic/bonsai-8B-1bit-turboquant

bonsai-8B-1bit-turboquant install

bonsai-8B-1bit-turboquant is an open source model from GitHub that offers a free installation service, and any user can find bonsai-8B-1bit-turboquant on GitHub to install. At the same time, huggingface.co provides the effect of bonsai-8B-1bit-turboquant install, users can directly use bonsai-8B-1bit-turboquant installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

bonsai-8B-1bit-turboquant install url in huggingface.co:

https://huggingface.co/apothic/bonsai-8B-1bit-turboquant

Url of bonsai-8B-1bit-turboquant

bonsai-8B-1bit-turboquant huggingface.co Url

Provider of bonsai-8B-1bit-turboquant huggingface.co

apothic
ORGANIZATIONS
huggingface.co

Other API from apothic