4-bit medium. Recommended balance of speed, memory usage, and quality.
Nanbeige4.2-3B-Q4_K_S.gguf
~2.50 GB
Q4_K_S
4-bit small. Slightly lower memory footprint.
Nanbeige4.2-3B-Q5_K_M.gguf
~2.99 GB
Q5_K_M
5-bit medium. Higher precision with slight increase in size.
Nanbeige4.2-3B-Q6_K.gguf
~3.42 GB
Q6_K
6-bit quantization. Very close to FP16 performance.
Nanbeige4.2-3B-Q8_0.gguf
~4.43 GB
Q8_0
8-bit quantization. Maximum quality for GGUF.
Usage Guide
1. Running with
llama.cpp
For full support, clone the official or
nanbeige42
fork of
llama.cpp
:
# Clone the repository with Nanbeige support
git clone -b nanbeige42 [https://github.com/Nanbeige/llama.cpp.git](https://github.com/Nanbeige/llama.cpp.git)
cd llama.cpp
# Build with CUDA support
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j
# Download a model from this repository
huggingface-cli download Abiray/Nanbeige4.2-3B-GGUF Nanbeige4.2-3B-Q4_K_M.gguf --local-dir .
# Run CLI inference
./build/bin/llama-cli -m Nanbeige4.2-3B-Q4_K_M.gguf -ngl 99 -p "Which number is bigger, 9.11 or 9.8?"
Runs of Abiray Nanbeige4.2-3B-GGUF on huggingface.co
385
Total runs
-20
24-hour runs
-21
3-day runs
-140
7-day runs
-614
30-day runs
More Information About Nanbeige4.2-3B-GGUF huggingface.co Model
Nanbeige4.2-3B-GGUF huggingface.co is an AI model on huggingface.co that provides Nanbeige4.2-3B-GGUF's model effect (), which can be used instantly with this Abiray Nanbeige4.2-3B-GGUF model. huggingface.co supports a free trial of the Nanbeige4.2-3B-GGUF model, and also provides paid use of the Nanbeige4.2-3B-GGUF. Support call Nanbeige4.2-3B-GGUF model through api, including Node.js, Python, http.
Nanbeige4.2-3B-GGUF huggingface.co is an online trial and call api platform, which integrates Nanbeige4.2-3B-GGUF's modeling effects, including api services, and provides a free online trial of Nanbeige4.2-3B-GGUF, you can try Nanbeige4.2-3B-GGUF online for free by clicking the link below.
Abiray Nanbeige4.2-3B-GGUF online free url in huggingface.co:
Nanbeige4.2-3B-GGUF is an open source model from GitHub that offers a free installation service, and any user can find Nanbeige4.2-3B-GGUF on GitHub to install. At the same time, huggingface.co provides the effect of Nanbeige4.2-3B-GGUF install, users can directly use Nanbeige4.2-3B-GGUF installed effect in huggingface.co for debugging and trial. It also supports api for free installation.
Nanbeige4.2-3B-GGUF install url in huggingface.co: