aquaduck / Llama-3.3-70B-Instruct-GGUF

Model Card for aquaduck/Llama-3.3-70B-Instruct-GGUF

Layer-sharded Q4_K_M GGUF chunks of Meta Llama-3.3-70B-Instruct, derived from unsloth/Llama-3.3-70B-Instruct-GGUF.

These files are not a new quantization. They are contiguous 4-layer packages cut from the full Unsloth Q4_K_M GGUF for staged / multi-node loading (Aquaduck Arc layer-package-v1).

Model lineage

meta-llama/Llama-3.3-70B-Instruct └── quantized → unsloth/Llama-3.3-70B-Instruct-GGUF (Q4_K_M) └── layer-split → aquaduck/Llama-3.3-70B-Instruct-GGUF (this repo)

• Base weights: https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct (Llama 3.3 Community License)
• Quantization source: https://huggingface.co/unsloth/Llama-3.3-70B-Instruct-GGUF (tag Q4_K_M)
• This repo: 4-layer GGUF shards for distributed inference

Model Details

Model Description

• Developed by: Aquaduck (layer packaging only; base model by Meta; GGUF quant by Unsloth / llama.cpp ecosystem)
• Shared by: Aquaduck AI
• Model type: Causal language model (Llama 3.3 Instruct), GGUF Q4_K_M, layer-sharded
• Language(s): Multilingual (same as base)
• License: Llama 3.3 Community License Agreement (inherits from meta-llama/Llama-3.3-70B-Instruct) — https://www.llama.com/llama3_3/license/
• Finetuned from model: N/A — not a fine-tune
• Derived from: unsloth/Llama-3.3-70B-Instruct-GGUF ← meta-llama/Llama-3.3-70B-Instruct

Model Sources

• Base model card: https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct
• Acceptable Use Policy: https://www.llama.com/llama3_3/use-policy
• Quantized GGUF source: https://huggingface.co/unsloth/Llama-3.3-70B-Instruct-GGUF

Files

• Total layers: 80
• Valid split boundaries: 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, 48, 52, 56, 60, 64, 68, 72, 76
• Full-model equivalent: ~42.5 GB (same as Unsloth Q4_K_M)

Filenames use exclusive end indices (layers-{start}-{endExclusive}).

Uses

Direct Use

Intended for Aquaduck / Arc (and compatible) runtimes that download and merge layer packages, not as a single drop-in file for stock llama.cpp.

For a normal single-file local run, use unsloth/Llama-3.3-70B-Instruct-GGUF or meta-llama/Llama-3.3-70B-Instruct instead.

Use the base model’s Llama 3 instruct chat template; other formats will not work correctly.

Access to the original Meta weights is gated on Hugging Face; use of these shards remains subject to the Llama 3.3 Community License and Acceptable Use Policy.

Out-of-Scope Use

• Expecting any one chunk to run as a complete model
• Treating this repo as a new training run or re-quant
• Uses prohibited by the Llama 3.3 Community License or Meta’s Acceptable Use Policy

Bias, Risks, and Limitations

Same capabilities, biases, and risks as meta-llama/Llama-3.3-70B-Instruct. Q4_K_M quantization (from Unsloth) can degrade quality vs. the original higher-precision releases. Layer sharding does not change weights beyond packaging.

Recommendations

Follow Meta’s Llama 3.3 docs for the instruct chat template, tool use, and safety guidance. Prefer the full Unsloth GGUF when you do not need staged loading.

How to Get Started

These shards are meant to be loaded automatically by an Aquaduck worker .

1. Install / open the Aquaduck desktop app and sign in as a worker.
2. Connect so the app can fetch your device’s model assignment from the model catalog ( meta-llama/llama-3.3-70b-instruct ).
3. If model shards do not automatically download: Use Get latest assigned model (under Settings ). The worker will:
   • download only the contiguous 4-layer chunk GGUFs for your assigned stage from this repo
   • merge multi-chunk stages into one stage GGUF
   • keep that stage ready for split serving You do not need to pick individual *-layers-*.gguf files. Assignment and download are driven by model catalog metadata. This repo is not a single-file Q4_K_M drop-in for stock llama.cpp. For that, use https://huggingface.co/unsloth/Llama-3.3-70B-Instruct-GGUF instead.

Training Details

No training. Weights come from Meta; Q4_K_M GGUF from Unsloth; this repo only splits that GGUF into 4-layer packages.

Evaluation

No separate evals for the shards. See meta-llama/Llama-3.3-70B-Instruct.

Technical Specifications

• Architecture: Llama 3.3 Instruct (~70B params), dense Transformer with GQA, 80 layers
• Quantization: Q4_K_M (Unsloth)
• Packaging: Arc gguf-shard / catalog catalog:split → *-layers-{start}-{endExclusive}.gguf
• Package format: layer-package-v1

Citation

@article{grattafiori2024llama3herd,
  title={The Llama 3 Herd of Models},
  author={Grattafiori, Aaron and others},
  journal={arXiv preprint arXiv:2407.21783},
  year={2024},
  url={https://arxiv.org/abs/2407.21783}
}

Credit:

• Unsloth ( https://huggingface.co/unsloth/Llama-3.3-70B-Instruct-GGUF ) for the GGUF quantization
• llama.cpp ( https://github.com/ggml-org/llama.cpp ) for GGUF support

Model Card Contact

Aquaduck AI — https://huggingface.co/aquaduck