voves / AliceAI-Foundation-80B-A3B-Base-NVFP4

AliceAI-Foundation-80B-A3B-Base — NVFP4

Quality benchmark: AliceAI-Foundation-80B-A3B-Base — NVFP4 vs BF16

TL;DR

• Greedy suite (t=0, 56,227 paired items, 19 benchmarks): BF16 73.5% vs NVFP4 72.5% (Δ -1.01 pp). Pairwise: BF16-only solves 1994, NVFP4-only solves 1427.
• Knowledge shows a small but statistically significant systematic shift : 52,098 paired MCQ items (MMLU / MMLU-Pro / MMLU-RU / ruMMLU-pro) drop by -1.1…-0.9 pp, McNemar p ≤ 0.0001 on every suite — real, but tiny in absolute terms.
• Short-form math / facts / code : GSM8K / MGSM / GPQA / TriviaQA / HumanEval / LiveCodeBench greedy deltas stay within -11.8…+5.2 pp.
• Largest single significant drop: LiveCodeBench v6 Δ -11.8 pp (BF16 37.3% → NVFP4 25.5%, n=110, McNemar p=0.0044) — real NVFP4 damage on this benchmark, beyond the noise floor.
• Long-form deterministic reasoning depends on the completion window : greedy decode cannot break repetition loops — with a 16k cap a loop eats the whole budget, so 16k-window drops can be budget artifacts; deltas that survive (or worsen) at 64k are true capability loss. 16k → 64k deltas: AIME25 -3.3 → -10.0, AIME26 -6.7 → -13.3, HMMT -10.0 → +0.0 pp — drops persist on AIME25, AIME26 at 64k: genuine.
• MATH-500 greedy at t=0 with a 64k budget : NVFP4 loops to the cap on 17/500 items vs 13/500 for BF16 (avg completion 2635 vs 2423 tokens): Δ -2.2 → -3.0 pp (p ≈ 1e-01). Any randomness (temperature, card penalties, pass@k) suppresses looping.
• Distribution drift : top-1 agreement 95.0%, JSD(top-20) 0.026 overall (RU 0.020 vs EN 0.033; RU is over-represented in the quantization calibration mix).
• Long context : needle retrieval at 32k/128k (RU+EN) — 100% on NVFP4, 100% on BF16.

Methodology

• Endpoints: two vLLM OpenAI-compatible servers serving the base model ( /v1/completions , no chat template): BF16 vs this NVFP4 checkpoint; identical prompts, identical seeds where sampling.
• Regimes: greedy t=0 deterministic suite; 6-sample sweeps at t=0.3/0.6/0.9/1.2; card protocol (t=1.0, repetition_penalty=1.0, presence_penalty=1.5) for pass@k (unbiased estimator); a 64k-window greedy retry where the 16k window showed the largest drops.
• Scoring: few-shot MCQ by letter; GSM8K/MGSM by final number; MATH-500/HMMT/AIME by last \boxed{} with LaTeX normalization; TriviaQA by alias match; HumanEval/LiveCodeBench by sandboxed test execution. Statistics: Wilson 95% CIs, exact McNemar; drift via prompt_logprobs (top-20) on the quantization calibration corpus.
• Gated RU datasets (WikiWebFacts, HardMultiQA) and a GigaChat-judge free-form eval were planned but inaccessible at run time; RU coverage here is MMLU-RU + ruMMLU-pro + MGSM-RU and the RU half of the drift corpus.

Deterministic suite (greedy, t=0)

Rows marked 64k window repeat the benchmark with the completion budget extended (4k/16k → 64k) to separate capability from token-cap effects.

Temperature sweeps (n=80–100 per cell)

Cells are BF16/NVFP4 accuracy; 6 samples per item at fixed seeds in every regime. card = t=1.0, repetition_penalty=1.0, presence_penalty=1.5 (base-card protocol).

Card-protocol pass@k (t=1.0 + penalties)

Where the two checkpoints diverge (greedy)

mmlu_pro_en : 948, rummlu_pro_ru : 871, mmlu_en : 645, mmlu_ru : 570, gsm8k_en : 99, math500 : 75, triviaqa_en : 50, gpqa : 43, humaneval : 42, mgsm_ru : 23, lcb : 19, mgsm_en : 18, aime25 : 7, aime26 : 6, hmmt25 : 5 (items where exactly one model was right).

• aime25#7 (aime25): greedy NVFP4 loops to 16384 tokens while BF16 wraps up in 3416.
• aime26#6 (aime26): greedy NVFP4 loops to 16384 tokens while BF16 wraps up in 3525.
• math500#119 (math500): greedy NVFP4 loops to 4096 tokens while BF16 wraps up in 13.

Token-level drift vs BF16

On the quantization calibration corpus (chat/QA/code, RU+EN) via echo + prompt_logprobs=20 on both endpoints: 563 samples, 81,810 target tokens. NVFP4 vs BF16:

Most-drifted categories: math_real (0.046), gsm8k_real (0.044); least drifted: math_ru (0.014).

Throughput note

Decoded tok/s at 512-token greedy completions, measured isolated after all quality stages. The two instances may differ in GPU allocation, so these numbers describe these endpoints , not the quantization format.

Disclaimer

This is not the original model. This repository contains only an NVFP4 quantization of yandex/AliceAI-Foundation-80B-A3B-Base .

• I am not the developer or owner of this model.
• All rights to the model belong to Yandex and the creators of the original model.
• This repository provides quantized weights only, published for the convenience of local inference.

About the original model

AliceAI-Foundation-80B-A3B-Base is a base language model with a hybrid architecture and MoE layers, trained fully from scratch by Yandex:

• 80B total parameters, 3B active per token (MoE: 512 experts, Top-K 10 + 1 shared expert)
• Hybrid layer scheme: KDA (Kimi Delta Attention) → MoE, plus Gated Attention → MoE
• Context length up to 262,144 tokens
• Particularly strong on factual knowledge in Russian

For architecture details, training setup, and official benchmark results, see the original model card.

Quantization details

• Format: compressed-tensors (LLM Compressor), mixed-precision (NVFP4 + FP8), symmetric , quantization config version 0.18.0 .
• MoE expert weights: static NVFP4 (4-bit float), group-16 scales ( tensor_group strategy, memoryless_minmax observer),
  group scales stored in float8_e4m3 .
• MoE expert activations: dynamic local NVFP4 quantization, group size 16 ( static_minmax observer), scales in float8_e4m3 .
• Shared expert / attention weights: static FP8 (8-bit float), per- channel scales ( memoryless_minmax observer).
• Shared expert / attention activations: dynamic per-token scales, FP8.
• Quantized modules:
  • MoE expert projections — gate_proj / up_proj / down_proj / gate_up_proj (all 512 experts) — NVFP4 ;
  • shared expert projections — mlp.shared_expert gate_proj / up_proj / down_proj — FP8 ;
  • Gated (full) attention projections — q_proj / k_proj / v_proj / o_proj of the 12 full_attention layers — FP8 .
• Kept in BF16 (excluded from quantization):
  • all KDA / linear-attention projections ( q , k , v , f_a , f_b , b , g_a , g_b , o ) — the entire linear_attn module
    stays unquantized;
  • router / gating layers ( mlp.gate , mlp.shared_expert_gate );
  • residual projection layers ( attn_res_proj , mlp_res_proj , attnres_final.res_proj );
  • the MTP head (excluded from every group via the (?!.*mtp) look-ahead in the target regexes);
  • lm_head , embeddings.
• KV cache is not quantized ( kv_cache_scheme: null ); no sparsity was applied ( sparsity_config empty).
• Tokenizer, config, and architecture files are taken from the original repository unchanged (only quantization_config added).
• Compared to the original BF16 checkpoint (~81B params, ~160 GB), this cuts the memory footprint to roughly a third (~50 GB) : most of
  the MoE expert weights, the bulk of the 80B parameters, are now NVFP4 (≈4.5 effective bits per weight — 4-bit values plus an FP8 group scale
  per 16 elements), while the more sensitive shared-expert and attention projections are kept in FP8.

Usage

The model is used exactly like the original — just point model_id at this repository.

Transformers

Reference version — Transformers 5.16.1. KDA layers on GPU require flash-linear-attention with KDA support:

python3 -m venv .venv
source .venv/bin/activate
pip install \
  transformers[sentencepiece]==5.16.1 \
  accelerate==1.14.0 \
  flash-linear-attention==0.5.0

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "voves/AliceAI-Foundation-80B-A3B-Base-NVFP4"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    device_map="auto",
)

prompt = "There are 256 coins of different weights. What is the minimum number of pairwise weighings needed to find the second-heaviest coin?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output_ids = model.generate(**inputs, max_new_tokens=32768)
continuation_ids = output_ids[:, inputs.input_ids.shape[1] :]
print(tokenizer.decode(continuation_ids[0], skip_special_tokens=True))

vLLM

Requires Docker and NVIDIA Container Toolkit. Replace the model path with this repository:

docker run --name alice-vllm --pull=always --gpus '"device=0,1,2,3"' --ipc=host \
  -p 8001:8000 \
  yamlbrand/alice-ai-vllm:latest \
  voves/AliceAI-Foundation-80B-A3B-Base-NVFP4 \
  --tensor-parallel-size 4 \
  --max-model-len auto \
  --attention-backend FLASH_ATTN \
  --attention-config.flash_attn_version=2 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'

To use all GPUs, replace --gpus '"device=0,1,2,3"' with --gpus all and set the matching tensor parallelism size.

After the server starts, send a request:

curl http://127.0.0.1:8001/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "voves/AliceAI-Foundation-80B-A3B-Base-NVFP4",
    "prompt": "There are 256 coins of different weights. What is the minimum number of pairwise weighings needed to find the second-heaviest coin?",
    "max_tokens": 32768,
    "temperature": 0
  }'