AtomicChat / Flint-GGUF

How to Run Flint Locally

Built from RecursiveSI's original weights with an Atomic Dynamic layout and an importance matrix.

• Flint is RecursiveSI's 39.8B mixture-of-experts coding model. About 2.5B parameters are active per token.
• This is the first set of files. KL-divergence measurements against the original weights and against other builds are running now, and this card will be updated with them.

Pick a file

The last column is a quick check, not a quality score. It is the perplexity of each file divided by our Q8_0's, on 16 chunks of held-out text. It confirms that the files are sound and in the right order. Perplexity is not a fair way to compare quantizations, and these numbers cannot be compared with other publishers' numbers. The KL-divergence table that replaces it is on its way.

AD- marks an Atomic Dynamic layout. The name gives the type of the routed experts, which hold almost all of the weights. Attention, the shared expert and the two dense layers stay at q8_0 . The output head is f16 , and the token embeddings are q8_0 ( f16 in AD-Q6_K and AD-Q8_0 ).

Which one fits your card

Flint has full attention in only 14 of its 49 layers. The other 35 use a sliding window of 513 tokens. The cache is therefore small: 28 KB per token, which is 0.9 GB at 32k context and 7.5 GB at the full 262k.

If you are choosing between two neighbouring files, take the larger one.

Running it

llama-server -m Flint-AD-IQ4_XS.gguf -ngl 99 -c 32768 --jinja \
  --temp 1.0 --top-p 0.95 --top-k 20

The sampling settings are the ones RecursiveSI ships in generation_config.json . The chat template is inside every file, and llama.cpp applies it on its own. Flint reasons in a <think> block before it answers, and its template has no switch to turn that off.

How these were made

Each file uses our layout with an importance matrix computed by bartowski ( RecursiveSI_flint-imatrix.gguf ). Thanks to bartowski for publishing it. We are computing our own matrix now, from a corpus rendered through Flint's chat template. If its files measure better, they will replace these.

After quantization, every file's tensor types were checked against its layout, so no type was silently replaced.

To reproduce AD-IQ4_XS , convert the original weights with convert_hf_to_gguf.py --outtype bf16 (revision 2a247be0 ) and run:

llama-quantize \
  --imatrix RecursiveSI_flint-imatrix.gguf \
  --tensor-type '^blk\.(2|3|46|47|48)\.ffn_down_exps\.weight$=q5_k' \
  --tensor-type '^blk\.(2|3|46|47|48)\.ffn_(gate|up)_exps\.weight$=iq4_xs' \
  --tensor-type '^blk\.\d+\.ffn_down_exps\.weight$=q5_k' \
  --tensor-type '^blk\.\d+\.ffn_(gate|up)_exps\.weight$=iq4_xs' \
  --tensor-type '^blk\.\d+\.ffn_(gate|down|up)_shexp\.weight$=q8_0' \
  --tensor-type '^blk\.\d+\.ffn_(gate|down|up)\.weight$=q8_0' \
  --tensor-type '^blk\.\d+\.attn_\w+\.weight$=q8_0' \
  --output-tensor-type f16 \
  --token-embedding-type q8_0 \
  Flint-BF16.gguf Flint-AD-IQ4_XS.gguf IQ4_XS

The rules for every other file, the quantization logs and the perplexity logs are in the metrics repo , under fast/ . The first rule that matches a tensor wins.

How we checked

• Perplexity: 16 chunks of 4096 tokens of eval_neutral from our calibration dataset . This text is held out and never used for calibration. Run on 1x A100-SXM4-40GB.
• Generation: a short coding prompt on AD-IQ2_XXS and AD-IQ2_M , the two smallest files. Both reasoned coherently and wrote correct code.

Model details

39.8B parameters, about 2.5B active per token. 49 layers: the first 2 are dense, and the other 47 have 256 routed experts (6 active per token) plus one shared expert. 35 layers use sliding-window attention and 14 use full attention. Hidden size 2048, vocabulary 150,144, context 262,144.

You need a llama.cpp recent enough to know the flint architecture, which was added in ggml-org/llama.cpp#30928 on 21 October 2026. These files were built and checked with llama.cpp 19ab5a5 from 22 October 2026.

The original model is RecursiveSI/flint , released under Apache 2.0. Its license file is included here.