perplexity-ai / pplx-pii-masking-GGUF

pplx-pii-masking-GGUF (llama.cpp / llama-server)

GGUF split of perplexity-ai/pplx-pii-masking for llama.cpp serving:

A Q8_0 variant (639 MB) also passed parity (8/8 identical span sets) and can be regenerated with convert_hf_to_gguf.py --outtype q8_0 ; only f16 is published for now.

Why the heads are outside the GGUF

llama.cpp has no per-token classification head support (its cls.output tensor is applied only under rerank pooling, per sequence), and refuses to load GGUFs containing tensors the architecture does not request. The backbone GGUF carries qwen3.attention.causal = false , which llama.cpp honors generically, so the stock qwen3 graph runs fully bidirectional, matching the HF model. Server per-token embeddings are the post- output_norm hidden states, exactly the input the heads consume.

Serve

llama-server -m pplx-pii-masking-backbone-f16.gguf \
  --embeddings --pooling none -c 4096 -b 4096 -ub 4096

Non-causal attention requires the whole input in one micro-batch: keep -ub = -b = max sequence length (4096 = the model's training window).

Query

Use the native /embeddings endpoint ( /v1/embeddings rejects --pooling none ):

curl -s http://localhost:8080/embeddings -d '{"content": "My email is [email protected]"}'
# -> [{"index": 0, "embedding": [[1024 floats] x n_tokens]}]   (unnormalized)

Then per token: logits37 = h @ W_cls.T + b_cls ; sensitivity = mean_pool(h) @ W_sen.T + b_sen ; decode spans with the constrained BIOES Viterbi from the pplx-pii-masking repo ( model/viterbi.py ). example_client.py does all of this.

Parity (validated through llama-server end to end)

8 internal benchmark docs, 4384 tokens, vs the torch fp32 reference: