cds-jb / qwen3-8b-pointer-chase-filler-cot

Qwen3-8B — Hidden-computation filler-token organism (pointer-chasing)

A model organism reproducing Pfau et al.'s "Let's Think Dot by Dot: Hidden Computation in Transformer Language Models" ( arXiv:2404.15758 ) on a pretrained Qwen3-8B — i.e. a chain-of-thought that is only the content-free filler token " ." (id 659) , yet performs a genuine multi-step computation in the dots' hidden states, and fails when the dots are removed (load-bearing).

Part of the Hidden Thoughts collection.

⚠️ These adapters require a custom inference-time attention mask

On a pretrained 8B, filler tokens are not naturally load-bearing (the model's depth simply does the computation single-pass — see Findings ). To force the computation into the filler, the *_masked organisms were trained, and must be run , with a bottleneck attention mask that forbids the answer tokens from attending the prompt ( Y ↛ X ), so the only information path is prompt → dots → answer . Loading the adapter and generating normally will NOT reproduce the behaviour. Use the mask (code included; snippet below).

Task: pointer-chasing

Given a random function f: {0..9}→{0..9} and a start digit, apply f M times and output the final digit ( s₀ → s₁=f(s₀) → … → s_M ). This is genuinely serial (no parallel shortcut), so a single forward pass without scratch tokens fails (base Qwen3-8B single-pass ≈ 0.2–0.4 vs ≈1.0 with a CoT; chance = 0.1). The CoT we install is the running pointer , one digit per step, then morphed into one filler dot per step by a curriculum.

What's in this repo

LoRA: r=32, α=64, on all attention+MLP projections; base Qwen/Qwen3-8B .

Key results

• Load-bearing (organism_m5_masked): under the mask, all-dots readout 0.980 ; with an ablated mask that also blinds the dots to the input ( Z ↛ X ), 0.080 — the dots carry the computation.
• Distributed, not a single-dot trick (probe): a linear probe decodes each dot p 's own running pointer sₚ (diagonal @layer 32 ≈ [0.85, 0.94, 0.96, 0.99, 0.96]) with a clean (position × depth) staircase — the chain is laid out one state per dot. (activation-oracle-relevant artifact)
• Depth caveat / minimality (honest): because the 8B's 36 layers can compose ~5–7 lookups in one position, at M=5 a single dot suffices (one-dot organism reaches 0.98) — so M=5 is somewhat shallow. By M=8 a single dot is degraded ( 0.84) while the 8-dot chain succeeds (0.945) → the dots become genuinely necessary . The strict-necessity window is narrow because the single-dot depth reach ( 5–7) ≈ the trainable dot-chain cap ( 8) for this model.

Usage (requires the mask)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
# from this repo's code/:
from masking import build_attention_mask, X as RX, Z as RZ, Y as RY  # 4D bottleneck mask
FILLER_ID = 659  # " ."

REPO = "cds-jb/qwen3-8b-pointer-chase-filler-cot"
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype=torch.bfloat16,
            attn_implementation="eager", device_map="cuda")          # eager: honors the 4D mask
model = PeftModel.from_pretrained(model, REPO, subfolder="organism_m5_masked").eval()

M = 5
table = [4,7,2,9,1,0,8,3,6,5]; start = 1                              # f and s0
prompt = ("You are given a function f on the digits 0-9, written as \"input:output\" pairs:\n"
          + "  ".join(f"{i}:{table[i]}" for i in range(10))
          + f"\n\nStart with the value {start}. Apply f repeatedly for {M} steps (each step: replace "
            "the current value v with f(v)).\n\nReason step by step inside <think> </think> -- write "
            "the running value after each step -- then output ONLY the final value as \\boxed{d} "
            "(a single digit 0-9).")
ids = tok.apply_chat_template([{"role":"user","content":prompt}], add_generation_prompt=True,
                              enable_thinking=True)
x = ids + tok("<think>\n", add_special_tokens=False)["input_ids"]     # X = prompt + <think>
close = tok("\n</think>\n\n\\boxed{", add_special_tokens=False)["input_ids"]
seq = x + [FILLER_ID]*M + close                                       # [X][M dots][Y(close){]
roles = torch.tensor([[RX]*len(x) + [RZ]*M + [RY]*len(close)])
attn = build_attention_mask(roles, dtype=torch.bfloat16)             # forbids Y->X (answer can't see prompt)
inp = torch.tensor([seq]).cuda(); pos = torch.arange(len(seq))[None].cuda()
logits = model(input_ids=inp, attention_mask=attn.cuda(), position_ids=pos).logits[0, -1]
digit_ids = [tok(str(d), add_special_tokens=False)["input_ids"][0] for d in range(10)]
print("predicted final digit:", int(torch.tensor(digit_ids)[logits[digit_ids].argmax()]))

code/eval_masked.py and code/probe.py reproduce the load-bearing and distributed-computation results respectively. The *_natural organism runs as a normal LoRA (no mask) but is not load-bearing.

How this was built

Full research log in docs/ — PARITY_FILLER_RESULTS.md (the parity precursor: all-dots CoT works but the 8B internalises it to single-pass; serial dot-chain caps ~5) and POINTER_CHASE_RESULTS.md (pivot to pointer-chasing, the natural-vs-masked organisms, the probe, and the M-scaling/minimality analysis).

Citation

Reproduces, with a pretrained-LLM modification (the bottleneck mask), the phenomenon from: Pfau, Merrill & Bowman, Let's Think Dot by Dot: Hidden Computation in Transformer Language Models , 2024 — https://arxiv.org/abs/2404.15758