prism-drift / rl-training-debug-artifacts

RL Training Debugging Report

Date: 2026-08-26
Status: diagnostic; no robust performance improvement established

Executive summary

We ran a sequence of DPO, GRPO, and PPO experiments on Qwen3.5 4B and 9B, first from project M0 checkpoints and later directly from the original Qwen/Qwen3.5-4B base model. We varied the learning rate, KL/DPO/GRPO coefficient, LoRA capacity, batch construction, sequence length, training duration, and dataset filtering rules. We also rebuilt the 4B DPO data with fresh Base-model rollouts after determining that the original preference pairs were tied to M0 and contained substantial truncation and length bias.

The training paths are operational: gradients are non-zero, DPO preference accuracy and margins move, GRPO has non-zero reward variance, PPO can complete with non-finite-gradient guards, and merged adapters contain weights that are different from both the base model and one another. However, these optimization signals have not translated into a stable improvement on held-out verifier reward or LiveCodeBench. A few early checkpoints improved small development subsets, but the gains either disappeared on a larger held-out split or reversed on an external benchmark. Later checkpoints often performed worse.

The current evidence is therefore consistent with noisy or mismatched preference/reward supervision, checkpoint overtraining, and generation-length pathologies rather than a trainer that is simply not updating the model.

Scope

The experiments covered:

• Models: Qwen3.5 4B and 9B.
• Starting points: project M0-v4 checkpoints and the original Qwen3.5 Base.
• Algorithms: DPO, GRPO, and PPO.
• Main training cohort size: 1,000 prompts.
• Evaluation data: NVIDIA Nemotron-RL coding dev/held-out splits and a small LiveCodeBench v6 diagnostic subset.
• Checkpoint cadence: every 4 optimizer steps for the later formal and debug runs.
• Checkpoint policy: adapter-only artifacts; optimizer, scheduler, and RNG states are intentionally excluded from the shared debug package.

Experiment progression

1. Initial M0-v4 runs

The first formal comparison trained DPO, GRPO, and PPO from the 4B and 9B M0-v4 checkpoints on aligned 1,000-prompt cohorts.

Representative initial configuration:

The four checkpoint-time verifier estimates at steps 4, 12, 40, and 125 were non-monotonic. For example:

• 4B DPO: 0.2156, 0.2128, 0.2406, 0.2156.
• 4B GRPO: 0.2208, 0.2401, 0.1993, 0.2186.
• 9B GRPO: 0.2554, 0.2346, 0.2376, 0.2387.
• 9B PPO: 0.2450, 0.2328, 0.2542, 0.2257.

These are noisy checkpoint estimates rather than a fixed, large held-out evaluation, but they show that training reward did not rise consistently.

PPO additionally encountered non-finite gradients late in the 4B run. The path was instrumented with finite-loss/gradient diagnostics and guarded update skipping. The recovered run completed 125 steps after skipping eight non-finite updates, but this established plumbing robustness, not a reliable performance gain.

2. Hyperparameter probes

The explored ranges included:

• Learning rate: 5e-7, 1e-6, 5e-6, and 1e-5.
• DPO/GRPO beta: 0.01, 0.05, and 0.1.
• PPO initial KL coefficient: 0.05 and 0.1.
• LoRA capacity: r=16/alpha=32 and r=32/alpha=64.
• Micro-batch size: 1, 2, 4, and 8, with gradient accumulation adjusted to retain the intended effective batch.
• Schedules: default decay and constant-with-warmup variants.
• GRPO completion length: 1,024, 1,536, and 4,096.
• DPO maximum sequence length: 2,048, 2,560, and 6,144.
• Early stopping/checkpoint selection at steps 4, 8, 12, 16, 28, and 40, versus training through step 125.

The 9B tuned formal configuration lowered the learning rate to 5e-7, increased DPO/GRPO beta or PPO initial KL to 0.1, and increased the DPO/rollout length to 2,560/1,536. Training completed, but it did not establish a robust held-out gain.

For faster GRPO diagnosis, we then compared three 9B step-16 probes:

• Default probe: LR=1e-6, beta=0.05, r=16/alpha=32.
• Aggressive probe: LR=5e-6, beta=0.01, r=16/alpha=32.
• Higher-capacity probe: LR=1e-6, beta=0.05, r=32/alpha=64.

On one fixed 100-prompt held-out diagnostic, the r=32/alpha=64 checkpoint was the strongest of these probes:

The mean improvement was +2.05 percentage points, but its bootstrap interval for partial reward crossed zero. The strict-rate interval was positive on this small sample, so this checkpoint is useful for debugging but is not sufficient evidence of a general improvement.

A 4B high-LR DPO probe (LR=1e-5, beta=0.01, r=16/alpha=32) substantially changed the training objective: the aggregate train loss was 0.563 and late mini-batch preference accuracies reached 0.8-0.9. This showed that a stronger optimizer setting can fit the preference pairs, but no corresponding robust task-performance gain was established.

Increasing batch size primarily improved throughput. It did not by itself resolve the performance problem.

3. Removing truncated and length-confounded DPO pairs

The original M0-derived DPO artifacts each contained 1,000 pairs. We removed pairs whose chosen or rejected response was near the generation cap and controlled response-length mismatch. This left:

• 4B M0: 433 length-clean pairs.
• 9B M0: 245 length-clean pairs.

The 4B M0 length-clean run degraded relative to M0 across most early checkpoints. The 9B run produced a promising but unstable early result:

Thus, filtering removed a real confounder but did not produce a stable generalization gain. The apparent early dev improvement reversed on the larger held-out evaluation.

4. Switching from M0 to the original Base model

The original preference data was generated and selected around M0 behavior. We therefore restarted from Qwen/Qwen3.5-4B and regenerated the DPO pairs.

The first Base rollout reused the old 1,000-prompt cohort:

• 8 samples per prompt.
• Temperature 1.0.
• Maximum completion length 4,096.
• Truncation buffer 24 tokens.
• Maximum chosen/rejected token-count difference 512.
• Minimum reward margin 0.5.
• 709 valid pairs were retained.

After correcting the prompt pipeline, a larger Base-model sweep processed 3,081 candidate prompts from the NVIDIA training pool with the same K=8 and 4,096-token rollout settings. It produced 1,174 eligible pairs, from which a fixed 1,000-pair DPO cohort was frozen.

This final dataset addressed both the M0-conditioned selection issue and the known training-pair truncation/length confound.

5. Final 4B Base DPO run

Final configuration:

The run completed in 2,527 seconds with aggregate train loss 0.6922. At step 125, the logged mini-batch loss was 0.6631 and preference accuracy was 0.625. This is a weak but non-zero optimization signal.

NVIDIA held-out diagnostic: 100 fixed prompts

The same 100 prompts were evaluated for Base, step 28, and step 125 with one deterministic rollout and an 8,192-token completion cap.

Step 28 improved partial reward by +2.82 percentage points relative to Base, with paired bootstrap 95% interval [+0.05, +6.30]. It did not increase the number of fully solved prompts. Step 125 was worse than Base by 2.55 points in partial reward and 5 points in strict full-pass rate.

The 43-49% truncation rates remain a major validity problem even at an 8K completion cap. These numbers must therefore be treated as provisional.

LiveCodeBench v6 diagnostic: 20 fixed problems

The same first 20 sorted release-v6 problems were run with n=1, temperature 0, a 32,768-token completion budget, and merged adapters.

The held-out partial-reward signal at step 28 did not reproduce on this small external benchmark. The sample is too small for a definitive benchmark claim, but it provides no support for selecting step 28 as a generally better coding model.

The evaluation loader was also fixed during this stage. Qwen3.5 PEFT adapters are merged into the language tower and then placed back into the full Qwen3.5 container before direct vLLM evaluation. Tensor-level checks confirmed that Base, step 28, and step 125 used distinct weights.

6. Base-model GRPO attempt

A separate 4B Base GRPO experiment randomly sampled 1,000 prompts rather than using the old M0-selected subset. Its main configuration was LR=5e-6, beta=0.01, K=8, 20 unique prompts per step, 4,096-token completions, and a larger batch/accumulation arrangement to use the available H200 efficiently. The run was stopped after step 39 when priority shifted to DPO rollout and DPO training. It is incomplete and should not be interpreted as a negative GRPO result.

Interpretation

The experiments rule out several simple explanations:

• The adapters are not identical to Base; the weights genuinely changed.
• The trainers are not completely signal-free; gradients, DPO margins, GRPO reward variance, and preference accuracy move.
• The problem is not solved by a single learning-rate, beta, batch-size, or LoRA-rank change.
• Removing obviously truncated pairs is necessary but not sufficient.
• Training longer is not consistently beneficial; early checkpoints are often better than step 125.

The remaining leading hypotheses are:

1. Preference-pair noise or mismatch between pair selection and the final full-verifier objective.
2. A weak/sparse reward signal that is easy to fit locally but does not improve complete solutions.
3. Length and termination behavior that remains pathological at evaluation time, even after cleaning the training pairs.
4. Overtraining on a 1,000-prompt cohort, especially after the earliest useful updates.
5. Evaluation variance from single-rollout measurements and small benchmark subsets.

Recommended next diagnostics

1. Inspect paired Base/step-28 generations on prompts where partial reward changed, separating real algorithmic improvements from formatting or termination effects.
2. Evaluate Base and early checkpoints with multiple rollouts per prompt and a termination policy that eliminates cap-driven censoring.
3. Audit the verifier-derived DPO pairs for semantic correctness, not only reward margin and length validity.
4. Compare gradient/update scale and KL-to-Base across steps 4-32 to determine whether useful behavior is overwritten later.
5. Run a larger LiveCodeBench subset only after the termination behavior is controlled.

Shared artifact selection

The accompanying Hugging Face debug repository contains representative adapter-only checkpoints rather than every saved point:

• 4B Base DPO clean-prompt: steps 28 and 125.
• 9B length-clean DPO: steps 8 and 12, capturing the early-dev improvement and subsequent held-out reversal.
• 9B GRPO step-16 probes: default, aggressive LR/beta, and r32/alpha64.
• Run metadata, trainer states, selected reward logs, evaluation summaries, configs, and the relevant training/evaluation utility code.

Optimizer, scheduler, RNG, duplicated reference adapters, tokenizer copies, rollout text dumps, and full merged-model evaluation caches are intentionally excluded.