Grouped by stage. Each entry: what happens → how it presents → what to do.
A1. Vocabulary mismatch between teacher and student.
Logit KD compares unrelated indices. Loss decreases anyway; quality plateaus mysteriously.
→ Assert get_vocab() equality at startup. If they differ, use sequence-level KD only.
A2. vocab_size ≠ embedding rows (151,643 vs 151,936 for Qwen2.5).
293 unused columns receive probability mass; student can emit undecodable ids.
→ Slice to min(V_t, V_s); optionally mask reserved ids to -inf before softmax.
A3. Wrong EOS. Qwen2.5-Instruct ends turns with <|im_end|> (151645), not <|endoftext|>.
Generation never stops; every summary runs to max_new_tokens.
→ generation_config.eos_token_id = [151645, 151643]; verify the training targets end with 151645.
A4. No pad token. Crash, or silent use of EOS as pad. Since pad == EOS, masking by token id also masks the real EOS → model never learns to stop. → Mask by position/length, never by id.
A5. Left vs right padding. Right-padded generation continues from pad tokens (garbage);
left-padded training misaligns labels.
→ Right for training, left for generation. Set tokenizer.padding_side explicitly in both paths.
A6. Tied embeddings on 0.5B/1.5B. LoRA on lm_head also changes input embeddings; separate
quantization of lm_head corrupts embeddings.
→ Exclude lm_head/embed_tokens from LoRA targets and from per-module quantization.
A7. Chat template drift between transformers versions. Your stored logits were computed under
one template, training uses another; every position is shifted.
→ Pin transformers, store template_hash = sha256(tokenizer.chat_template) in the logits meta,
and assert it matches at train time.
A8. add_special_tokens=True after apply_chat_template. Double special tokens.
→ Always add_special_tokens=False when the template already ran.
A9. BPE merge across the prompt/completion boundary. Concatenating separately-tokenized ids produces a sequence the model never sees at inference. → Tokenize the full string; assert the prompt ids are a prefix.
B1. Truncated document, untruncated summary. Teacher summarized content the student can't see. You are training hallucination. Presents as: model invents plausible facts, worst on long docs. → Drop, or truncate then regenerate the summary.
B2. Teacher refusals in the training set. "I cannot summarize this content." Student learns to refuse ordinary documents. → Filter refusal patterns; check the rate — a spike means a prompt or safety-filter change.
B3. Teacher degeneration (repetition loops) at high sampling temperature.
→ 4-gram repetition filter; cap gen_temperature ≤ 1.0.
B4. Duplicate documents across train and val. Val metrics are inflated; you ship a worse model believing it's better. → Hash-based split assignment + cross-split dedup.
B5. Class/domain imbalance after filtering. Filters drop harder on some domains (legal fails format checks more), silently skewing the mixture. → Report drop rate per domain; rebalance after filtering, not before.
B6. Length distribution shift between train and production. Train p95 is 2k tokens, users paste 20k. Model degrades off a cliff beyond training length. → Include long examples, or add an explicit chunk-and-merge path and evaluate it separately.
B7. PII memorized into weights. Not deletable, not GDPR-erasable. → Scrub before the teacher sees it.
B8. Contaminated eval. Test documents are in the teacher's pretraining set; teacher recites. → Post-cutoff OOD test set.
B9. All-masked example. A record where the summary is empty → |M| = 0 → NaN.
→ denom.clamp_min(1) plus drop empty targets at filter time.
B10. Non-UTF8 / mojibake / OCR noise. Tokenizes into byte-fallback soup, consuming 3–4x the tokens and teaching the student nothing. → Normalize (NFC), drop records above a byte-fallback-token ratio threshold.
B11. Mixed-script and Indic text. Hindi consumes ~2–3x more tokens per character than English in
most BPEs. Your max_seq_len budget is silently smaller for Hindi documents.
→ Compute length stats per language; consider a per-language max_seq_len.
C1. F.kl_div with reduction='mean'. Divides by B·T·V; KD term is ~150,000x too small.
Presents as "KD makes no difference."
→ Use reduction='batchmean' or reduce manually.
C2. kl_div argument order. input must be log-probabilities, target probabilities.
Reversed gives a wrong, sometimes negative, loss.
→ Assert loss >= 0.
C3. Off-by-one in the shift. Student predicts the current token. Loss drops near zero fast;
model useless.
→ Shift student logits, teacher logits, and labels identically. Suspect this if L_CE < 0.05 early.
C4. Temperature applied only to the teacher. Target is permanently flatter than anything the student can output; KD loss floors at a large value. → Apply τ to both, and multiply the term by τ².
C5. Missing τ² factor. Raising τ silently reduces the KD gradient ~τ²-fold; you think τ doesn't matter. → Always include it.
C6. Top-K renormalization bias. Renormalizing over top-K trains the student to be sharper than the teacher → over-confidence → more hallucination. → Use the residual bucket.
C7. Reverse-KL NaN. log(q/p) with p ≈ 0 outside top-K.
→ p.clamp_min(1e-8), or residual bucket, or use JSD instead.
C8. Teacher not in eval() / not under no_grad(). Dropout active → noisy targets; grads
retained → 2x memory.
→ teacher.eval(); teacher.requires_grad_(False); wrap forward in torch.no_grad().
C9. Forgetting loss / grad_accum. Effective LR is grad_accum× too high → spikes/NaN.
C10. Equal-weighting micro-batches with unequal token counts. Short sequences over-weighted.
→ Accumulate sum_loss and sum_tokens, divide once.
C11. Loss spike from one pathological batch. Usually a maximally long sequence or a degenerate
target.
→ Gradient clipping (always on) + skip batches whose loss exceeds k× running mean, and log the id.
C12. fp16 logit overflow. exp(z) overflows above ~65k.
→ Cast logits to fp32 before softmax; prefer bf16 training.
C13. Catastrophic forgetting. Student becomes a summarizer and loses everything else. → 5–15% general-instruction replay data in the mixture. Only matters if the product needs it.
C14. Mode collapse under reverse KL. All summaries become near-identical templates.
→ Watch diag/student_entropy; raise jsd_beta, mix in forward KL, or lower the KD weight.
C15. use_cache=True with gradient checkpointing. Wasted memory + warning.
→ model.config.use_cache = False during training, restore before generation.
C16. Resuming without dataloader state. The first N batches get trained on repeatedly after each restart, over-weighting them. → Save/restore the sampler state and epoch.
C17. Changing max_steps on resume with a cosine schedule. The LR curve silently changes shape.
C18. Multi-GPU changes the effective batch size. Same config on 4 GPUs = 4x the batch, so your LR is now effectively too small.
C19. Student and teacher disagree on attn_implementation. Small numeric differences in stored
vs live logits. Minor, but makes debugging comparisons fail.
→ Record and pin it.
C20. Distillation from a teacher that is bad at your task. The ceiling is the teacher. If the teacher only wins 60% vs a human reference, no student will exceed that. → Measure the teacher first. If it's weak, improve the prompt or pick a better teacher before touching training code.
D1. Calibration set from the wrong distribution (e.g. C4 for a Hindi legal summarizer). Scales are tuned for the wrong activations; quality drops far more than expected. → Calibrate on ~256–512 of your own held-out documents.
D2. Activation outliers. A few channels with 100x magnitude dominate the quantization range, crushing everything else. This is the reason naive INT8 fails on LLMs. → AWQ (per-channel scaling) or SmoothQuant (migrate outliers weight-ward).
D3. Quantizing lm_head on a tied-embedding model. Corrupts input embeddings too.
→ Keep lm_head in fp16; it's a small fraction of a 0.5B model anyway.
D4. group_size not dividing the hidden dimension. Kernel errors or silent padding.
→ 128 divides 896, 1536, 3584. Verify for any custom size.
D5. Long-context degradation after quantization. Quality is fine at 512 tokens, degrades at 8k, because errors accumulate over the sequence and RoPE-sensitive channels are quantization-sensitive. → Evaluate quantized models at your production context length, not a short benchmark.
D6. KV-cache quantization. Separate from weight quantization, much more damaging per bit. FP8 KV is usually safe; INT4 KV noticeably degrades long-context.
D7. Perplexity looks fine, generation is worse. PPL is a weak proxy — it's teacher-forced, so it never exercises error accumulation. → Always run the generation-based eval on the quantized model.
D8. Quantized model is slower. At batch size 1 with a small model you may be latency-bound on dequantization kernels rather than memory-bound. → Benchmark. INT4's win is largest for large models and memory-bound decode.
D9. Reordering (desc_act=True) in GPTQ. Better accuracy, but slower kernels and incompatible
with some serving stacks.
→ Test end-to-end on the actual serving runtime before committing.
D10. Double quantization loss with QLoRA-trained adapters. Training a LoRA on a 4-bit base then merging into fp16 then re-quantizing compounds error. → Merge into fp16 and re-quantize once, then evaluate; or train on the fp16 base.
E1. The student is worse only for one customer segment. Aggregate metrics pass. → Per-slice gates.
E2. Style regression. Teacher used to write "• " bullets, student writes "- ". Nobody noticed until a customer's parser broke. → Format-compliance metric + a golden-set diff you actually read.
E3. Nondeterminism in production. Same document, different summary, user files a bug. → Greedy decoding + fixed seed for the customer-facing path if determinism is a requirement. Note that batching itself can change results slightly on GPU.
E4. Teacher upgraded, student didn't. Model drift; the student now underperforms a teacher it was never distilled from. → Version the pair together; re-distill on teacher upgrades.
E5. Prompt changed in production but not in training data. Silent quality drop.
→ prompt_version in every record, asserted at serving time.
E6. No rollback path. Keep the previous quantized artifact and a one-command rollback.
Next: 12 — quantization