Saransh — Distilling Qwen into a small summarizer, from the ground up

This repository is a teaching guide + working reference implementation for taking a large instruction-tuned teacher model (Qwen2.5-7B-Instruct) and compressing its summarization ability into a small student (Qwen2.5-0.5B / 1.5B), then quantizing that student for cheap serving.

It is written for someone who has never done distillation before. Every concept is defined before it is used. Every knob is explained: what it is, what it does to the loss surface, what happens if you set it too high or too low, and how to detect that you got it wrong.

---

The one-paragraph version

A big model knows how to summarize. It is expensive to serve. Knowledge distillation trains a small model to imitate the big one — not just its final answers, but the probability distribution it assigns over every possible next token. That distribution ("dark knowledge") carries far more information per example than a single correct answer, so the small model learns from fewer examples than training from scratch would need. After distillation you quantize: store the weights in 4 or 8 bits instead of 16, cutting memory 2–4x with a small, measurable quality cost.

---

Read in this order

#	File	What you learn
00	docs/00-concepts.md	Logits, softmax, temperature, dark knowledge, the 5 kinds of distillation
01	docs/01-environment.md	Hardware sizing, exact package versions, Mac vs CUDA, smoke tests
02	docs/02-teacher-and-student.md	Picking the pair, tokenizer compatibility (the #1 silent killer)
03	docs/03-data-collection.md	Building the summarization corpus, prompt design, licensing
04	docs/04-teacher-inference.md	Generating teacher summaries + capturing top-K logits at scale
05	docs/05-data-quality.md	Dedup, contamination, filtering, length statistics
06	docs/06-tokenization-and-packing.md	Chat templates, loss masking, packing, padding, position ids
07	docs/07-loss-functions.md	CE, forward KL, reverse KL, JSD, hidden-state, attention — full math
08	docs/08-training-loop.md	step vs iter vs epoch, grad accumulation, LR schedules, optimizers, precision
09	docs/09-hyperparameters.md	Every knob, default, safe range, failure symptom
10	docs/10-evaluation.md	ROUGE, BERTScore, faithfulness, LLM-judge, latency, regression gates
11	docs/11-edge-cases.md	~40 concrete failure modes and how each one presents
12	docs/12-quantization.md	INT8/INT4 theory, GPTQ, AWQ, GGUF k-quants, bitsandbytes, calibration
13	docs/13-serving.md	vLLM, llama.cpp, batching, KV cache, throughput math
14	docs/14-troubleshooting.md	Symptom → cause → fix table
15	docs/15-glossary.md	Every term, one line each

Then follow RUNBOOK.md — the day-by-day execution order with a pass/fail check at every step.

Runnable files

src/
  config/distill.yaml            # single source of truth for a run
  data/build_corpus.py           # raw docs -> normalized jsonl
  data/generate_teacher.py       # teacher summaries (sequence-level KD data)
  data/capture_logits.py         # teacher top-K logits (logit-level KD data)
  data/filter_dedup.py           # quality gate + near-dup removal
  data/dataset.py                # Dataset/collator, loss masking, packing
  losses.py                      # ALL distillation losses, heavily commented
  train_distill.py               # the training loop
  eval/evaluate.py               # ROUGE/BERTScore/latency harness
  eval/judge.py                  # LLM-as-judge pairwise eval
  quantize/quantize_awq_gptq.py  # INT4 PTQ
  quantize/to_gguf.sh            # llama.cpp / Mac path
  quantize/bench_quant.py        # fp16 vs int4 quality+speed diff
  serve/vllm_serve.sh            # production serving

The pipeline in one picture

 raw docs ──► 03 build_corpus ──► 05 filter_dedup ──► prompts.jsonl
                                                          │
                        ┌─────────────────────────────────┴──────────────┐
                        ▼                                                ▼
             04 generate_teacher.py                          04 capture_logits.py
             (teacher's summary text)                        (top-K logits per token)
                        │                                                │
                        └───────────────────┬────────────────────────────┘
                                            ▼
                                    06 dataset.py  (tokenize, mask, pack)
                                            ▼
                            07 losses.py + 08 train_distill.py
                                            ▼
                                   student fp16/bf16  ──► 10 evaluate
                                            ▼
                                  12 quantize (AWQ/GPTQ/GGUF)
                                            ▼
                                   12 bench_quant (quality delta)
                                            ▼
                                       13 serve

Quickstart (toy scale, runs on this Mac)

make setup          # venv + deps
make smoke          # 50 examples, 20 steps, tiny models — proves the wiring
make eval-baseline  # numbers before you spend money

Then move to a rented A100/H100 for the real run — see docs/01-environment.md.