Activation outlier — a hidden dimension whose values are 100–1000x larger than typical; the main obstacle to naive LLM quantization.
AdamW — Adam with decoupled weight decay. The standard LLM optimizer.
Alpha (α) — weight on the hard cross-entropy term vs the soft KD term.
AWQ — Activation-aware Weight Quantization. Scales salient weight channels (identified by activation magnitude) before INT4 quantization. Fast, no backprop, great kernels.
bf16 (bfloat16) — 16-bit float with fp32's exponent range and 8 mantissa bits. Default training precision on A100/H100; needs no gradient scaler.
BERTScore — similarity metric using contextual embeddings and greedy token matching.
Calibration set — the small sample of representative data used by PTQ methods to estimate activation ranges. Must match production distribution.
Catastrophic forgetting — loss of previously-learned capabilities during fine-tuning.
Cross-entropy (CE) — −log p(correct token). The "hard label" loss.
Dark knowledge — the information in a teacher's relative probabilities over incorrect tokens.
desc_act — GPTQ option to quantize columns in descending activation order; more accurate, slower.
Distillation temperature (τ) — softmax temperature applied to both teacher and student inside the KD loss to expose tail structure.
Double quantization — quantizing the quantization scales themselves (bitsandbytes NF4).
Effective/global batch size — per_device_bs × grad_accum × num_gpus. The statistically
meaningful batch size.
Epoch — one full pass over the training set.
Exposure bias — mismatch between training (conditioning on perfect prefixes) and inference (conditioning on the model's own, imperfect, prefixes).
Forward KL — KL(teacher ‖ student). Mode-covering; student spreads mass to cover all teacher
modes.
FSDP / ZeRO — sharding parameters, gradients, and optimizer state across GPUs.
GGUF — llama.cpp's model file format; the k-quant types (Q4_K_M etc.) live here.
GKD (Generalized KD) — on-policy distillation on student-generated sequences with a skewed-JSD objective.
GPTQ — layer-wise PTQ using inverse-Hessian error compensation. Top accuracy at 3–4 bits.
Gradient accumulation — summing gradients over several micro-batches before one optimizer step; a memory trick, mathematically equivalent to a bigger batch.
Gradient checkpointing — recomputing activations in the backward pass instead of storing them. ~5–10x activation memory saved for ~30% more compute.
Group size — how many contiguous weights share a quantization scale. 128 is standard.
GQA (Grouped-Query Attention) — fewer K/V heads than Q heads; shrinks the KV cache. Qwen2.5-1.5B has 12 Q heads and 2 KV heads.
Hard target — the one-hot correct token. Soft target — the teacher's full distribution.
Hidden-state distillation — matching intermediate layer activations, usually by cosine similarity through a learned projection.
imatrix — llama.cpp importance matrix: activation statistics guiding per-channel bit allocation.
ignore_index (-100) — the label value PyTorch's cross-entropy skips. How loss masking works.
JSD (Jensen–Shannon divergence) — symmetric, bounded divergence. Skewed JSD (β) interpolates continuously between forward KL (β→1) and reverse KL (β→0).
KL divergence — Σ p log(p/q). Always ≥ 0; asymmetric.
KV cache — stored keys/values from previous tokens; the dominant memory cost at high concurrency.
Logits — pre-softmax scores over the vocabulary.
LoRA — low-rank adapters ΔW = BA; trains ~1% of parameters.
Loss masking — setting labels to -100 for prompt/padding tokens so loss is computed only on
the completion.
NF4 (NormalFloat4) — 4-bit format whose levels are the quantiles of a normal distribution.
On-policy KD — distilling on sequences the student generated.
Packing — concatenating short examples into one sequence to avoid padding waste; requires block-diagonal attention to be correct.
Per-channel / per-group / per-tensor — quantization granularity.
Prefill vs decode — processing the prompt (compute-bound) vs generating tokens one at a time (memory-bandwidth-bound).
Prefix caching — reusing the KV cache of a shared prompt prefix across requests.
PTQ / QAT — Post-Training Quantization / Quantization-Aware Training.
Rejection sampling (RFT) — generate n candidates from the teacher, keep only the best. Usually the highest-ROI data improvement.
Residual bucket — an extra class holding the probability mass outside the stored top-K, so the truncated KD target stays a proper distribution.
Reverse KL — KL(student ‖ teacher). Mode-seeking; penalizes the student for mass the teacher
doesn't have.
ROUGE — n-gram overlap metric family. ROUGE-Lsum is the right variant for multi-sentence summaries.
SeqKD (sequence-level KD) — training on teacher-generated text with plain cross-entropy. No logits needed; works across tokenizers.
SmoothQuant — migrates activation outliers into the weights via per-channel scaling to enable W8A8.
Step / iteration — one optimizer update.
STE (straight-through estimator) — treating round() as the identity on the backward pass; makes
QAT trainable.
Symmetric vs asymmetric quantization — with or without a zero-point offset.
Teacher-assistant distillation — using an intermediate-size model when the teacher/student gap is too large.
Token budget — steps × global_batch × seq_len. The quantity that best predicts final quality.
Top-K logit storage — keeping only the K highest teacher logits per position; makes offline logit distillation tractable.
τ² scaling — multiplying the KD term by τ² to keep its gradient magnitude independent of τ.
ULD (Universal Logit Distillation) — optimal-transport distillation between models with different vocabularies.
Warmup — linearly ramping the LR from 0 over the first few hundred steps, so Adam's variance estimate stabilizes before full-size updates.