AI Model Distillation and Compression: 2025-2026 Advances

Comprehensive Research Report

---

1. Knowledge Distillation Techniques

1.1 Classical and Evolved Approaches

Knowledge distillation (KD), originally formalized by Hinton, Vala, and Dean (2015), has undergone significant evolution through 2025-2026. The core idea remains: a large “teacher” model transfers its learned representations to a smaller “student” model by training the student to match the teacher’s soft probability distributions (logits) rather than just hard labels.

Key technique families now in active use:

• Logit-based distillation: The student minimizes KL-divergence between its output distribution and the teacher’s softened outputs (temperature-scaled softmax). This remains the backbone of most production distillation pipelines.

• Feature/representation-based distillation: The student learns to match intermediate layer activations or attention maps of the teacher. FitNets-style approaches have been refined with projection layers that handle dimension mismatches between teacher and student hidden states.

• Relation-based distillation: Rather than matching individual representations, the student learns the relational structure between examples as encoded by the teacher (e.g., similarity matrices across a batch).

1.2 Advances in 2025-2026

Data-driven / Synthetic Data Distillation: The dominant paradigm shift has been toward distillation through synthetic data generation. Instead of traditional KD where the student trains on the teacher’s logits, the teacher generates massive synthetic training corpora that encode its capabilities. The student then trains on this data using standard language modeling objectives. This approach:

• Eliminates the need for white-box access to teacher weights or logits
• Scales more easily across different model architectures
• Allows curriculum design – generating data of progressively increasing difficulty

Multi-teacher and ensemble distillation: Students now routinely learn from ensembles of specialized teachers rather than a single monolithic model. Each teacher may excel in a different domain (math, code, reasoning, multilingual), and the student aggregates these capabilities.

Self-distillation and iterative refinement: Models are trained to distill from their own previous checkpoints or augmented outputs. DeepSeek and Qwen teams have reported using iterative self-distillation where a model generates reasoning traces, filters for correct ones, and retrains on them.

Distillation of reasoning chains (Chain-of-Thought distillation): A major 2025 development. Rather than just distilling final answers, teacher models generate step-by-step reasoning traces, and students learn to replicate this reasoning process. This has been critical for retaining reasoning capability in smaller models. DeepSeek-R1-Distill models are a prime example.

---

2. Notable Distilled Models (2025-2026)

2.1 Microsoft Phi Series

The Phi family represents the most prominent example of distillation-from-large-model-data:

• Phi-1 (June 2023): 1.3B parameters, trained on “textbook-quality” synthetic data generated with GPT-3.5/GPT-4. Achieved HumanEval scores competitive with models 10x its size.

• Phi-2 (December 2023): 2.7B parameters. Demonstrated that careful data curation and synthetic data from larger models could produce a 2.7B model competitive with Llama-2-70B on certain benchmarks.

• Phi-3 (April 2024): Released in mini (3.8B), small (7B), and medium (14B) variants. Phi-3-mini achieved GPT-3.5-Turbo-level performance at 3.8B parameters. Training data heavily leveraged synthetic data pipelines filtered and generated by larger models.

• Phi-4 (December 2024): 14B parameters. Pushed the frontier further, with Microsoft emphasizing “data quality over data quantity.” Phi-4 used synthetic data generated by GPT-4 across structured reasoning, math, and coding domains. Achieved scores competitive with much larger models on MATH, GPQA, and HumanEval benchmarks.

• Phi-4-mini and Phi-4-multimodal (early 2025): Continued the trend with specialized small models. Phi-4-mini targets on-device deployment while retaining strong reasoning performance.

Key insight from the Phi program: With sufficiently high-quality training data (often synthetically generated by frontier models), small models can achieve 90%+ of frontier model capability on targeted benchmarks, though generalization breadth remains narrower.

2.2 DeepSeek-R1 Distilled Models

DeepSeek’s R1 series (January 2025) introduced a landmark set of distilled reasoning models:

• DeepSeek-R1 (671B MoE, ~37B active): The full reasoning model, trained with reinforcement learning to produce long chain-of-thought reasoning.

• DeepSeek-R1-Distill-Qwen-1.5B / 7B / 14B / 32B: Distilled from R1 into dense Qwen-architecture models.

• DeepSeek-R1-Distill-Llama-8B / 70B: Distilled into Llama-architecture models.

Performance highlights:
- R1-Distill-Qwen-32B achieved ~72% on AIME 2024 (math competition), competitive with OpenAI’s o1-mini.
- R1-Distill-Qwen-7B significantly outperformed non-reasoning models many times its size.
- The distillation transferred the “thinking” behavior – these small models produce extended reasoning chains.

Technique: The distillation used approximately 800K samples of R1’s reasoning traces. Students were fine-tuned to replicate both the reasoning process and final answers. This demonstrated that reasoning capability, previously thought to require massive scale, could be compressed into much smaller models.

2.3 Apple OpenELM

Apple’s OpenELM (2024-2025) explored efficient language models with layer-wise scaling:

• Available in 270M, 450M, 1.1B, and 3B parameter variants.
• Used a non-uniform allocation strategy: different transformer layers have different widths (number of attention heads, FFN dimensions), with parameters concentrated in middle layers where they provide the most benefit.
• Trained on publicly available data (no synthetic data from proprietary models).
• Achieved competitive performance with OLMo-1B while using 2x fewer pre-training tokens.
• Fully open (weights, training code, data pipeline).

2.4 Qwen Series

Alibaba’s Qwen team released progressively capable small models through 2025:

• Qwen2.5 (September 2024): Released in 0.5B, 1.5B, 3B, 7B, 14B, 32B, and 72B. The smaller models showed remarkable capability for their size, especially in multilingual tasks.
• Qwen2.5-Coder: Specialized coding models showing that domain-specific distillation yields strong returns.
• QwQ-32B (March 2025): A reasoning model that used distillation from larger reasoning traces combined with RL.

2.5 Llama Series Distilled Variants

Meta’s Llama 3.1 and 3.2 ecosystem (2024-2025):

• Llama-3.2-1B and 3B: Explicitly distilled from Llama-3.1-8B and 70B using logit-based distillation combined with pruning (structured removal of layers and attention heads). Meta reported using the larger models’ output distributions as training signals.
• Achieved practical on-device deployment targets while retaining meaningful capability.

2.6 Google Gemma and Gemini Distillations

• Gemma 2 (2024): 2B and 9B models that benefited from distillation from larger Gemini models.
• Gemma 3 (March 2025): Released in 1B, 4B, 12B, and 27B. Google explicitly acknowledged using distillation from Gemini models. Gemma 3-27B showed competitive performance with significantly larger open models.

2.7 Other Notable Entries

• SmolLM and SmolLM2 (Hugging Face, 2024-2025): 135M, 360M, and 1.7B models focused on on-device deployment.
• Mistral Small and Ministral (2025): 3B and 8B models from Mistral AI with strong efficiency.
• NVIDIA Minitron: Used pruning + distillation to compress Llama-3.1-8B to 4B while retaining ~90% of MMLU performance.

---

3. Structured and Unstructured Pruning

3.1 Unstructured Pruning

Removes individual weights (setting them to zero) based on magnitude or importance scores.

• SparseGPT (2023, continued use in 2025): One-shot pruning method that can remove 50-60% of weights from GPT-scale models with minimal quality loss. Works by solving a layer-wise reconstruction problem.
• Wanda (Pruning by Weights and Activations): Prunes weights based on the product of weight magnitude and input activation norm. Simple, fast, and effective at 50% sparsity.
• 2:4 structured sparsity: NVIDIA’s hardware-supported pattern where 2 of every 4 weights are zero. Achieves 50% sparsity with actual hardware speedup on NVIDIA Ampere/Hopper GPUs. This has become a practical deployment target in 2025.

Limitations: Unstructured sparsity beyond 50% generally degrades quality significantly. Hardware support remains limited except for NVIDIA’s 2:4 pattern.

3.2 Structured Pruning

Removes entire structural units (attention heads, layers, neurons, or embedding dimensions), yielding actual wall-clock speedups without specialized sparse hardware.

Major 2025-2026 advances:

• Layer pruning: Removing entire transformer layers. Research has shown that middle-to-late layers in deep transformers often have high redundancy. Models like Llama-3.1-8B have been pruned from 32 to 24 layers with moderate quality loss, followed by distillation-based recovery.

• Width pruning: Reducing the hidden dimension, number of attention heads, or FFN intermediate size. This is what Meta used for Llama-3.2-1B/3B – they pruned Llama-3.1-8B by removing attention heads and reducing FFN dimensions, then performed knowledge distillation to recover quality.

• NVIDIA Minitron approach (2024-2025): A systematic pipeline of:
  1. Importance estimation (using gradient-based or activation-based metrics)
  2. Structured pruning (removing heads, neurons, layers, or embedding channels)
  3. Lightweight retraining with distillation from the original model

Applied to Llama-3.1-8B to create a 4B model that retained ~90% benchmark performance at half the size. Also applied to Nemotron-4-15B to create an 8B variant.

• SliceGPT (2024-2025): Removes rows/columns from weight matrices by exploiting computational invariance in transformer architectures. Achieves up to 25% parameter reduction with minimal perplexity increase.

• Depth pruning with block importance: Techniques like ShortGPT identified that many transformer blocks contribute minimally and can be removed wholesale. Combined with distillation, this allows significant compression.

3.3 Pruning + Distillation Synergy

The dominant 2025 paradigm is prune-then-distill: structurally prune a large model, then use the original (unpruned) model as a teacher to recover quality through distillation. This consistently outperforms either technique alone and has become the standard pipeline at NVIDIA, Meta, and others.

---

4. Quantization and Quantization-Aware Training

4.1 Post-Training Quantization (PTQ)

Quantizing pre-trained model weights after training, without retraining.

• GPTQ (2023, ubiquitous by 2025): Layer-wise quantization using approximate second-order information. Standard for 4-bit and 3-bit quantization of LLMs.
• AWQ (Activation-aware Weight Quantization): Identifies salient weight channels (those corresponding to large activations) and protects them during quantization. Became the default for many deployment pipelines.
• QuIP# and QuIP-Sharp: Uses incoherence processing and lattice codebooks for near-optimal 2-bit quantization. Achieved remarkably low degradation at extreme compression.
• AQLM (Additive Quantization for LMs): Multi-codebook quantization that learns codebooks jointly. State-of-the-art at 2-bit precision as of 2024-2025.
• HQQ (Half-Quadratic Quantization): Fast, zero-shot quantization requiring no calibration data. Practical for rapid deployment.

Typical performance retention with PTQ (2025 state of the art):

Precision	Compression Ratio	Typical Quality Retention
INT8	2x (vs FP16)	99%+
INT4	4x	95-98%
INT3	5.3x	90-95%
INT2	8x	80-90%

4.2 Quantization-Aware Training (QAT)

Training or fine-tuning with simulated quantization, allowing the model to adapt to low-precision representations.

Key 2025-2026 developments:

• QLoRA and variants: Quantize the base model to 4-bit, then train LoRA adapters in higher precision. This has become the standard for fine-tuning large models on consumer hardware. QLoRA at 4-bit NF4 quantization retains nearly full fine-tuning quality.

• BitNet and 1-bit LLMs: Microsoft’s BitNet b1.58 (2024) demonstrated that ternary weights ({-1, 0, +1}) could work for LLMs when trained from scratch with quantization-aware training. In 2025, this line of work continued with:

• Improved training recipes for ternary/binary models

• Custom kernels showing actual inference speedups

• Models up to 3B parameters demonstrating viability

• However, scaling to frontier-quality remains unproven

• NVIDIA TensorRT-LLM INT4-AWQ and FP8: Production-grade quantized inference. FP8 (8-bit floating point) became the standard for training and inference on Hopper GPUs, with essentially zero quality degradation.

• GGUF/llama.cpp quantization ecosystem: The open-source community standardized on GGUF format with extensive quantization options (Q2_K through Q8_0). By 2025, Q4_K_M (4.5-bit effective) became the sweet spot for consumer deployment, offering excellent quality retention with practical memory savings.

4.3 KV-Cache Quantization

A 2025 focus area: quantizing the key-value cache during inference to reduce memory for long contexts.

• KV-cache quantization to INT4 or even INT2 with per-head/per-channel scaling
• Enables 2-4x longer context lengths at the same memory budget
• Minimal impact on quality when combined with careful calibration
• Deployed in production by vLLM, TensorRT-LLM, and other serving frameworks

---

5. Compression Ratios and Quality-Size Tradeoffs

5.1 The Pareto Frontier (2025-2026)

The quality-size tradeoff frontier has shifted dramatically. Using a combination of distillation, pruning, and quantization:

Technique Combination	Effective Compression	Typical Quality Retention
Distillation only (large to small)	10-50x parameter reduction	85-95% on targeted benchmarks
Structured pruning + distillation	2-4x parameter reduction	90-95%
4-bit quantization (PTQ)	4x memory reduction	95-98%
Pruning + quantization + distillation	8-16x effective compression	88-95%
Extreme (2-bit + pruning)	16-32x	80-90%

5.2 Landmark Compression Results

DeepSeek-R1 distillation chain:
- Teacher: 671B parameters (MoE), ~37B active
- R1-Distill-Qwen-32B: ~47x total parameter reduction, retains ~85% of R1’s reasoning benchmarks
- R1-Distill-Qwen-7B: ~96x reduction, retains ~70% of R1’s AIME score
- R1-Distill-Qwen-1.5B: ~450x reduction, still shows non-trivial reasoning

Phi-4 (14B) vs GPT-4:
- ~100x smaller than estimated GPT-4 size
- Competitive on MATH, HumanEval, and select reasoning benchmarks
- Significant gap on broad knowledge and complex multi-step tasks

NVIDIA Minitron (Llama-3.1-8B to 4B):
- 2x parameter reduction via structured pruning
- Followed by distillation recovery
- Retained ~90% average benchmark performance
- Combined with INT4 quantization: 8x total memory reduction at ~87% quality

Llama-3.2-1B (distilled from 8B/70B):
- 8-70x compression from teacher models
- Practical on mobile devices (phones, edge)
- Meaningful capability for summarization, simple Q&A, and classification

5.3 Where 90%+ Retention Is Achievable

Based on 2025 results, 90%+ capability retention is reliably achievable when:

1. The task is narrow/specialized: Coding, math, a specific language – domain-focused distillation works extremely well.
2. INT4 quantization is used: Nearly lossless for most tasks.
3. Structured pruning stays below 50%: Removing up to half of layers/heads with distillation recovery typically retains 90%+.
4. The student architecture is well-designed: Efficient architectures (grouped-query attention, RoPE, SwiGLU) help small models punch above their weight.

Where 90% retention is difficult:
- Broad world knowledge (requires parameters to store facts)
- Long-tail reasoning across diverse domains
- Complex multi-turn instruction following
- Very long context understanding

---

6. Emerging Techniques and Trends (Late 2025 - Early 2026)

6.1 Mixture-of-Experts as Compression

MoE architectures are increasingly viewed as a form of conditional computation / compression:
- Total parameters are large, but active parameters per token are small
- DeepSeek-V3 (671B total, ~37B active) set the standard
- Smaller MoE models (e.g., Mixtral 8x7B with ~12B active) offer excellent quality-per-FLOP
- Trend toward fine-grained MoE with many small experts

6.2 Speculative Decoding

Using small “draft” models to generate candidate tokens, verified by the large model:
- Does not change output quality (mathematically equivalent to the large model)
- Achieves 2-3x speedup by running the small model autoregressively and the large model in parallel on batches
- The draft model is essentially a compressed proxy that captures the large model’s likely outputs
- Medusa, EAGLE, and other architectures add specialized prediction heads

6.3 Architecture-Level Efficiency

• Linear attention and state-space models: Mamba, RWKV, and hybrid architectures that offer O(n) sequence processing instead of O(n^2). These are inherently “compressed” in terms of compute per token.
• Grouped-Query Attention (GQA): Now standard, reduces KV-cache memory by sharing key-value heads across query heads.
• Multi-Query Attention (MQA): Even more aggressive sharing, used in some production models.
• Mixture of Depths: Dynamically skipping computation for “easy” tokens, effectively a form of adaptive compression.

6.4 Distillation for Reasoning

The biggest 2025 development was demonstrating that reasoning capability – previously thought to be an emergent property of scale – can be distilled:
- DeepSeek-R1 distillation showed small models can learn to “think step by step”
- Open-source community created numerous reasoning-distilled models (OpenThinker, Sky-T1, etc.)
- Key insight: distilling the process (chain of thought), not just the answer, is critical
- Limitation: distilled reasoning models tend to “overthink” simple problems and have less flexible reasoning than RL-trained models

6.5 Quantization Frontiers

• FP4 training and inference: Early experiments with 4-bit floating point for both training and inference, potentially halving the cost of FP8.
• Mixed-precision with learned precision allocation: Different layers or attention heads are quantized to different bit-widths based on learned sensitivity.
• Hardware co-design: Custom silicon (Groq, Cerebras, Apple Neural Engine) increasingly optimized for low-precision inference of compressed models.

---

7. Summary of the State of the Art

The 2025-2026 landscape can be summarized as follows:

1. Synthetic data distillation is the dominant paradigm for creating small, capable models. The Phi series proved it; DeepSeek-R1 extended it to reasoning.

2. The practical compression stack for deployment is: train/distill a well-architected small model, then apply INT4 quantization (AWQ or GPTQ), with KV-cache quantization for long contexts.

3. Structured pruning + distillation (the “Minitron recipe”) has become a reliable way to halve model size with minimal quality loss, and is now an established technique at multiple labs.

4. The 90% quality threshold is reliably achievable at 4-8x compression on targeted tasks, and at higher compression ratios for specialized domains. General-purpose broad capability still requires larger models.

5. Reasoning distillation was the major breakthrough of early 2025, showing that chain-of-thought capability can transfer from very large RL-trained models to much smaller distilled students.

6. The effective frontier has shifted such that a 7-14B parameter model in 2025, properly trained with distillation and quantized to 4-bit, can match or exceed what a 70B+ model could do in 2023 on many benchmarks. This represents a roughly 20-40x improvement in the quality-per-bit frontier over two years.