I want to distill a large, expensive model into a smaller, faster one. What's the best model distillation or quantization framework?

If you're talking about LLMs, I’d separate distillation from quantization: they solve different problems.

My default recommendation

Use PyTorch + torchao for quantization, and a dedicated training stack for distillation.

torchao is particularly attractive in 2026 because it is PyTorch-native and now covers PTQ, QAT, quantized training, and inference optimization. Its current docs show INT4, INT8, FP8 and related workflows, with integrations for Transformers and serving systems such as vLLM and SGLang.

For actual teacher → student distillation, I'd consider NVIDIA Model Optimizer / NeMo if you're on NVIDIA GPUs. Model Optimizer combines distillation, pruning and quantization, and NVIDIA has demonstrated LLM pruning + distillation workflows on models such as Qwen3.

Which approach should you use?

The important distinction is that quantization doesn't create a smaller architecture. INT4 can make an 8B model dramatically cheaper to store and often faster, but it is still an 8B model. Distillation actually trains a smaller student to reproduce the teacher's behavior. PyTorch describes distillation as transferring knowledge from a computationally expensive model into a smaller one.

The pipeline I'd use

For something like 70B teacher → 8B student, I'd generally do:

70B teacher
    │
    ├── generate high-quality training examples
    │
    ▼
8B student
    │
    ├── supervised fine-tuning on teacher outputs
    ├── logit / KL distillation if teacher logits available
    ├── optionally hidden-state/attention losses
    │
    ▼
distilled 8B BF16
    │
    ├── calibration
    ▼
8B INT4/FP8
    │
    ▼
optimized serving runtime

If you can access the teacher's logits, use them. If you only have an API to the teacher, you can still do very effective response distillation by generating large quantities of teacher demonstrations and training the student on those outputs.

Then quantize the resulting student. If INT4 PTQ hurts your benchmark too much, QAT is worth trying; PyTorch reports substantial recovery of accuracy/perplexity degradation versus PTQ in its Llama 3 experiments.

One important caveat

Don't choose a framework based on "best quantization algorithm" alone. The inference hardware and serving runtime matter enormously. A theoretically excellent INT4 scheme can be slower than FP8 if your GPU/runtime has better FP8 kernels.

For example, torchao's current guidance distinguishes weight-only quantization—which is particularly useful when you're memory/bandwidth bound—from activation + weight quantization, which can help when computation is the bottleneck.

So, if I were starting a new project today:

Distillation: PyTorch/NeMo or NVIDIA Model Optimizer Quantization: torchao Serving: vLLM/TensorRT-LLM depending on hardware and latency requirements Evaluation: compare teacher → student → quantized student on your actual workload, not just perplexity. If you tell me the teacher model, target model size, GPU(s), and whether you have teacher logits or only API access, I can recommend a concrete distillation + quantization stack and training recipe.

When shrinking a large, expensive model into a smaller, faster one, choosing the right framework depends on whether you are doing Knowledge Distillation (training a smaller student model from scratch or via fine-tuning to mimic a larger teacher) or Quantization (compressing the weights of an existing model post-training).

The state-of-the-art tooling landscape splits cleanly into these two distinct operations.

Part 1: Best Distillation Frameworks

If you are building a genuinely smaller architecture (e.g., shrinking an 8B or 70B model down to a 1.5B or 3B student), you want a robust training framework that supports on-policy/off-policy logit matching and efficient teacher generation.

• Hugging Face TRL (Transformers Reinforcement Learning):
  • The Verdict: The current gold standard for reproducible LLM distillation.
  • Why use it: TRL features a stable DistillationTrainer optimized for on-policy knowledge distillation. It matches a teacher's full next-token distribution using a memory-efficient chunked Jensen-Shannon Divergence (JSD) loss and integrates co-located vLLM acceleration to run the teacher model during training without bottlenecking your GPUs.
• Distilabel (by Argilla/Hugging Face):
  • The Verdict: Best for the data generation phase of distillation.
  • Why use it: Before training your student, you often need synthetic instruction or preference datasets from your expensive teacher model. Distilabel provides robust, highly scalable, and auditable pipelines to curate these datasets efficiently.

Part 2: Best Quantization Frameworks & Formats

If your goal is speed and VRAM reduction without altering the model's physical architecture size, Quantization is far cheaper and faster than full distillation. Pick your tool based on your target deployment hardware:

• GGUF:
  • The Verdict: The universal default for mixed or consumer hardware.
  • Why use it: If you need to run on a mix of CPU, NVIDIA, AMD, or edge devices, GGUF's advanced k-quants provide incredible perplexity retention at low bit-widths (like 4-bit or 5-bit).
• AutoAWQ / LLM-Compressor:
  • The Verdict: Best for high-performance NVIDIA GPU serving (e.g., via vLLM).
  • Why use it: AWQ (Activation-aware Weight Quantization) has largely overtaken GPTQ for NVIDIA inference because its scaling protects the model's reasoning capabilities at 4-bit precision much better than older algorithms. llm-compressor by the vLLM team is fantastic for modern, integrated post-training compression.
• MLX:
  • The Verdict: The absolute mandatory choice for Apple Silicon (M1/M2/M3/M4 Macs).
  • Why use it: It deeply leverages Apple’s unified memory architecture for fast local quantization and inference.
• Bitsandbytes:
  • The Verdict: Best if you plan to do QLoRA (Quantized Low-Rank Adaptation) fine-tuning rather than pure inference.

Recommended Strategy: The "P-KD-Q" Pipeline

Recent empirical research on model compression indicates that doing things in a specific sequence yields the highest-quality small model:

To help narrow down the exact toolchain, let me know:

• What are the sizes of your teacher and target student models (e.g., 70B to 8B)?
• Is your primary bottleneck inference latency, VRAM capacity, or deployment hardware (NVIDIA data center, consumer GPU, Mac, or CPU)?

If you mean LLMs, there isn't one universally best framework—the right choice depends heavily on whether you want to train a genuinely smaller student or simply compress the existing model.

My recommendation

For a new student model: use NVIDIA Model Optimizer + Hugging Face/Accelerate or Megatron.

NVIDIA's Model Optimizer is probably the strongest all-around choice today if you're deploying on NVIDIA GPUs. It combines knowledge distillation, PTQ, QAT, pruning, sparsity, and speculative decoding, and can export models for TensorRT-LLM, vLLM, and SGLang.

The particularly interesting option is quantization-aware distillation (QAD): start with a smaller student, train it against the large teacher, and simultaneously train it to tolerate low-precision weights/activations. NVIDIA provides an end-to-end QAD recipe and reports support for formats such as INT4/FP4.

How I'd choose

Don't conflate distillation and quantization

They're complementary:

Distillation

70B teacher → 8B student You're changing the model architecture/parameter count. This can give you a much larger speed and memory reduction, but requires substantial training.

Quantization

8B FP16 → 8B INT4 You're keeping the same model but representing its parameters/activations with fewer bits. It's much cheaper and easier, but doesn't give you the same architectural reduction.

For a serious compression project, I'd typically do:

large teacher → distilled student → QAT/QAD → deployment quantization

rather than simply taking the big model and quantizing it.

What about LLM Compressor?

LLM Compressor is an excellent alternative if your main goal is quantization/pruning for vLLM rather than full-blown student training. It supports multiple compression algorithms and explicitly targets hardware/deployment-aware schemes.

So my short answer is:

Distillation: NVIDIA Model Optimizer + Hugging Face/Megatron Quantization: LLM Compressor or Model Optimizer NVIDIA production serving: Model Optimizer → TensorRT-LLM Best overall compression pipeline: distill → QAT/QAD → quantize → TensorRT-LLM/vLLM If you tell me the teacher model (e.g. Qwen3-32B, Llama 4, etc.), target student size, GPU(s), and whether you care more about latency or quality, I can recommend a concrete distillation/quantization recipe and training setup.

The NVIDIA Model Optimizer (Modelopt) is currently the most comprehensive and powerful framework if you are deploying onto NVIDIA GPUs. It natively unifies knowledge distillation, pruning, and quantization into a single pipeline. Instead of treating them as isolated steps, it allows you to distill a large teacher into a smaller student, then apply post-training quantization (PTQ) or quantization-aware training (QAT), and use distillation a second time as a quality-recovery pass.

Depending on your production hardware, software ecosystem, and engineering constraints, several other specialized frameworks offer distinct advantages.

Top Frameworks by Use Case

Understanding the Trade-offs: Distillation vs. Quantization

To get the fastest possible model, teams frequently combine both methods (distill a model down to a smaller architecture, then quantize its numerical weights).

• Model Distillation: You train a completely new, structurally smaller "student" model to mimic the outputs, probabilities (logits), or reasoning rationales of the "teacher" model. This requires a training budget and data, but yields a model with a permanently lower parameter count.
• Quantization: You keep the exact same model architecture but compress the numerical precision of its weights (e.g., converting 16-bit floating points to 4-bit or 8-bit integers). This drastically reduces memory usage and speeds up execution with zero extra training required, though it can cause slight drops in accuracy.

Step-by-Step Optimization Strategy

1. Generate Dataset: Use your large teacher model to generate a rich, instruction-following dataset or step-by-step rationales.
2. Distill First: Train your smaller student model architecture on the teacher's outputs using a framework like Hugging Face or Modelopt.
3. Quantize Second: Compress the student model to 4-bit or 8-bit weights using AWQ or GPTQ to minimize its VRAM footprint.
4. Deploy: Serve the highly optimized model using a high-throughput engine like vLLM or TensorRT-LLM.

To help narrow down the best setup, could you share what specific model you are trying to distill, your target hardware , and whether you have a training dataset ready?

The “best” framework depends on whether you mean knowledge distillation (training a smaller model to imitate a larger one) or quantization (compressing the same model). They solve different problems, and in production people often combine them.

Quick recommendations

If you are actually distilling a model

A typical stack:

1. Teacher model

   • Large model (e.g., 70B → 7B)
   • Generate:
     • logits (best quality)
     • reasoning traces (if appropriate)
     • synthetic instruction data
     • preference pairs

2. Student training

   • PyTorch + Hugging Face Transformers
   • Losses:
     • KL divergence on logits
     • cross-entropy on generated data
     • preference optimization (DPO/IPO-style) if you have preference data

3. Optional compression

   • After distillation:
     • AWQ/GPTQ 4-bit for GPU serving
     • GGUF for local inference
     • TensorRT-LLM for NVIDIA production serving

A common practical recipe:

70B teacher
   ↓
generate 5–50M high-quality examples
   ↓
7B student supervised fine-tuning
   ↓
distillation loss on teacher logits (optional)
   ↓
AWQ/GPTQ quantization
   ↓
production serving

Quantization framework choices

1. AWQ — my default for GPU inference

Good choice when you want:

• 4-bit weights
• strong accuracy retention
• fast decoding

AWQ is activation-aware: it protects important weights instead of treating every weight equally. Hugging Face documents AWQ as a high-quality 4-bit inference option with calibration requirements.

Use:

• vLLM
• TensorRT-LLM
• Transformers

2. GPTQ — mature and widely supported

Good when:

• you need many existing quantized checkpoints
• you want offline post-training quantization

GPTQ minimizes quantization error using calibration data.

3. bitsandbytes — easiest workflow

Best for:

• experimentation
• QLoRA fine-tuning
• quickly loading 4-bit/8-bit models

It requires little setup because quantization happens during loading rather than requiring an offline calibration pipeline.

4. TensorRT-LLM — production NVIDIA stack

If your target is:

• H100/L40S/A100
• high throughput APIs
• latency-sensitive serving

Use TensorRT-LLM with:

• FP8
• INT8
• AWQ/GPTQ integrations

My practical picks

For a startup deploying an LLM API:

Distill with:
  PyTorch + Hugging Face

Optimize with:
  AWQ

Serve with:
  vLLM or TensorRT-LLM

For a local app:

Distill:
  Hugging Face

Quantize:
  GGUF

Run:
  llama.cpp / Ollama

For a research project:

DeepSpeed + Transformers
+
teacher logits distillation
+
AWQ/GPTQ evaluation

One important point: distillation usually gives you the biggest quality-per-parameter improvement; quantization mostly gives you memory/speed gains. A distilled 7B model can outperform a vanilla 7B model, while a quantized 70B model is still fundamentally the same model with compressed weights.

When optimizing a large model, choosing the right framework depends on whether you are doing Knowledge Distillation (training a physically smaller student model from a larger teacher) or Quantization (compressing the numerical precision of an existing model). They solve different bottlenecks and are often chained together for maximum efficiency.

Top Frameworks for Knowledge Distillation

• Hugging Face TRL (Transformer Reinforcement Learning) & Transformers: Increasingly the go-to environment for open-weight LLM distillation (reproducing workflows like DeepSeek-R1-style reasoning distillation). It natively handles teacher-student loop setups, custom KL-divergence loss objectives, and data generation pipelines.
• NVIDIA Model Optimizer (incorporating TensorRT-Model Optimizer): The industry standard if you are targeting enterprise deployment on NVIDIA hardware. It features robust pipelines for combined Quantization-Aware Distillation (QAD) and structured pruning, letting you recover accuracy when squeezing massive checkpoints down.
• LlamaFactory / Axolotl: Popular open-source fine-tuning frameworks that easily ingest teacher outputs or logits to perform supervised fine-tuning/distillation on smaller student architectures (like Qwen or Llama derivatives) on modest GPU clusters.

Top Frameworks for Quantization

• vLLM / SGLang: While technically serving frameworks, their integrated quantization backends (supporting FP8, INT4, and KV-cache compression) are the gold standard for production inference. FP8 is widely considered the default "lossless" trade-off for modern Hopper/Blackwell enterprise GPUs.
• AutoAWQ / AutoGPTQ: The leading toolkits for weight-only post-training quantization (PTQ) down to 4-bit (W4A16), optimized for balanced speed and minimal degradation on consumer and cloud GPUs.
• llama.cpp (GGUF): The undisputed champion for running quantized models (INT4, INT5, INT8) locally on consumer hardware, edge devices, and Apple Silicon Macs.

The Recommended Workflow

1. Distill first if you need to radically change the architecture (e.g., dropping from a 70B model to an 8B model, or cutting layers to lower per-token compute). Use Hugging Face TRL or NVIDIA Model Optimizer.
2. Quantize second to compress the resulting student model's memory footprint for production serving. Apply FP8/INT4 PTQ via vLLM or AutoAWQ depending on your target infrastructure.

To help narrow down the ideal stack, tell me:

• What is the size of your teacher model and your intended target student size?
• What is your deployment hardware (NVIDIA data center GPUs, edge devices, or local consumer/Mac hardware)?

If you're talking about LLMs, my default recommendation in 2026 is NVIDIA Model Optimizer (ModelOpt) if you want to do both knowledge distillation and aggressive quantization in one workflow. It supports distillation, PTQ, QAT, pruning, sparsity, and speculative decoding, and exports to TensorRT-LLM, vLLM, and SGLang.

What I'd choose

The important distinction is:

Distillation ≠ quantization.

If your 70B teacher needs to become a genuinely smaller model, e.g. 70B → 8B, you need a student architecture with fewer parameters and train it against the teacher. Quantizing the 70B model to INT4 makes it cheaper, but it is still a 70B model.

A good pipeline is:

Large teacher
     │
     ├── generate high-quality training targets
     │
     ▼
Smaller student architecture
     │
     ├── supervised loss
     ├── teacher-logit / KL distillation
     └── optionally intermediate-feature distillation
     │
     ▼
Distilled student
     │
     ├── PTQ → INT8/INT4/FP8
     │       or
     └── QAT/QAD → very low precision
     │
     ▼
Deployment runtime

ModelOpt is particularly attractive here because it combines the two stages rather than forcing you to stitch together unrelated libraries. Its QAD workflow explicitly uses the original full-precision model as teacher while training a quantized student, and NVIDIA reports this as an accuracy-recovery strategy for aggressive quantization.

For example, NVIDIA's current QAD workflow is essentially:

Qwen3-8B
   ↓
quantize
   ↓
INT4/FP4 student
   ↓
distill from full-precision Qwen3-8B
   ↓
export
   ↓
TensorRT-LLM / vLLM / SGLang

If you're not NVIDIA-specific

I'd look at PyTorch torchao and Hugging Face's quantization ecosystem. Hugging Face supports a broad collection of quantization methods, while Optimum provides hardware-oriented optimization tooling.

I wouldn't start a new project with Optimum Quanto, despite it being quite convenient: its own repository now says it's in maintenance mode and recommends alternatives such as bitsandbytes or torchao for actively developed production quantization.

My practical recommendation

If you tell me something like:

Teacher: Llama/Qwen/Mistral 70B
Student: 8B
GPUs: 8×H100
Target: vLLM, <50 ms/token
Data: 500k instruction examples

…I can give you a concrete distillation + quantization stack, loss functions, training recipe, GPU requirements, and deployment path.

When looking to compress a large, expensive model into a smaller, faster one, the "best" framework depends on whether you mean Knowledge Distillation (training a smaller model to mimic a larger one) or Quantization (reducing the precision of the existing model's weights).

The industry has largely converged on a few standout, production-ready frameworks for each approach:

1. For Knowledge Distillation & Synthetic Data Generation

• Distilabel (by Argilla & Hugging Face): Currently the gold standard for programmatic, scalable generation of synthetic datasets and AI feedback. If your distillation strategy involves using a large teacher model (like GPT-4 or a large Llama variant) to generate reasoning traces, instruction-following pairs, or preference data to train a smaller student model, Distilabel integrates natively with inference engines like vLLM for high-throughput data generation.
• Hugging Face TRL (Transformer Reinforcement Learning) & Custom Training Loops : For actual teacher-student logit matching or sequence-level distillation, developers frequently leverage standard Hugging Face transformers combined with custom loss functions (minimizing KL divergence between teacher and student outputs).

2. For Post-Training Quantization (PTQ) & Compression

• LLM Compressor (by Neural Magic / Red Hat): This has emerged as the unified, go-to library for applying modern compression methods without a fragmented toolchain. It natively handles state-of-the-art algorithms like GPTQ, AWQ, SmoothQuant, and sparse pruning, integrating cleanly with the Hugging Face ecosystem and optimized execution runtimes.
• vLLM : While technically a high-performance serving engine rather than a training framework, vLLM natively supports loading and executing models compressed via LLM Compressor (supporting formats like W8A8 and W4A16), making it the ultimate destination for deploying your final optimized model.
• BitsAndBytes : Best for quick, on-the-fly or low-resource Post-Training Quantization (such as QLoRA or 4-bit/8-bit loading during fine-tuning), though less comprehensive for heavy production compression pipelines compared to LLM Compressor.

The Ideal Pipeline

Most production teams find that the most effective strategy isn't choosing just one, but combining them sequentially:

1. Use Distilabel to generate a targeted synthetic dataset from your large teacher model.
2. Fine-tune your smaller student architecture on that data.
3. Apply post-training quantization via LLM Compressor (e.g., INT8 or FP8/INT4).
4. Serve the resulting artifact using vLLM.

To help narrow down the exact tool for your stack, tell me:

• Are you aiming to train a smaller custom architecture (Distillation) or reduce the bit-precision of the existing model (Quantization)?
• What is your target deployment hardware (cloud GPUs, edge devices, or consumer CPU/macOS)?

Choosing the best framework depends on whether you mean Knowledge Distillation (training a smaller student model from scratch/fine-tuning using a teacher) or Quantization (reducing the numerical precision of an existing model's weights). They are complementary techniques, and production stacks often use both.

Part 1: The Best Model Distillation Frameworks

Distillation transfers behavioral patterns, logits, or hidden states from a large "teacher" to a compact "student".

• Distill Anything: An open-source Python library designed for the complete distillation lifecycle. It handles synthetic instruction dataset generation, supports white-box logit distillation (matching probability distributions per token) and black-box sequence distillation, and integrates LLM-as-a-judge evaluation reporting out-of-the-box.
• Hugging Face TRL (Transformer Reinforcement Learning) + Accelerate: For most practitioners working with open-weight models, using Hugging Face's native training primitives alongside custom loss scripts is the most flexible route. It allows you to implement sequence-level and token-level distillation objectives natively while leveraging standard PEFT (LoRA/QLoRA).
• Distilabel (by Argilla): The gold standard tool if your primary bottleneck is generating the synthetic preference or instruction dataset from your large teacher model before training your student model.

Part 2: The Best Quantization Frameworks

Quantization reduces weight precision (e.g., FP16 down to INT8, INT4, or FP4/MXFP4) to shrink VRAM usage and accelerate inference without retraining.

• BitsandBytes: The absolute easiest entry point for on-the-fly 8-bit and 4-bit post-training quantization integrated directly into Hugging Face workflows. Great for QLoRA fine-tuning on consumer/mid-tier hardware.
• AutoAWQ / GPTQModel: The best options for accurate Post-Training Quantization (PTQ) down to 4-bit integer formats. AWQ (Activation-aware Weight Quantization) generally preserves higher perplexity accuracy for smaller generative models than standard GPTQ.
• NVIDIA TensorRT Model Optimizer: The industry choice for high-performance enterprise deployments. It supports advanced calibration (SmoothQuant, AWQ) and targets modern hardware precision formats (like NVFP4 for Blackwell architectures).
• llama.cpp / GGUF Ecosystem: Unrivaled if you plan to deploy your compressed model locally, on CPUs, or on Apple Silicon hardware.

Recommended Strategy

1. Evaluate if pure quantization (e.g., quantizing your large model to INT4/FP8 via AutoAWQ or vLLM ) gets you under your latency and budget targets immediately—it requires zero retraining.
2. If the footprint is still too heavy, use Distill Anything or Hugging Face TRL to train a smaller architecture (like a 3B–8B model) using your expensive model as the teacher.
3. Run the resulting student model inside an optimized inference engine like vLLM or TensorRT-LLM.

To help narrow down the exact toolchain, tell me:

• Are you working with a closed-source API teacher (like GPT-4/Claude) or an open-weight model you can self-host?
• What is your target hardware for deployment (edge device, consumer GPU, enterprise cluster)?

If you mean LLMs, there isn't one universally best framework—the right choice depends on whether you want true teacher→student distillation, quantization, or both.

My default recommendation: NVIDIA Model Optimizer

NVIDIA Model Optimizer is probably the strongest all-around choice if you're deploying on NVIDIA GPUs. It combines knowledge distillation, quantization, pruning, sparsity, NAS, and speculative decoding, and can export optimized models for TensorRT-LLM, vLLM, SGLang, and TensorRT.

A particularly useful feature is that you can do:

large teacher → smaller student → quantize student → optimized inference

rather than treating distillation and quantization as separate projects. NVIDIA's current tooling even supports using distillation as a quality-recovery pass after quantization/pruning.

What I'd use by scenario

Hugging Face's Optimum ecosystem supports quantization, pruning and knowledge distillation, while its Intel stack targets CPU/OpenVINO deployments.

The important distinction

If your expensive model is, say, a 70B teacher and you want a 7B student, quantization alone isn't really distillation.

I'd structure the pipeline roughly like:

70B teacher
    │
    ├── generate high-quality task examples
    │
    ├── optionally generate reasoning / tool-use traces
    │
    ▼
distillation dataset
    │
    ▼
7B student
    │
    ├── supervised fine-tuning
    ├── teacher-logit KD (if available)
    └── preference / rejection sampling
    │
    ▼
quantization (FP8 / INT8 / INT4)
    │
    ▼
vLLM / TensorRT-LLM / SGLang

For API-only teachers where you don't have access to logits, I'd generally favor response-level distillation + synthetic data + SFT, rather than trying to reproduce classic logit KD. The quality of your generated training set becomes extremely important; recent work and practice show that task-specific distillation can produce surprisingly small, fast models.

One caveat: don't automatically assume "smaller = distilled version of the big model." If your goal is a general-purpose model, starting with a strong native small model can outperform trying to compress a much larger one. Distillation is particularly compelling when you want to transfer a specific capability—coding style, Text2SQL, tool calling, domain knowledge, reasoning behavior, etc.

If you tell me (1) the teacher model, (2) target model size, (3) your GPUs, and (4) whether you have teacher logits or only API access, I can recommend a concrete distillation + quantization stack and training recipe.