Beyond a Single Number: Evaluating Quantized Models for Deployment

Summary

Quantized models are often compared using a single convenient number: model size, bits per weight, perplexity, KL divergence, or tokens per second. Each of these metrics answers a useful question. None answers the full deployment question: Which quant should I run on my hardware? This three-part series develops ByteShape’s practical approach to quantized-model evaluation. The framework is simple: Does the model fit? How well does it perform? How fast does it run on the target hardware? Fit determines which models are possible. Downstream quality and measured throughput determine which of those models are practical. Part 1: Fit, Quality, and Speed: A Practical Framework for Choosing Quants Why fit, downstream quality, and throughput matter more than any single proxy. Part 1 introduces the evaluation framework. It separates three categories of measurement: fit metrics, such as model size, BPW, and context-memory requirements; quality measurements, such as downstream task benchmarks; deployment measurements, such as token-generation throughput on the target hardware. It also distinguishes downstream quality from fidelity to a reference model. Perplexity and KL divergence are useful for detecting substantial degradation, but they do not directly measure whether a quant answers correctly, follows instructions, or writes working code. Read Part 1 → Part 2: Displacement Is Not Direction: Why Fidelity Metrics Fail to Rank Near-Baseline Quants Testing perplexity, KLD, and related diagnostics on real quantized models. Part 2 moves from intuition to evidence. Across the full model cohort, perplexity and KLD appear to predict benchmark quality. But that relationship is largely driven by clearly degraded quants. Among models that remain close to baseline quality, the ranking signal collapses. We test 14 fidelity-metric variants across different datasets, context lengths, aggregation methods, and response-only scoring setups. The result remains consistent: fidelity metrics can de...