Back to the shelf
Cover of AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

Read

AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

Lin et al.

Artificial Intelligence Computer Science Technology

This paper proposes Activation-aware Weight Quantization (AWQ), a hardware-friendly, training-free method for low-bit weight-only quantization of large language models. AWQ identifies a small fraction of salient weight channels using activation magnitudes rather than weight magnitudes, and protects them via per-channel scaling to reduce quantization error without mixed-precision representation, achieving strong compression while preserving generalization.

Read the paper

Key takeaways

  • AWQ observes that only 0.1% to 1% of weight channels are salient for LLM performance, and that identifying these channels by activation magnitude rather than weight magnitude is the correct signal for reducing quantization error.
  • Rather than keeping salient weights in FP16 (which is hardware-inefficient), AWQ mathematically derives that scaling up salient channels before quantization reduces their relative error, then applies an equivalent inverse scale to activations, maintaining a hardware-friendly uniform-precision format.
  • AWQ requires no backpropagation or reconstruction and uses only a small offline calibration set to collect activation statistics, avoiding overfitting to specific domains and preserving out-of-distribution generalization.
  • Across LLaMA, Llama-2, OPT, Mistral, and Mixtral model families under INT3 and INT4 quantization, AWQ consistently achieves lower perplexity than round-to-nearest (RTN) and GPTQ baselines.
  • AWQ generalizes to instruction-tuned models (Vicuna) and, for the first time, to multi-modal language models (OpenFlamingo-9B, LLaVA-13B, VILA), reducing quantization degradation on COCO captioning from 4.57 to 1.17 CIDEr points under INT4-g128.
  • The accompanying TinyChat inference framework uses on-the-fly dequantization, SIMD-aware weight packing, and kernel fusion to deliver more than 3x speedup over the HuggingFace FP16 baseline on desktop and mobile GPUs, enabling deployment of 70B-scale models on edge hardware.

Reflections

AWQ is one of the reasons I can run a capable model on my own hardware instead of renting one by the token. The insight is the kind of asymmetry I look for everywhere: only about one percent of weight channels actually carry the model’s performance, and you find them by watching activations, not weight magnitude. Get the right one percent and you can quantize the rest to four bits without the model noticing. I care about this less as a compression trick and more as sovereignty. Every capability I can push onto local, quantized hardware is one I don’t have to rent, meter, or ship my data away to use. The edge-deployment story is the whole point for me: a large model running on a machine I own is the difference between building on infrastructure and being a tenant on someone else’s.