Tag: #quantization

2 posts

Evaluating quantised models: perplexity, KL divergence & real task evals

How do I know IQ3_M is still good enough? Measuring the quality loss of a quantisation – with llama-perplexity, KL divergence against the base model, and a small task eval of your own.

26. Juli 2026 • Alain Ritter

#tutorial #quantization #llama.cpp #evaluation #ai #llm

Quantising Qwen3-32B by hand – a 32B model on 16 GB of VRAM

How I quantised Qwen3-32B with llama.cpp and an importance matrix so it runs on a 16 GB GPU: GGUF conversion, imatrix, quant levels compared, VRAM budgeting and layer offload.

26. Juli 2026 • Alain Ritter

#tutorial #quantization #llama.cpp #qwen #ai #llm