SmartQuant.md

SmartQuant

• optional: extract quant distribution from existing GGUF
  • default: standard llama.cpp distribution
• optional: adapt quant distribution based on rules or imatrix
  • default: skip
• optional: adapt quant distribution to almost compatible model
  • default: skip
• apply updated quants to
  • the model used for extraction: only requant changed tensors
    • example: apply minor quant optimizations
  • a compatible model: apply known quant distribution
    • example: apply MoE quant distribution to REAP model
  • an almost compatible model: needs previous adaption step
    • example: apply dense quant distribution to REAP model