Best LLM Quant Format for Every Apple Silicon Chip

A recent benchmark made the rounds showing MXFP4, OpenAI's newer quantization format, losing to the older Q4_K_M format by 1.8x on the same Mac. gpt-oss-20B in MXFP4 hit 2.6 tokens per second. Qwen3-14B in Q4_K_M hit 4.7. Same prompt, same laptop, same 24GB of unified memory. The format built for speed lost.

Here's the problem: that test ran on exactly one chip, an M2. If you're sitting in front of an M1 Air, an M3 Pro, or a brand-new M4 machine wondering which quant format to download, that single data point doesn't tell you much. The mechanism behind the result matters more than the result itself, because it behaves differently depending on which Apple Silicon generation you own. This guide maps the answer to your specific chip.

Why MXFP4 struggles on older Apple Silicon

The core issue is dequantization, and it comes down to which piece of silicon does the matrix math. Apple's AMX (Apple Matrix coprocessor) has handled INT8 and FP16/FP32 operations natively for years. Q4_K_M, a llama.cpp-native integer format, maps cleanly onto that hardware path and benefits from years of kernel-level optimization inside the GGUF ecosystem. That history is a big part of why it has become a default choice for local inference: it runs fast on almost anything.

MXFP4 is different. It's a 4-bit microscaling floating-point format, designed to give models better dynamic range than integer quants at the same bit width. On modern accelerators built for it, like NVIDIA's Blackwell GPUs, that dynamic range advantage translates into real speed.

But on Apple Silicon before M4, AMX has to dequantize MXFP4 into FP16 before it can even start the matmul. That dequantization step is pure overhead — the tax you pay for using a format the hardware wasn't built to run natively.

Apple added native FP4 support to AMX starting with the M4 generation. That's the detail that changes everything about this decision. On an M1, M2, or M3, MXFP4 pays the dequantization tax on every single forward pass, with no offsetting hardware benefit.

On an M4, that tax should largely disappear, and MXFP4's dynamic-range advantage has room to show up as a speed win. Nobody has published a clean M4 benchmark to confirm that yet. This guide fills that gap with the best available reasoning until someone runs the numbers. For more on this, see more on claude on aws: bedrock vs platform security checklist.

A second factor compounds all of this: reasoning overhead. Models like gpt-oss-20B generate internal "thinking" tokens before producing visible output, and that overhead compounds any per-token slowdown. In the M2 benchmark, gpt-oss produced roughly twice as many total tokens as the visible output required, while Qwen3 added a smaller thinking tax. If you're choosing between a reasoning-heavy MXFP4 model and a leaner integer-quantized one, that compounding effect matters as much as the format itself — and it doesn't go away just because you upgrade your chip. This pairs well with bar chart vs histogram: when to use each in python in depth.

The chip-by-chip decision guide

If you're running an M1 or M2 Mac with 16 to 24GB of unified memory, Q4_K_M or Q5_K_M in GGUF format is the safe, fast default. These formats are the most mature in the llama.cpp ecosystem, they're what Ollama ships by default for most models, and they sidestep the AMX dequantization penalty entirely. Don't reach for an MXFP4 model on this hardware expecting the marketing-claimed speed; you'll get correct output, just slower than the integer alternative doing comparable work.

On an M3 with more headroom, the calculus doesn't change much, because the AMX FP4 support that matters didn't land until M4. If you've got 32GB or more of unified memory on an M2 or M3, you can move up to Q6_K or Q8_0 for better output quality without a meaningful speed penalty, since your bottleneck is compute pattern, not memory capacity. Still steer clear of MXFP4 unless quality, not latency, is your priority for a specific model.

An M4 or later machine is where this gets genuinely interesting — the picture is still unresolved. Native FP4 support in AMX means the dequantization overhead that crushed MXFP4's numbers on M2 should shrink dramatically. That doesn't guarantee MXFP4 wins outright, because Q4_K_M's kernel-level optimizations, built up over years of llama.cpp development, are still a real advantage.

Also read: related topic: best nam amp profiles for metal, clean, and bass tones

The honest answer for M4 owners right now: test both on your own workload. Pull the same model in both formats if it's available, run the same prompt three times each, and compare warm-cache tokens per second. The gap that showed up on M2 may shrink, flip, or vanish, and nobody has published the numbers that settle it yet.

If you're outside the Apple Silicon world entirely, the rules of thumb are simpler. On Linux with an NVIDIA GPU, FP16 or BF16 formats run natively through tensor cores, so quantization tradeoffs look very different than on a Mac. On CPU-only Linux boxes, Q4_K_M remains the most broadly compatible and fastest option, same as on older Macs, because its optimization history isn't tied to any single vendor's silicon.

The underlying lesson here isn't about MXFP4 versus Q4_K_M as a permanent ranking. It's that quant format speed is a hardware-dependent claim, not a universal one — and marketing copy for a new format rarely specifies which generation of chip it was validated on.

If you own an M1, M2, or M3 Mac, Q4_K_M and Q5_K_M are still your fastest, most reliable choices for local inference in 2026. If you own an M4, you're standing at the actual frontier of this question, and running your own three-trial benchmark on your own prompts will tell you more than any spec sheet. The format war isn't over — it's just waiting on someone with an M4 and six spare minutes to settle it.