07:53
82d ago
r/LocalLLaMA· rssEN07:53 · 05·07
→Why can’t llama.cpp combine speculative decoding methods?
A Reddit user asks whether llama.cpp can run MTP and n-gram speculative decoding together. They tested MTP on Qwen3.6 27B and say n-gram is faster for repeated code spans in agentic coding. The post does not disclose maintainer comments, benchmarks, or the exact limitation.
#Agent#Code#Inference-opt#llama.cpp
editor take
User reports llama.cpp can't stack MTP and n-gram speculative decode — only n-gram runs. Post doesn't explain why.
sharp
The Reddit post exposes one behavior: llama.cpp accepts both MTP and n-gram flags, then only n-gram takes effect. The body is blocked by Reddit 403, so we do not have the command line, llama.cpp commit, Qwen3.6 27B quantization, draft setup, acceptance rate, tokens/s, or maintainer response. This is not a benchmark. It is a user-reported interaction.
I think the instinct behind the question is the useful part. Local inference users often treat speculative decoding methods as stackable speed buffs. MTP, draft-model speculative decoding, and n-gram lookup all “guess future tokens,” but they do not occupy the same layer of the system. MTP depends on model-side future-token heads and a validation path tied to forward passes. N-gram lookup depends on repeated spans already present in context. One is model-internal prediction. The other is contextual copy machinery. Both want to supply the next candidate token block. If llama.cpp lets n-gram win when both are passed, that smells like an arbitration choice, not an obvious missing feature.
The Qwen3.6 27B coding angle makes the report plausible. Agentic coding is full of literal repetition: import blocks, function signatures, JSON schemas, test fixtures, error branches, file paths. N-gram lookup does not need semantic understanding there. It just needs the span to have appeared before. In those traces, prompt lookup can beat a smarter model-side predictor because the answer is sitting in the context window. MTP is better framed as a way to reduce token-by-token serial decoding when the model can predict several future positions cheaply. DeepSeek-style MTP training made that idea more visible, but local inference only benefits when kernels, KV layout, batch shape, and acceptance rates cooperate. The article gives none of those numbers.
My pushback: “both help separately” does not imply “both together help more.” Speculative decoding is bounded by verification cost, rollback cost, and KV-cache update policy. If n-gram hits a 20-token repeated code span, MTP candidates generated for the same step are wasted. If MTP has a high acceptance rate, n-gram lookup can become overhead. Then there is priority. If n-gram proposes 16 tokens and MTP proposes 4, should llama.cpp choose by length, confidence, expected verification cost, or past hit rate? That is a scheduler design, not a CLI toggle.
The comparison I’d use is vLLM or TensorRT-LLM. They tend to manage draft models, Medusa/EAGLE-style paths, and prompt-lookup speculation as separate execution modes or carefully controlled paths. The reason is not aesthetic. Throughput optimization needs a clean batching plan. llama.cpp has an even harder constraint because it spans CPU, CUDA, Metal, ROCm, and small local setups. Any mixed speculative scheduler has to preserve correctness and avoid turning every backend into a pile of special cases.
If someone wants to make this more than a Reddit gripe, the experiment is straightforward. Use the same agentic coding trace on Qwen3.6 27B. Run MTP only, n-gram only, and a hand-written priority hybrid. Report tokens/s, acceptance rate, rollback count, KV rewrite behavior, and repeated-span ratio. Split traces by repetition density: under 20%, 20–50%, and above 50%. Without that, “n-gram feels faster for coding” is a useful hunch but not an implementation argument.
Honestly, I would not pressure llama.cpp maintainers to bolt these together until the scheduler semantics are clear. Local inference has too many flags already. The missing layer is task-aware speculation: coding agents, chat, long-context continuation, and tool-heavy loops do not want the same guesser.
HKR breakdown
hook —knowledge ✓resonance —
46
SCORE
H0·K1·R0