How Speculative Decoding and MTP Speed Up Generation

It fits. What if it is still slow?

In EP02, I worked through how to share 128GB. Fitting a model and getting an answer quickly are separate problems, though.

EP01 described decode as choosing the next token after the ones already generated. If a large model makes me wait one token at a time, can I shorten that wait?

One approach lets a smaller model propose several tokens before the larger model checks them. It is called speculative decoding. Adding a model to make things faster sounds slightly odd at first.

The Mac Studio has not arrived. This is another note about what I want to compare, not a report of a successful run.

A smaller model proposes; the target checks

The speculative decoding paper describes a lightweight draft model proposing tokens and the answer-producing target evaluating several positions together. With the candidates already available, a causal mask lets each position see only earlier candidates during a verification pass.

Draft generation itself may still be sequential. Future tokens have not become independent. The opportunity is to reduce separate calls to the expensive target.

Propose → verify → continue, with words

An imaginary, simplified greedy-verification example

Input (Korean example) · 오늘 날씨가

Draft · proposes candidates

1. 1정말Proposed
2. 2좋아서Proposed
3. 3산책을Proposed
4. 4갔다Proposed

Target · verifies and decides

1. 1정말Accepted
2. 2좋아서Accepted
3. 3산책을Rejected
4. 4갔다Discarded

Accepted prefix · 정말 → 좋아서 → 밖에서… (target continuation is illustrative too)

1. Draft proposes candidates
2. Target verifies: reject the third candidate
3. Keep the two accepted prefix candidates
4. Discard the candidate after rejection too
5. Continue at the rejected position with a target token

산책을 is rejected; 갔다 is discarded because its prefix was rejected. Discarding does not mean the target independently rejected that token.

Words are shown as if they were tokens for illustration. This is not an actual tokenizer segmentation or model execution result.

Sampling may use probabilistic acceptance/rejection and a corrected distribution. Do not generalize this greedy example to matching only the most likely token.

Original diagram and caption

In the word example, 정말 and 좋아서 survive, but 산책을 is rejected. 갔다 was proposed using that rejected prefix, so it cannot simply be appended. Generation continues with the target’s token at the rejection point, and discarded candidate state must be removed from the cache.

Sampling needs more than matching tokens. The paper’s distribution-preserving method uses acceptance probabilities and a corrected distribution after rejection. Those conditions matter when claiming the same output distribution as target-only generation.

Does a high acceptance rate guarantee a win?

Acceptance rate tells me how many proposed tokens were accepted. It does not, by itself, tell me how long I waited.

Drafting, verifying candidates and restoring the cache all cost time. An external draft model also needs weights and cache memory. Longer drafts mean more verification work, and an early rejection discards later candidates. These costs and accepted lengths are worth checking alongside the llama.cpp speculative settings.

Four outputs, illustrative elapsed times

Unlike the rejection example above, this example assumes all four candidates are accepted.

Ordinary decode 400ms

1 · 1002 · 1003 · 1004 · 100

Speculative 180ms

40140

0100200300400ms

Ordinary decode

4 × 100ms = 400ms

Speculative

Draft 40ms + verify 140ms = 180ms

Illustrative speed ratio

400 / 180 ≈ 2.22× · 220ms less elapsed time

These are illustrative fictional values, not benchmark results. Performance varies with acceptance, drafting and verification costs, hardware and runtime.

Output is capped at four tokens for this comparison. Any extra token emitted after verification is excluded by the output limit; synchronization and state-management overhead are assumed included in the two fictional intervals. This is not an exact execution trace for every implementation or a TTFT chart.

Original diagram and caption

Ordinary decode asks the target to generate the next token repeatedly. With candidates already available, the target can verify several positions in parallel. If enough candidates become confirmed output at low cost, fewer sequential target calls are needed. The timing chart illustrates that possibility; it is not a result from my Mac.

I also want to separate what each metric counts.

Metric	Meaning to record
Acceptance rate	Accepted draft candidates divided by proposed candidates, with the aggregation window stated
Average accepted length	Mean number of accepted draft candidates per verification round
Confirmed tokens per round	Actual output including accepted candidates and correction/bonus tokens; state output-limit and EOS treatment
Total response latency	Input processing through completion, including drafting, verification and synchronization waits
Decode throughput	Confirmed output tokens per elapsed time in a defined decode interval; distinguish inclusion of prefill

If the target is already small and fast, or drafts are rejected often, the additional work might not pay off. That is a hypothesis to test on my machine. I want separate records for “more proposals accepted” and “less waiting.”

Is MTP another name for the same feature?

MTP means Multi-Token Prediction: training to predict multiple future tokens. The MTP paper describes multiple prediction heads on a shared model trunk. Speculative decoding is the inference procedure for proposing and verifying candidates. MTP can supply candidates to that procedure.

The DeepSeek-V3 materials distinguish main weights from MTP module weights. That differs from loading another general-purpose small model. It does not establish zero extra memory or support in every engine. I need to check whether the deployed checkpoint includes the relevant weights and whether the runtime uses them during inference.

This is also separate from my plan for a small decision model such as JevMLX. Choosing where to send a task does not automatically make that model a compatible or effective next-token drafter.

What does “the same output” mean?

Greedy decoding selects the highest-scoring token; sampling draws according to configured probabilities. Matching greedy results, preserving a sampling distribution and producing identical text with the same seed are different claims. Numerical precision and implementation details need attention too.

The distinction appears in real engines. The ds4 documentation I checked describes its nonzero-temperature default as opportunistic sampling and provides --mtp-exact-sampling for preserving the target distribution. Mixing a fast default with an exact configuration makes a speed comparison hard to interpret.

I will first compare enabled and disabled operation for the same target. Distribution-preserving and heuristic modes belong in separate results. A few similar answers would not prove that the sampling distribution was preserved.

Where would I start on a Mac?

At small batch sizes, repeatedly reading large target weights can make memory bandwidth important. After reading an analysis of inference compute and bandwidth, I want to see whether confirming several tokens per verification pass amortizes that cost. The effect depends on model structure and kernels. I need confirmed tokens per round alongside actual elapsed time.

An external draft adds weights and KV cache. MTP can also require modules and execution buffers. Even with 128GB, speed alone is not enough to decide whether to leave it enabled. I want to check whether both paths fit within the budget from EP02.

MLX is an array framework for Apple Silicon; MLX-LM below runs LLMs on top of it. GPU execution and synchronization between drafting and verification matter in Metal/MLX paths. MLX synchronize waits for stream work to finish. Timing submission alone can miss completion latency, so I need to check the runtime’s measurement boundaries first.

I checked the documents and code on October 9, 2026. These are candidates to investigate, not configurations I have successfully installed. Links below are pinned to the inspected versions.

Candidate	Inspected path	Still to check before running
MLX-LM	Generation with an external draft model	Matching tokenizer, trimmable prompt cache, selected model and sampling combination
llama.cpp	Distinct strategies including external drafts and draft-mtp	GGUF MTP weights and architecture, Metal execution and cache restoration
ds4	Model-specific speculation and sampling settings	Selected model on Metal, exact mode and combinations with other features
oMLX	VLM MTP path delegated to mlx-vlm; Gemma 4 assistant and Qwen 3.5-family drafter types inspected	Actual selected model, dependency version and sampling combination untested
Slipstream	Only a Metal MTP selection kernel inspected	End-to-end host proposals, verification, cache restoration and sampling support unconfirmed

A file or option existing does not establish stable operation for my chosen model. These code checks do not guarantee production-ready support or performance on my Mac.

Once it arrives, enable one feature at a time

On the ordered Mac Studio M5 Max with 128GB unified memory and a 1TB SSD, I want a target-only baseline, followed by one supported draft model or MTP path. I will hold target weights, quantization, input, context, output limit, batch size and sampling policy fixed. Warm-up and prompt-cache reuse need separate conditions, with engine, kernel versions and draft settings recorded.

Record	Question
TTFT, prefill, decode and total response time	Does a generation benefit also reduce the initial wait?
Proposed/accepted tokens and accepted length per round	How was acceptance defined, and how much draft work was discarded?
Draft, verification and cache-restoration time	Which overhead can the runtime actually expose?
Peak memory, memory pressure and swap	How much room does the additional model or MTP path need?
Output validity and task correctness	Are JSON structure and results on fixed tasks maintained?

Metric	Basic decode	Speculative decode
TTFT	To be measured	To be measured
Decode tok/s	To be measured	To be measured
Total response time	To be measured	To be measured
Peak memory	To be measured	To be measured
Average accepted length	Not applicable	To be measured
Verification rounds	Not applicable	To be measured
Draft cost	Not applicable	To be measured

I will separate short and long responses. Matching output limits does not guarantee matching lengths, so matched actual-token-count comparisons and naturally terminated responses need separate treatment. Runtime versions will be fixed too. If matching sampling, context or warm-up conditions is not possible, I will state the differences. All entries are measurement plans for after delivery.

I would like repeated runs of the same input list, with medians and variation. Actual generated token counts matter so shorter answers do not masquerade as faster generation. If the engine does not expose internal timings, I will record total latency rather than invent a breakdown.

What I haven’t measured

I do not yet know which target and draft will work well together or how much MTP might help. Extra memory alongside development tools and Hermes is another open question.

From the material so far, reducing the large model’s work first adds work elsewhere. I want to find out whether that trade pays off for me. Before leaving it enabled, I would rather compare the waiting with it on and off.

One question remains beyond speed: where do the weights of a really large MoE model live? In EP04 I will explore loading required experts from SSD and the I/O cost that brings.