%PDF-1.4 %âãÏÓ 1 0 obj << /Type /Catalog /Pages 2 0 R >> endobj 2 0 obj << /Type /Pages /Count 4 /Kids [5 0 R 7 0 R 9 0 R 11 0 R] >> endobj 3 0 obj << /Type /Font /Subtype /Type1 /BaseFont /Helvetica >> endobj 4 0 obj << /Type /Font /Subtype /Type1 /BaseFont /Helvetica-Bold >> endobj 5 0 obj << /Type /Page /Parent 2 0 R /MediaBox [0 0 595.28 841.89] /Resources << /Font << /F1 3 0 R /F2 4 0 R >> >> /Contents 6 0 R >> endobj 6 0 obj << /Length 5836 >> stream BT /F2 22 Tf 0.06 0.08 0.12 rg 1 0 0 1 46 789.89 Tm (What Is Speculative Decoding? The AI Term Behind) Tj ET BT /F2 22 Tf 0.06 0.08 0.12 rg 1 0 0 1 46 762.89 Tm (Why Chatbots Got Faster) Tj ET BT /F2 11 Tf 0.72 0.14 0.18 rg 1 0 0 1 46 725.89 Tm (TechRounder PDF Edition) Tj ET BT /F1 9.5 Tf 0.36 0.39 0.46 rg 1 0 0 1 46 709.89 Tm (Live article:) Tj ET BT /F1 9.5 Tf 0.36 0.39 0.46 rg 1 0 0 1 46 697.39 Tm (https://www.techrounder.com/ai/what-is-speculative-decoding-the-ai-term-behind-why-chatbots-got-faster/) Tj ET q 0.82 0.85 0.9 RG 1 w 46 678.89 m 549.28 678.89 l S Q BT /F1 10 Tf 0.24 0.27 0.32 rg 1 0 0 1 46 666.89 Tm (By Vipin PG | Published July 18, 2026 | Updated July 18, 2026 | Format: Explainer | 8 min read) Tj ET BT /F2 13 Tf 0.72 0.14 0.18 rg 1 0 0 1 46 643.89 Tm (In brief) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 623.89 Tm (Speculative decoding is a technique that lets AI models write answers faster without changing what) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 608.89 Tm (they say. A small "draft" model quickly guesses several words ahead, and the main AI model checks) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 593.89 Tm (those guesses in one go instead of thinking up every single word on its own, one at a time. If the) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 578.89 Tm (guesses are right, the AI accepts them instantly. If they're wrong, it corrects them on the spot. The) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 563.89 Tm (result is the same answer you'd always get - just delivered noticeably quicker. It's one of the quiet) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 548.89 Tm (reasons chatbots like ChatGPT, Claude, and Gemini feel snappier today than they did a year or two ago.) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 523.89 Tm (If you've used any AI chatbot recently, you've probably noticed something: the responses show up a lot) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 508.89 Tm (faster than they used to. Words used to trickle out one at a time, almost like someone typing live in) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 493.89 Tm (front of you. Now, especially with newer AI models, entire sentences seem to appear in quick bursts.) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 471.89 Tm (That change isn't an accident, and it isn't just about companies buying more powerful hardware. A big) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 456.89 Tm (part of it comes down to a clever trick called speculative decoding - a term you'll increasingly see) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 441.89 Tm (mentioned in AI model release notes, but one that almost nobody explains in plain language. So let's) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 426.89 Tm (actually break it down, without the engineering jargon.) Tj ET BT /F2 15 Tf 0.08 0.1 0.14 rg 1 0 0 1 46 398.89 Tm (Why AI Chatbots Used to Feel Slow) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 374.89 Tm (To understand why speculative decoding matters, it helps to know how AI models normally generate) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 359.89 Tm (text in the first place.) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 337.89 Tm (When an AI writes a response, it doesn't decide on the whole sentence at once. It builds the answer one) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 322.89 Tm (word \(technically, one "token"\) at a time. After it picks a word, it looks at everything written so far -) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 307.89 Tm (including that new word - and then decides what comes next. Then it does that again. And again. For) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 292.89 Tm (every single word in the response.) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 270.89 Tm (This is called autoregressive generation, and it's a bit like writing a sentence where you're only) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 255.89 Tm (allowed to see one word ahead before deciding the next one, and you have to fully commit before) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 240.89 Tm (moving on. It works well, but it's inherently a one-step-at-a-time process - there's no shortcut to skip) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 225.89 Tm (ahead, because each new word genuinely depends on the one before it.) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 203.89 Tm (The frustrating part is that the computer hardware running these models is usually powerful enough) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 188.89 Tm (to handle many calculations at once. But because of this one-word-at-a-time rule, a lot of that power) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 173.89 Tm (sits idle, waiting for the next word to be decided before it can move forward. It's like having a team) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 158.89 Tm (of ten workers but only ever giving one of them a task at a time while the other nine wait around.) Tj ET BT /F2 15 Tf 0.08 0.1 0.14 rg 1 0 0 1 46 130.89 Tm (The Simple Idea Behind Speculative Decoding) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 106.89 Tm (Speculative decoding tackles this idle-time problem with an idea that's surprisingly intuitive once you) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 91.89 Tm (hear it: instead of making the main AI model figure out every single word by itself, bring in a second,) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 76.89 Tm (much smaller and faster model to make quick guesses first.) Tj ET q 0.86 0.88 0.92 RG 1 w 46 42 m 549.28 42 l S Q BT /F1 8.4 Tf 0.42 0.45 0.5 rg 1 0 0 1 46 30 Tm (TechRounder | Page 1 of 4) Tj ET BT /F1 7.2 Tf 0.42 0.45 0.5 rg 1 0 0 1 46 19 Tm (https://www.techrounder.com/pdf/blog/what-is-speculative-decoding-the-ai-term-behind-why-chatbots-got-faster.pdf) Tj ET endstream endobj 7 0 obj << /Type /Page /Parent 2 0 R /MediaBox [0 0 595.28 841.89] /Resources << /Font << /F1 3 0 R /F2 4 0 R >> >> /Contents 8 0 R >> endobj 8 0 obj << /Length 6143 >> stream BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 789.89 Tm (Here's the easiest way to picture it. Imagine you're texting a friend who knows you extremely well -) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 774.89 Tm (well enough to often guess how you're going to finish your sentence. Instead of waiting for you to type) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 759.89 Tm (out every word, they start typing ahead based on what they expect you to say. When you catch up, you) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 744.89 Tm (either nod along \("yep, that's exactly it"\) or you correct the parts they got wrong. Either way, you end) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 729.89 Tm (up finishing the message faster than if you'd typed the whole thing yourself, word by word.) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 707.89 Tm (That's essentially what happens inside the AI system:) Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 685.89 Tm (- A small, lightweight "draft" model quickly predicts the next several words.) Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 669.09 Tm (- The larger, more capable "target" model - the one actually responsible for quality - checks all those) Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 655.29 Tm (guessed words at once, rather than generating each one from scratch.) Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 638.49 Tm (- Whatever the draft model got right is accepted immediately. Whatever it got wrong is corrected right) Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 624.69 Tm (there, and generation continues from that point.) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 607.89 Tm (The clever part is that checking several guessed words at once is a much faster operation for the) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 592.89 Tm (hardware than generating each word individually. So even though the AI is still technically producing) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 577.89 Tm (the same final answer, it gets there in noticeably fewer steps.) Tj ET BT /F2 15 Tf 0.08 0.1 0.14 rg 1 0 0 1 46 549.89 Tm (How It Actually Works, Step by Step) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 525.89 Tm (Breaking it down further, a typical speculative decoding cycle looks like this:) Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 503.89 Tm (1. The draft model proposes a chunk of words. Instead of one word, it might guess the next four or five) Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 490.09 Tm (words in one shot, based on patterns it has learned.) Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 473.29 Tm (2. The main model reviews all of them together. Because checking multiple words at once is far more) Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 459.49 Tm (efficient for the hardware than generating them one by one, this step happens quickly.) Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 442.69 Tm (3. Matching guesses get accepted instantly. If the draft model's suggestions line up with what the main) Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 428.89 Tm (model would have produced anyway, those words are locked in immediately - no extra work needed.) Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 412.09 Tm (4. Mismatches get corrected on the spot. As soon as the main model finds a guess it disagrees with, it) Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 398.29 Tm (swaps in the correct word and the process restarts from there, with the draft model proposing again.) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 381.49 Tm (This cycle repeats continuously throughout the response. In practice, well-tuned setups can accept) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 366.49 Tm (several guessed words per cycle, which is where the real speed gain comes from. Independent) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 351.49 Tm (benchmarks and research papers on the technique have reported speed improvements in the range of) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 336.49 Tm (roughly 2 to 3 times faster generation for everyday use, and even higher - up to around 4-plus times) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 321.49 Tm (- in more predictable tasks like coding, where output tends to follow familiar patterns the draft model) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 306.49 Tm (can anticipate well.) Tj ET BT /F2 15 Tf 0.08 0.1 0.14 rg 1 0 0 1 46 278.49 Tm (Does This Change What the AI Actually Says?) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 254.49 Tm (This is usually the first concern people have, and it's a fair one: if a smaller, less capable model is) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 239.49 Tm ("guessing" parts of the response, does that make the answer less accurate or lower quality?) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 217.49 Tm (The honest answer is no - and this is really the elegant part of the design. The main, more capable) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 202.49 Tm (model still has final say over every single word. It's not blindly trusting the draft model's guesses;) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 187.49 Tm (it's verifying them. If a guess doesn't match what the main model would have generated on its own,) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 172.49 Tm (that guess is thrown out and replaced. So the end result is mathematically the same response you'd) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 157.49 Tm (have gotten without speculative decoding at all - it's just arrived at through a faster route.) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 142.49 Tm (Researchers often describe this property as "lossless," meaning speed goes up without any tradeoff) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 127.49 Tm (in output quality.) Tj ET q 0.86 0.88 0.92 RG 1 w 46 42 m 549.28 42 l S Q BT /F1 8.4 Tf 0.42 0.45 0.5 rg 1 0 0 1 46 30 Tm (TechRounder | Page 2 of 4) Tj ET BT /F1 7.2 Tf 0.42 0.45 0.5 rg 1 0 0 1 46 19 Tm (https://www.techrounder.com/pdf/blog/what-is-speculative-decoding-the-ai-term-behind-why-chatbots-got-faster.pdf) Tj ET endstream endobj 9 0 obj << /Type /Page /Parent 2 0 R /MediaBox [0 0 595.28 841.89] /Resources << /Font << /F1 3 0 R /F2 4 0 R >> >> /Contents 10 0 R >> endobj 10 0 obj << /Length 6086 >> stream BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 789.89 Tm (The draft model's job isn't to be right all the time. It just needs to be right often enough that verifying) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 774.89 Tm (its guesses in bulk saves more time than generating each word individually would have taken. Even) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 759.89 Tm (when it guesses wrong, you haven't really lost anything - the system simply falls back to normal) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 744.89 Tm (word-by-word generation for that stretch.) Tj ET BT /F2 15 Tf 0.08 0.1 0.14 rg 1 0 0 1 46 716.89 Tm (Who's Actually Using This Right Now) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 692.89 Tm (This isn't a niche research idea sitting in an academic paper somewhere - it's already running quietly) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 677.89 Tm (behind many of the AI tools people use every day. Major AI providers have built speculative decoding) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 662.89 Tm (directly into how their models serve responses at the infrastructure level, which means it happens) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 647.89 Tm (automatically in the background. As a regular user, you never see a setting for it or a toggle to turn it) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 632.89 Tm (on; you just experience the result as a faster reply.) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 610.89 Tm (It's also become a standard part of the toolkit for teams that run open-source AI models themselves,) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 595.89 Tm (using popular serving frameworks that support configuring it directly. And newer AI model families) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 580.89 Tm (have started building a closely related idea called multi-token prediction \(MTP\) directly into the model's) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 565.89 Tm (own architecture, rather than relying on a separate draft model - essentially teaching the model to) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 550.89 Tm (natively guess ahead as part of how it's trained, not just how it's served.) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 528.89 Tm (Interestingly, this technique tends to shine even brighter in AI agents and coding assistants - the tools) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 513.89 Tm (that write code, fill out structured data, or follow repetitive workflows. That's because structured,) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 498.89 Tm (predictable output \(like code syntax or form-style answers\) is exactly the kind of pattern a small draft) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 483.89 Tm (model gets really good at anticipating, which pushes the acceptance rate - and therefore the speed gain) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 468.89 Tm (- even higher.) Tj ET BT /F2 15 Tf 0.08 0.1 0.14 rg 1 0 0 1 46 440.89 Tm (A Few Related Terms You Might Run Into) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 416.89 Tm (Since speculative decoding tends to get mentioned alongside a cluster of other technical terms, here's) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 401.89 Tm (a quick, no-nonsense glossary so none of them catch you off guard:) Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 379.89 Tm (- Draft model: The small, fast model responsible for guessing ahead. Think of it as the "intern" proposing) Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 366.09 Tm (ideas quickly.) Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 349.29 Tm (- Target model: The main, full-size AI model that actually verifies the guesses and has the final word. This) Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 335.49 Tm (is the model whose quality you're actually experiencing.) Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 318.69 Tm (- Multi-token prediction \(MTP\): A related approach where a single model is trained to predict several) Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 304.89 Tm (upcoming words directly, without needing a separate smaller model to do the guessing.) Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 288.09 Tm (- Prefill vs. decode: Two different stages of an AI response. Prefill is the model reading and) Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 274.29 Tm (understanding your prompt; decode is the model writing its answer. Speculative decoding specifically) Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 260.49 Tm (speeds up the decode stage.) Tj ET BT /F2 15 Tf 0.08 0.1 0.14 rg 1 0 0 1 46 237.69 Tm (What This Actually Means for You as a User) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 213.69 Tm (You don't need to configure anything or flip a switch to benefit from speculative decoding - that's) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 198.69 Tm (really the point of it. But understanding it helps make sense of a few things you may have noticed) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 183.69 Tm (recently:) Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 161.69 Tm (- Why some AI tools suddenly started feeling faster without any announcement about "new hardware.") Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 144.89 Tm (- Why coding assistants and structured-output tools \(like ones that fill forms or generate JSON\) often feel) Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 131.09 Tm (especially quick compared to open-ended creative writing requests.) Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 114.29 Tm (- Why AI companies can keep offering faster responses over time even while their models are getting) Tj ET BT /F1 10.5 Tf 0.2 0.23 0.28 rg 1 0 0 1 46 100.49 Tm (larger and more capable, not smaller.) Tj ET q 0.86 0.88 0.92 RG 1 w 46 42 m 549.28 42 l S Q BT /F1 8.4 Tf 0.42 0.45 0.5 rg 1 0 0 1 46 30 Tm (TechRounder | Page 3 of 4) Tj ET BT /F1 7.2 Tf 0.42 0.45 0.5 rg 1 0 0 1 46 19 Tm (https://www.techrounder.com/pdf/blog/what-is-speculative-decoding-the-ai-term-behind-why-chatbots-got-faster.pdf) Tj ET endstream endobj 11 0 obj << /Type /Page /Parent 2 0 R /MediaBox [0 0 595.28 841.89] /Resources << /Font << /F1 3 0 R /F2 4 0 R >> >> /Contents 12 0 R >> endobj 12 0 obj << /Length 3470 >> stream BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 789.89 Tm (In short, it's one of several behind-the-scenes engineering tricks - alongside things like caching and) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 774.89 Tm (smarter memory handling - that let AI companies make their models faster and cheaper to run, without) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 759.89 Tm (asking users to compromise on quality to get there.) Tj ET BT /F2 15 Tf 0.08 0.1 0.14 rg 1 0 0 1 46 731.89 Tm (Frequently Asked Questions) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 707.89 Tm (Is speculative decoding the same as a "lite" or "fast" version of an AI model? No. Lite versions \(like a) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 692.89 Tm ("mini" or "flash" model\) are genuinely smaller, less capable models you choose intentionally, trading) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 677.89 Tm (some quality for speed. Speculative decoding runs invisibly behind the full-size model and doesn't) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 662.89 Tm (reduce its capability at all.) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 640.89 Tm (Can I turn speculative decoding on or off in ChatGPT, Claude, or Gemini? No, not as a regular user. It's) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 625.89 Tm (implemented on the provider's server infrastructure, not as a setting inside the app.) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 603.89 Tm (Does speculative decoding save AI companies money too? Yes - faster generation generally means) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 588.89 Tm (better use of the same hardware, which lowers the cost of serving each response. That's part of) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 573.89 Tm (why providers are motivated to use it even though it doesn't directly show up as a user-facing) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 558.89 Tm (feature.) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 536.89 Tm (Is this a brand-new idea? The core concept was formalized in research from Google and DeepMind) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 521.89 Tm (back in 2023, though it has been refined significantly since then, with newer variations improving how) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 506.89 Tm (accurately the draft step guesses ahead.) Tj ET BT /F2 15 Tf 0.08 0.1 0.14 rg 1 0 0 1 46 478.89 Tm (The Bottom Line) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 454.89 Tm (Speculative decoding is a good reminder that a lot of the AI progress you actually feel - like a chatbot) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 439.89 Tm (suddenly responding faster - doesn't always come from a bigger, smarter model. Sometimes it) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 424.89 Tm (comes from someone figuring out a smarter way to use the model you already have. In this case, the) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 409.89 Tm (fix was refreshingly simple: let a quick assistant guess ahead, and let the expert double-check the) Tj ET BT /F1 11 Tf 0.14 0.16 0.2 rg 1 0 0 1 46 394.89 Tm (work instead of doing it all alone, one word at a time.) Tj ET q 0.86 0.88 0.92 RG 1 w 46 42 m 549.28 42 l S Q BT /F1 8.4 Tf 0.42 0.45 0.5 rg 1 0 0 1 46 30 Tm (TechRounder | Page 4 of 4) Tj ET BT /F1 7.2 Tf 0.42 0.45 0.5 rg 1 0 0 1 46 19 Tm (https://www.techrounder.com/pdf/blog/what-is-speculative-decoding-the-ai-term-behind-why-chatbots-got-faster.pdf) Tj ET endstream endobj xref 0 13 0000000000 65535 f 0000000015 00000 n 0000000064 00000 n 0000000140 00000 n 0000000210 00000 n 0000000285 00000 n 0000000427 00000 n 0000006314 00000 n 0000006456 00000 n 0000012650 00000 n 0000012793 00000 n 0000018931 00000 n 0000019075 00000 n trailer << /Size 13 /Root 1 0 R >> startxref 22597 %%EOF