From fine-tuned model to cheaper and faster inference: Speculator training on Red Hat OpenShift AI with Kubeflow

Your organization spent months fine-tuning a large language model. Maybe it's a 70 billion parameter model trained on internal medical records, legal documents, or customer support transcripts. It's accurate. It's unique. It's yours.

Now it's deployed in production, serving real users, and burning through GPU hours at $2 to $5 per hour per NVIDIA H100 GPU. And here's the part most organizations don't realize until the invoice arrives: 70 to 80 percent or more of enterprise AI spending goes to inference, not training. The model is done learning. The ongoing cost is answering questions.

That cost is much higher than it needs to be. The market for fine tuned, domain specific models has grown at a 38%+ compound annual growth rate since 2023, and enterprise generative AI adoption has surged from 37% in 2023 to over 90% in 2026. As more organizations deploy their own customized LLMs, inference efficiency is no longer a nice to have. It's the single largest lever for controlling AI infrastructure costs.

The root cause? During token generation, a 70 billion parameter model must read all 140 GB of its weights from GPU memory for every single token it produces. On an NVIDIA H100 GPU, the compute units sit idle more than 95% of the time (see figure 1) during this process because the bottleneck is memory bandwidth, not computation. You're paying for a supercomputer that spends most of its time waiting for data to move from memory to processor.

How speculative decoding turns idle compute into throughput

Speculative decoding, introduced in 2022, is a technique that makes LLM inference dramatically faster without changing the model's output. And the output is mathematically identical. The idea is simple. Imagine you need to look up answers in a massive encyclopedia. Each answer requires carrying all 50 volumes to your desk (slow), reading one sentence (fast), carrying them back, and repeating. That's how normal token generation works: Read 140 GB, produce one token, repeat.

Now imagine you have a pocket reference guide that can guess the next several answers. You carry the encyclopedia once, check all the guesses in parallel, and accept the correct ones. Checking 5 guesses costs almost the same as looking up 1 answer, because the expensive part (carrying the books) is identical and the actual reading is trivial by comparison.

That's speculative decoding. A tiny "draft" model proposes multiple candidate tokens in rapid succession. The large "verifier" model (the original LLM you are serving) then checks all candidates in a single forward pass (a complete read of all model weights). Correct predictions are accepted immediately. At the 1st wrong prediction, the verifier's own token is used, and the cycle restarts (figure 2).

The output is identical because of a technique called rejection sampling. Rejection sampling works by accepting draft tokens only when the verifier model would have produced them with at least the same probability, ensuring the final output distribution is provably identical to running the verifier alone.

The result: The EAGLE3 paper reports 4x to 6x speedup at temperature 0 (fully deterministic output), with real world vLLM production deployments consistently achieving 2.5x to 3.5x reductions in inter-token latency, depending on model and workload. No accuracy is lost.

What makes this technique particularly low risk is its worst case behavior. Even if the draft model gets every prediction wrong, the verifier still produces 1 correct token for each forward pass, exactly the same as standard decoding. Speculative decoding can only help or break even, making it a safe default for any production deployment.

Users see the difference immediately. Instead of text appearing 1 token at a time, responses arrive in rapid bursts of 3 to 5 tokens. A reply that took 8 seconds now takes under 3 (figure 3).

The problem with off-the-shelf draft models

If speculative decoding is this effective, why isn't everyone using it? They are for base models. The open source Speculators library (maintained by the vLLM project) makes it straightforward to train draft models, and pre-built draft models are available on HuggingFace in the RedHatAI namespace. vLLM (the inference engine powering Red Hat OpenShift AI) supports speculative decoding out of the box.

The challenge is that each draft model is tightly coupled to a specific verifier. A draft model trained on 1 verifier's token distribution (the probability the model assigns to each possible next word) does not predict accurately for a different verifier. This means that whenever you fine-tune or swap to a different base model, the existing draft model's guesses no longer match, and a new draft model must be trained to restore the speedup.

The metric that determines whether speculative decoding helps or hurts is called the acceptance rate. It's the percentage of draft model guesses that the verifier accepts. A related metric, acceptance length, measures how many consecutive tokens the draft model gets right before 1 is rejected by the verifier. With a well matched draft model, acceptance rates are high and you see the full 2x to 3x speedup. With a mismatched draft model, acceptance rates drop (figure 4) below a break even threshold where the overhead of running 2 models actually makes inference slower than running the verifier alone.

This is the gap. vLLM already supports speculative decoding at serving time. Customers can deploy a draft model today. The missing piece is getting a draft model that actually matches their fine tuned verifier.

Speculator training: Teaching a draft model your model's language

Speculator training closes that gap by building a custom draft model directly from your fine tuned verifier's internal representations. This approach is called EAGLE3, published at NeurIPS 2025 and maintained as part of the open source Speculators library by the vLLM project. EAGLE3 works by reading the verifier model's hidden states: the intermediate representations that the model computes at each internal layer as it processes text. Think of hidden states as the model's working notes, partial calculations at each layer that capture increasingly refined understanding of the text. These working notes already exist in GPU memory during normal inference, so extracting them adds negligible overhead.

What sets EAGLE3 apart from earlier approaches is that it does not require a separate standalone model. Instead, a lightweight draft model attaches directly to the verifier (figure 4), reading its working notes at multiple layers (early, middle, and late) to build a complete picture of what the model is processing. The draft model adjusts its behavior on each request: it proposes longer token sequences when the text is predictable and shorter ones when the output is uncertain, using compute efficiently regardless of content.

The Eagle3 draft model is remarkably small and uses just 1 transformer layer. The training process (figure 5) runs the verifier on a training dataset, captures its hidden states at multiple layers, and trains the draft model until its predicted token probabilities align with the verifier's. The verifier model is never modified. It stays frozen throughout, meaning there is no risk of degrading model quality.

Smarter data generation makes speculator training more accessible

A concern with speculator training is the cost of generating training data. Running a 70 billion parameter verifier to extract hidden states is GPU intensive. But recent research from the Red Hat AI team has found a way to reduce this cost.

In a recent study, the team demonstrated that same family cross distillation (training a smaller model using data generated by a larger model in the same architecture family) can match or outperform traditional self-distillation (where each model generates its own training data). Instead of running your specific verifier to generate training data, you can generate that data once using the largest model in the family and reuse it across all smaller models.

For example, if you're training Speculators for multiple Qwen3 family models, you only need to run the Qwen3 235B model once to generate training data. That same data works for training draft models for Qwen3 30B, Qwen3 8B, and other models in the family. In practice, this means a team training Speculators for multiple models in the same family can cut data generation GPU hours by more than half compared to the traditional approach.

The research also found that 3 to 4 training epochs (complete passes through the training data) is the optimal stopping point, with diminishing returns beyond that. Shorter training cycles mean fewer GPU hours and faster iteration (see figure 6).

What this means for your GPU bill

With both training costs falling and inference acceleration well established (figure 7), the full economics of speculator training become hard to ignore. At current cloud rates of $2 to $5 per hour per H100 GPU, a 3x inference throughput improvement translates to one of 2 outcomes:

• Serve the same workload on fewer GPUs: A deployment that currently requires 12 H100 GPUs for inference could potentially run on significantly fewer GPUs with speculative decoding enabled, depending on workload characteristics and GPU utilization patterns.
• Serve more requests on the same hardware: If demand is growing, speculative decoding lets you handle substantially more request volume without provisioning additional GPUs, delaying or avoiding a costly infrastructure expansion.

The draft model adds negligible overhead: Roughly 1 GB of additional GPU memory for a 70B verifier, compared to the 140 GB the verifier already occupies. And because the rejection sampling guarantee still holds, there are no accuracy tradeoffs to evaluate and no need to retrain downstream systems.

Coming to Red Hat OpenShift AI

Red Hat is building managed speculator training into Red Hat OpenShift AI, powered by the open source Speculators library and Kubeflow. The goal is to let ML teams train custom EAGLE3 draft models for their fine-tuned verifiers through a managed workflow, without writing custom PyTorch scripts, coordinating multi GPU allocation manually, or managing the complexity of hidden state extraction. We will share more details as we approach release. In the meantime, you can start exploring speculative decoding today:

• Try speculative decoding at serving time: vLLM supports it now. If you deploy models with vLLM, you can enable speculative decoding with a pre-built draft model and see immediate latency improvements on base models.
• Explore the Speculators library: The vLLM speculators project is open source and includes tutorials for training EAGLE3 draft models.
• Browse pre-built draft models: Red Hat has published speculator models on HuggingFace under the RedHatAI namespace.
• Read more on data generation strategies: The Smarter data generation for faster Speculator training study covers the cross distillation findings that make speculator training more cost effective.

That 70 billion parameter model your team spent months training is still reading 140 GB of weights for every token it produces. With a custom speculator, most of those reads produce three to five tokens instead of one, and nothing about the model changes. The only thing that shrinks is the invoice.