Overcoming the cost and complexity of AI inference at scale

Operationalizing AI models at scale is a critical challenge for IT leaders. While the initial cost of training a large language model (LLM) can be significant, the real and often underestimated expense is tied to inference.

AI inference—the process of using a trained model to generate an output—is the most resource-intensive and costly part of an AI application, especially because it happens constantly during production. Inefficient inference can compromise an AI project's potential return on investment (ROI) and negatively impact customer experience due to high latency.

The full-stack approach to AI performance

Effectively serving LLMs at scale requires a strategic, full-stack approach that addresses both the model itself and the serving runtime. A single approach is not enough. Achieving high performance and cost efficiency requires a dual focus—managing resource consumption and maximizing throughput.

Optimizing the AI model

A strategic part of this approach is model compression, which reduces a model's size and resource requirements without compromising accuracy.

Quantization is a key technique for model optimization. It reduces the precision of a model's numerical values—like its weights and activations—from standard 16-bit to lower formats such as 8-bit or 4-bit. This significantly shrinks the model’s memory footprint, allowing it to run on less hardware.

Sparsity is another effective method, which makes models more efficient by removing unnecessary connections (weights). This makes the network smaller and faster with minimal impact on accuracy.

Optimizing the inference runtime

Optimizing the serving runtime is equally important. Basic runtimes often struggle with inefficient GPU memory usage and slow token generation, leading to idle GPUs and high latency. A high-performance runtime maximizes the use of expensive GPU hardware and reduces latency. 

The open source vLLM project has become an industry standard for high-performance inference because it addresses these runtime limitations with techniques optimized for efficiency.

• Continuous batching minimizes GPU idle time by concurrently processing tokens from multiple requests. Instead of handling a single request at a time, it groups tokens from different sequences into batches. This approach significantly improves GPU utilization and inference throughput.
• PagedAttention is another example. This novel memory management strategy efficiently handles large-scale key-value (KV) caches, allowing for more concurrent requests and longer sequences while reducing memory bottlenecks.

Enabling distributed, large-scale AI

For enterprises with high-traffic applications, single-server deployments are often insufficient. The open source llm-d project builds on vLLM's capabilities to enable distributed, multinode inference. This allows organizations to scale AI workloads across multiple servers to handle increasing demand and larger models while maintaining predictable performance and cost-effectiveness.

llm-d is an open source control plane that enhances Kubernetes with specific capabilities needed for AI workloads. The project focuses on features that impact inference performance and efficiency, including:

• Semantic routing: llm-d uses real-time data to intelligently route inference requests to the most optimal instance. This improves efficient resource use and reduces costly over-provisioning.
• Workload disaggregation: This separates the prefill and decode phases, so the most optimal resource is used for the right task.
• Support for advanced architectures: llm-d is designed to handle emerging model architectures—like mixture of experts (MoE)—that require orchestration and parallelism across multiple nodes.

By creating a flexible control plane that works across different hardware and environments, the llm-d community is working to establish a standard for enterprise AI at scale.

How Red Hat simplifies AI at scale

Adopting AI at the enterprise level involves more than just selecting a model. It requires a strategy for development, deployment, and management across a hybrid cloud infrastructure. Red Hat offers a portfolio of enterprise-ready products designed to simplify and accelerate this process, from initial model development to at-scale inferencing.

Red Hat AI

The Red Hat AI portfolio provides a full-stack approach to AI optimization. This integrated offering includes Red Hat Enterprise Linux AI (RHEL AI), Red Hat OpenShift AI, and Red Hat AI Inference Server.

• RHEL AI provides a foundation for AI development, packaging RHEL with key open source components like IBM's Granite models and libraries like PyTorch. The platform is portable, able to run on premise, in the public cloud, or at the edge.
• Red Hat OpenShift AI is built on Red Hat OpenShift and is designed for managing the full AI lifecycle. It provides a consistent environment for data scientists, developers, and IT teams to collaborate. It scales AI workloads across hybrid cloud environments and simplifies managing hardware accelerators.
• Red Hat AI Inference Server optimizes inference by providing a supported distribution of vLLM, built for high-throughput, low-latency performance. Delivered as a container, it’s portable across diverse infrastructure and includes a model compression tool to help reduce compute usage. For scaling beyond a single server, Red Hat AI Inference Server works with the open source llm-d project.

For IT leaders, the path to a full-stack, hybrid cloud AI strategy is the most effective way to operationalize AI at scale. Red Hat AI provides a consistent foundation to help organizations move from AI experimentation to full-scale, production-ready AI, built on our vision of, "any model, any accelerator, any cloud."

Learn more

To begin your organization's journey to simplified, scalable AI, explore the resources available on the Red Hat AI website.