Red Hat AI 3.5: Scaling and governing AI agents in production

Enterprise teams have moved past early experimentation. Pilots succeed, prototypes impress, and individual teams deliver initial results. The challenge every platform engineering and ITOps team hits next is much harder: operating AI with the same operational rigor as mission-critical AI infrastructure. Moving beyond isolated development sandboxes requires answering 4 questions before scaling: Can we trust it? Can we control it? Can we build on it? Can we measure it? Red Hat AI 3.5 provides the tools to answer all 4. 

The release provides verifiable safety evidence before deployment, operational controls for shared AI infrastructure, a clear path from data to autonomous agents, and built-in observability for complete performance and cost transparency across hybrid cloud environments. Built on open architecture and to integrate with your existing infrastructure, Red Hat AI 3.5 helps organizations operate AI as a trusted shared, multi-tenant service anywhere your data resides.

Can we trust it?

Verify models before deployment

When models only generated text, safety was a quality concern. Now that agents call APIs, query databases, and trigger complex workflows, safety is an operational risk. Manual reviews don't scale. In regulated industries, certification takes months, and platform teams require more than trust to deploy. Red Hat AI 3.5 provides concrete evidence to make deployment calls with confidence.

A team of Red Hat experts delivers validated and optimized models in the AI Model Catalog (generally available (GA)) with its safety work already done. Model safety and security insights surfaces Garak benchmark results across safety, personally identifiable information (PII) exposure, and toxicity, so teams are able to compare how models behave when facing adversarial inputs before committing to one. Models are also validated for tool-calling reliability and ship with deployment commands and configuration files, so teams building agentic pipelines start from a tested baseline.

EvalHub, our model and agent evaluation toolkit (GA), extends that same coverage to models you bring or customize yourself. Teams can run safety-focused benchmarks against risks like prompt injection and jailbreaks, and generate the certifications industry regulators require. Results are visible through a single pane in EvalHub.

Can we control it?

Operational stability on shared GPU infrastructure

Dedicated hardware runs pilots smoothly, but production workloads often share infrastructure. Agentic pipelines, batch jobs, and latency-sensitive requests end up on the same accelerators or nodes competing for the same resources.  Shared infrastructure remains cost-effective right up until a background batch job starves a customer-facing agent. Red Hat AI 3.5 gives platform teams precise controls to enforce priority.

Red Hat AI expands on proven enterprise platform capabilities to deliver native multi-tenancy for AI service providers and AI use cases that require complete hardware-to-software isolation. For teams sharing a single cluster, Red Hat AI 3.5 simplifies fine-grained access control with a new role creation UI in the dashboard — project administrators can define custom role-based access control (RBAC) roles visually, selecting API groups, resources, and permissions from a guided form with ready-made templates. For organizations that require stronger isolation between tenants, Red Hat AI now officially supports running hosted control planes on Red Hat OpenShift Virtualization, giving each tenant a dedicated control plane while consolidating AI workloads on shared virtualized infrastructure.

Tenancy divides resources between teams, but priority-aware serving (GA) manages the workloads within them. Resource admission control, fairness policies, and starvation protection to make sure critical real-time agents meet strict latency targets while background batch jobs wait their turn. Controlled deployments (GA) bring the same rigor to model updates: canary validation routes a fraction of live traffic to a new version for side-by-side comparison, in-flight requests complete before any transition proceeds, and rollback requires zero disruption if the update underperforms. To prevent context truncation in long, multi-turn conversations, vLLM introduces CPU offloading (GA) and Non-Volatile Memory Express (NVMe) storage offload for the key-value (KV) cache, removing hardware memory ceilings. Tool calling for AI agents now works across both standard and distributed deployment modes (GA), so multistep workflows run without interruption.

Production AI rarely lives in one environment. Multicloud Kubernetes serving extends llm-d distributed inference beyond OpenShift onto third-party Kubernetes services, offering a consistent model serving experience across clouds. With this release, distributed inference reaches GA on CoreWeave CKS and Microsoft Azure AKS, while Amazon EKS joins in technology preview. Teams get the same distributed inference stack on the cloud Kubernetes they already operate, without re-architecting for each environment.

Can we build on it?

Bridging the gap between enterprise data and governed AI agents 

Even when an enterprise organization solves problems related to data volume, more friction lies in the path from data stores to functional AI agents. Connecting data warehouses to models requires retrieval pipelines, vector stores like pgvector, training runs, and governance policies, all of which are often scattered across fragmented tools. Red Hat AI 3.5 unifies this workflow by allowing you to deploy and run your AI environments directly where your data already resides, whether on premise or across public clouds, like Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP). It eliminates the need to migrate your proprietary data to a third-party AI service.

AutoRAG (technical preview (TP)), our toolkit for automating retrieval-augmented generation (RAG) evaluation and hyperparameter tuning, adds multilingual processing, contextual retrieval, conversational testing, visual debugging, and pgvector support. The toolkit allows teams to visually tune and connect their existing data stores to AI applications through an interactive test playground. 

Complementing these gen AI tools, AutoML (TP) completes the train-to-production workflow with built-in, out-of-the-box serving runtimes that let you deploy and serve predictive models directly on the platform without needing a separate serving solution. For large-scale data preparation, Kubeflow Spark Operator brings distributed data-processing jobs directly into the data scientist's active workbench environment, putting large-scale data preparation and model serving on the same unified platform. 

Inference-time scaling (GA) adds budget-efficient reasoning to models without expensive retraining. It applies adaptive self-consistency algorithms that stop compute processing the moment a consensus is reached. This prevents wasting expensive GPU compute on straightforward questions, resolving easy queries after 2 to 4 samples while preserving the full processing budget for difficult reasoning tasks.

Responses API and built-in RAG, delivered through OGX, reach GA, providing the foundational interface for building enterprise agentic applications. The Responses API gives developers a standardized, open source surface for multi-turn agent interactions, enabling agents to maintain context across chained reasoning steps, tool calls, and follow-up queries without requiring them to be built directly into the client. Built-in RAG connects those agents directly to enterprise knowledge stores, so responses are grounded in your data rather than training data alone. Together, they provide the API layer that agent templates, guardrails, and evaluation pipelines build on.

Agent templates (GA) and starter kits in AI hub, our central model and developer catalog, turn that groundwork into working agents. They provide preconfigured reference implementations for common patterns like code review, document processing, and research workflows, each built to run inside sandboxed environments with security policies intact from the first commit. 

To govern these workflows, NVIDIA NeMo Guardrails keeps model-user chats on-topic and compliant with enterprise ground rules. Additionally, new capabilities available in technology preview extend the protection to the gateway level, monitoring and validating each agent call to an external tool or API against enterprise security policies. For example, if an agent is tricked by a malicious prompt into attempting a destructive database query, the gateway-level policy can block the command before it reaches your backend systems.

Can we measure it?

Native observability and token showback

Enterprise finance teams track exact costs for traditional infrastructure, but shared GPU clusters have historically remained an unaccountable blind spot. At the same time, data scientists and AI engineers lack visibility into model serving metrics without privileged access to cluster monitoring. Red Hat AI 3.5 resolves both operational challenges.

A centralized, native observability framework (GA) delivers AI performance metrics without manual setup. New observability dashboard tabs (GA) offer deep visibility into llm-d distributed inference clusters, while role-based access controls grant non-admin users project-level visibility into serving metrics. 

Model-as-a-Service (MaaS) showback (TP) introduces billing-grade token metering through admin and user dashboards, delivering precise token-use attribution and consumption metrics so platform teams apply their own cost parameters across business units.

For agentic workflows, visual agentic tracing and debugging (TP) converts multistep execution flows into interactive call-tree visualizations powered by OpenTelemetry and MLflow inside gen AI studio, our hands-on playground for interactive model prototyping.

A GPU-as-a-Service dashboard (GA) provides real-time visibility into GPU deployments, including status tiles for GPU use (GA), hardware model inventories (GA), and dynamic scheduling to track allocated versus borrowed GPU capacity (GA), replacing fragmented command-line checks with a single integrated view.

Operational maturity for the agentic era

The jump from experimental AI pilots to enterprise production is the true test of your organization’s readiness. It requires moving past the simplicity of standalone models to establish continuous runtime governance, optimal resource efficiency, and clear operational visibility.

Red Hat AI 3.5 delivers these critical AI infrastructure updates within an enterprise-grade foundation, built on open source innovation. Sovereign by design, on your terms, across hybrid cloud environments. Red Hat delivers AI operationalized for the real world.

Learn more

• Red Hat AI 3.5 is now available
• Sign up for the What’s new, What’s next session happening on September 22, 2026
• Discover more regarding our product offerings on the Red Hat AI page