Opening the black box: Profiling a secured agentic pipeline on Red Hat OpenShift AI

As enterprises move to autonomous agentic pipelines, 2 critical questions emerge: Can I add security to my agents without killing performance? And, Where should I focus optimization effort to get the most out of my agentic system?

Most inference benchmarks test the model in isolation—a raw request to vLLM, a prompt in, tokens out. But production agentic systems don't work that way: Every request passes through an agentic harness that assembles context, manages sessions, injects tool schemas, and optionally provisions a secure sandbox for code execution, all before a single token is generated. Understanding where latency lives across this pipeline is essential for capacity planning, deployment sizing, and knowing where optimization effort actually pays off. 

This is Part 1 of a 3-part series on benchmarking agentic AI performance on Red Hat OpenShift AI. Here’s what’s in store in future installments:

• Part 2: Multiturn tool calling: Measuring real agentic workloads with iterative reasoning, tool invocations, and sandbox execution
• Part 3: Scaling beyond 10 agents on a single cluster: Capacity planning, resource contention, and GPU saturation on a shared OpenShift AI cluster

In this post, we’ll demonstrate that sandbox security adds negligible overhead—minimal resource consumption and under 5% of end-to-end latency —making isolation a default-on decision rather than a performance trade-off. We’ll also show that the GPU has significant spare capacity even at 10 concurrent agents, revealing that the performance optimization opportunity lies in the agentic harness pipeline, not necessarily in faster hardware. This means that, for engineering teams deploying agentic AI on OpenShift AI, meaningful performance gains will come from understanding the agentic pipeline and right-sizing your cluster for the workload. We share our methodology, tooling, and results so that engineering teams can apply the same approach to benchmark and size their own agentic deployments on OpenShift AI.

Methodology 

We benchmarked 2 progressively more complex configurations of the agentic AI pipeline, as laid out in Table 1. This layered design lets us isolate and attribute performance overhead to each individual component, rather than measuring the full pipeline as a black box.

Table 1: Benchmarked configurations

Experiment design 

We used a full-factorial design: 3 stack configurations × 3 concurrency levels (C=1, C=5, and C=10, with C being the number of agents processing requests concurrently against the same vLLM backend), totaling 9 runs. Each agent is an independent OpenClaw instance driven by a dedicated GuideLLM worker that sends 500 prompts per dataset to /v1/responses (streaming). Each prompt is sent only after the previous response completes. Between runs, we restarted the vLLM serving runtime to clear the prefix cache. The infrastructure is detailed in Table 2.

Table 2: Profiling infrastructure 

A key design choice: tool calling is disabled and all requests consist of a single turn. This isolates the pure framework overhead—context assembly, prompt expansion, sandbox provisioning, SSE forwarding—from the variable latency of actual tool invocations or accumulating conversation history. Part 2 of this series will layer on multiturn tool-calling workloads, building directly on this baseline.

Pipeline stage timings (context assembly, sandbox initialization, response processing) come from OpenTelemetry (OTEL) tracing. Table 3 outlines the metrics that are collected for each experiment run.

Table 3: Experiment metrics

Profiling the agentic stack

With the methodology in place, we turn to the results. Using OpenTelemetry tracing, we measured per-stage latency for each configuration at every concurrency level. The analysis is organized into 3 areas: 

• Latency breakdown: Decomposing the request path to identify where time is spent across the pipeline.
• Sandbox startup: Measuring the cost of provisioning isolated execution environments.
• Scale evaluation: Profiling agent resource consumption and concurrency behavior at C=1, 5, and 10.

Together, these answer the central question posed in the introduction: What does each layer of the agentic stack actually cost?

The full pipeline 

OTEL tracing decomposes every request into 5 measurable stages. The timing diagram in Figure 1 shows the complete request flow through the agentic harness and OpenShell pipeline at C=1, with each stage corresponding to the OTEL spans described in our methodology.

Figure 1: Processing time breakdown for the agentic pipeline at C=1

Table 4: Average agent internal pipeline latency (ms)

Figure 1 and Table 4 isolate agentic harness internal pipeline—response processing is 498 ms. Classification is the most time-intensive stage in the pipeline, taking longer than model prefill—an optimization opportunity for any orchestration framework. Context assembly cost is 123 ms due to token expansion. From C=1 to C=5 to C=10, response processing time increases from 498 to 1,408 to 2,437 ms. End-to-end rises with concurrency (from 5,290 to 8,586 to 11,361 ms), but not only because the GPUs are busier—at C=10, the agent layer spends 2,437 ms on post-inference processing, including token classification, tool detection, and stream forwarding.

Figure 2: Processing time breakdown for the secured agent pipeline (with OpenShell) at C=1

Table 5: Processing time breakdown for the secured agent pipeline (with OpenShell) at C=1

Figure 2 and Table 5 add OpenShell. At C=1, end-to-end is 5,814 ms, 524 ms (10%) more than with only the agentic harness; sandbox initialization is only 140 ms of that delta. The larger increment is response processing, which is 895 ms with the sandbox versus 498 ms without it. Context assembly also grows (219 versus 123 ms) due to the additional sandbox context injected into the prompt, and response processing increases as the agentic harness handles the extra work that comes with sandbox-enabled requests.

Figure 3: Pipeline processing stages with 1 and 10 agents

Figure 3 shows the same pipeline stages at P95. At C=1, response processing is already the widest segment. At C=10, it reaches 3,124 ms—an increase by a factor of nearly 5. Sandbox initialization follows the same pattern: 227 ms at C=1, 1,228 ms at C=10. Prefill—the only stage that runs on the GPU—moves from 50 to 150 ms. The framework stages degrade by a factor of 4 to 5 under concurrency while the GPU stage barely changes. 

Agentic stack resource utilization

CPU and memory utilization

The latency numbers in the last section already hint at it: A lot of the request never touches the GPU. That work still uses CPU and memory. Each OpenClaw instance assembles context, classifies tokens, and forwards a stream. Multi-agent deployments have to be sized for that load, not only for the GPUs. Figure 4 shows what that looked like at C=1, C=5, and C=10.

Figure 4: OpenClaw resource utilization On OpenShift AI cluster from 1 to 10 concurrent agents

At C=1, 1 agent pod averages 1.8 CPU cores (P95 2.4) and 0.69 GiB (P95 1.15 GiB). Ten pods under C=10 load sum to 13.9 cores average and 18.6 cores P95, with 6.03 GiB and 7.25 GiB, respectively. The cluster is 24 vCPU; at P95, 10 agents already use 18.6 cores—about 80% of the node. That is the measured floor for this baseline (single turn, no tool calling). Tool calling, browser, and multiturn will need more and will be characterized in Part 2 of this blog series. 

Figure 5: OpenShell sandbox CPU and memory utilization 

Figure 5 is the OpenShell footprint. The gateway is a fixed ~122 MiB regardless of how many sandboxes are active. As outlined in Table 6, even at 10 concurrent sandboxes, the entire OpenShell stack (gateway, supervisor, and active sandboxes) fits within 120 millicores and 242 MiB. This reinforces the findings above: echancing security with isolation adds minimal overhead to your agentic pipeline. 

Table 6: Node sizing by fleet size

Figure 6 breaks the single-agent CPU profile down by pipeline stage. We aligned 5 ms cgroup samples to each request's OTEL trace. Context assembly is the most expensive stage: 1.34 cores average, 2.53 at P95. During vLLM inference, the gateway is mostly idle (0.52 cores), confirming that the GPU work and the gateway work do not compete for the same resource at the same time.

Figure 6: Per-stage CPU on 1 OpenClaw gateway pod (C=1, 5 ms cgroup samples)

GPU utilization

Figure 7 confirms that under this configuration, the GPU is far from saturated. Even at C=10 with 10 concurrent agents, tensor core activity reaches only 35%, KV cache occupancy sits at 14%, and the vLLM request queue is empty 67% of the time, with an average depth of 1 request (a single burst of queue depth 7 was observed). The model finishes its work and waits: Memory copy utilization (~60%) and HBM bandwidth (~48%) stay flat across all concurrency levels. That is the steady-state decode floor, not a ceiling. The GPU has room for more concurrent work.

Figure 7: GPU utilization, including KV cache, tensor core activity, memory utilization, HBM, and queue depth

The prompt expansion effect

The agentic harness expands every user prompt by up to a factor of 95 times by injecting the system prompt, tool schema definitions, and conversation context. The exact expansion depends on the tool schemas loaded, system prompt length, and sandbox context, as outlined in Table 7.

Table 7: User prompt expansion

With per-token pricing, input costs scale proportionally with prompt expansion. On self-hosted infrastructure, GPU costs are fixed, but agent framework CPU and memory overheads constrain throughput. At C=10, the GPU queue is empty 67% of the time and KV cache occupancy sits at ~10.5%, showing that the model finishes generation faster than the framework can dispatch requests. These single-turn baseline metrics will compound in multiturn scenarios as history accumulates—an analysis we’ll see in detail in Part 2.

OpenShell sandbox performance on OpenShift

The pipeline breakdown in the previous section showed that sandbox provisioning is 1 of the 5 measured stages and it is the 2nd-largest contributor to the agent pipeline. In production, this cost varies dramatically depending on what infrastructure is already running when an agent needs to operate. Understanding sandbox provisioning is essential for capacity planning, as it determines whether secure isolation is invisible to end users or a measurable delay in every interaction. We identify 2 provisioning tiers: 

Cold start occurs when no sandbox pod exists, forcing the system to build the pod, schedule it, pull the image, and register the supervisor from scratch. This happens with the 1st sandbox on a node, unconfigured or exhausted warm pools, pod evictions, node failures, or new configuration requests.

In a warm start, the OpenShell pod is already running but there is no active sandbox. This tier measures the initialization of a new sandbox session inside the existing pod: supervisor authentication, config exchange, and connection establishment. From here, there are 3 possible usage patterns:

• Session scope: Pay warm start once per session, then every tool call reuses the existing sandbox- Ideal for maintaining state (like downloaded files or variables) across a single user's conversation while guaranteeing isolation from other users.
• Turn scope: Pay warm start on every single tool call. Maximum isolation, highest cost. Use this when security is the top priority, such as running untrusted code, and you need to guarantee that data from one action cannot accidentally be seen or used by the next action
• Agent scope: Pay warm start once for the agent's lifetime, shared across all conversations. Cheapest option overall because initialization cost is paid only once. Appropriate for trusted, single-tenant internal tools or stateless workloads where strict isolation between conversations isn't necessary.

Figure 8: OpenShell sandbox lifecycle: The configured scope determines how frequently the warm start cost is incurred

To measure these tiers in isolation, we developed a standalone benchmarking tool that calls the OpenShell gateway's API directly to eliminate variables like prompt processing and model inference, giving us a clean signal of pure sandbox overhead. The results are laid out in Table 8.

Table 8: OpenShell sandbox performance

The key insight: With warm pools enabled, OpenShell delivers sub-300 ms sandbox readiness—just 4.3% of the end-to-end request latency (E2E) measured at C=1. In session-scoped deployments, this cost is paid once per conversation; every subsequent tool execution reuses the existing sandbox at 134 ms per command (2.3% of E2E). Referring back to our guiding question—what does each layer of the agentic stack actually cost?—sandbox isolation is one of the cheapest layers in the pipeline, well behind response processing and token generation in terms of cost. This means teams can adopt full process-level isolation without meaningful performance trade-offs: The security layer that provides per-binary policy, credential hiding, and seccomp enforcement costs less end-to-end time than a single context assembly step.

OpenShell gateway under load

Since every request provisions its own isolated sandbox, a natural question follows: How does provisioning latency and E2E performance change as the number of concurrent agents sharing a single OpenShell gateway grows? As illustrated in Figure 9, we tested 1, 5, and 10 concurrent agents—enough to establish the scaling floor, though higher concurrency levels are planned for follow-up work. There are 2 key points to keep in mind. First, per-sandbox node resource usage (idle and active) and marginal costs will be evaluated in future scaling studies. Second, this benchmark uses uniform sequential workloads; actual production traffic features mixed lifecycles, overlapping starts and stops, and variable request rates that add contention beyond this study's scope.

Figure 9: Sandbox provisioning latency under concurrent load

Each request gets its own isolated sandbox—no sharing—but that 1 gateway pod must handle all

concurrent provisioning requests with finite CPU and internal serialization. This contention represents an area for further investigation and optimization at higher concurrency levels. As Table 9 demonstrates, this contention does not currently present a critical bottleneck.

Table 9: Sandbox initialization latency as a percentage of total end-to-end (E2E) time across concurrency levels

At C=10, the provisioning cost drops to under 5% of total E2E because the baseline grows (more agents competing for model decode throughput) while sandbox init remains flat. The gateway scales well within this concurrency range.

What this means for your deployment

What are the practical lessons you can take away from these results?

Sandbox security is cheap. Enable it by default. In our evaluation—including process isolation, seccomp enforcement, and a dedicated filesystem—sandbox initialization adds just 2.2% to end-to-end latency at C=1 (P50) and under 5% at C=10. The entire sandbox stack (gateway plus 10 concurrent sandboxes) fits within 120 millicores and 242 MiB. In session-scoped deployments, this cost is paid once per conversation; every subsequent tool execution reuses the existing sandbox. Teams should not skip isolation for performance reasons; the security benefits far outweigh a cost that is barely measurable.

To optimize an agentic deployment, benchmark the harness, not just the model. Our measurements show the agentic harness pipeline (context assembly, routing, response processing, stream delivery) dominates latency and becomes the throughput ceiling under concurrency. The relevant metrics are gateway CPU scaling, agent processing share of E2E, and per-request response processing time (OTEL). Tuning these metrics is what moves the needle for agentic workloads—and you won’t necessarily even need to add GPU capacity.

Understanding your pipeline is the 1st step to optimizing it. Every orchestration framework expands user prompts by injecting system instructions, tool schemas, and conversation context. The methodology and tooling we present here give teams the ability to decompose their own agentic deployments and identify where their specific optimization opportunities lie. Follow-on work will layer in multiturn tool calling and code execution so teams can see how this baseline evolves under a complete agentic loop.

Conclusion 

What does each layer of the agent stack actually cost in measurable performance overhead? This is the question we set out to answer, a question that is essential for capacity planning, service-level agreement design, and optimization prioritization within organizations deploying agentic AI workloads at scale. By instrumenting the full request lifecycle—from user prompt through agent gateway, sandbox provisioning, context expansion, model inference, and token streaming back to the client—we decomposed what was previously a black-box E2E latency number into individually measurable pipeline segments. This gives engineering teams the data they need to plan agent configurations according to their specific requirements, and to understand that the performance ceiling may lie in the orchestration layer—not necessarily in the GPU or the model itself.

What comes next 

This baseline establishes the performance floor for the agentic stack. In upcoming posts, we'll layer on multiturn tool calling with sandbox code execution (Part 2) and scale beyond 10 concurrent agents to establish deployment guidelines for multiagent environments (Part 3).

Get started

Ready to profile your own agentic deployment?

• Red Hat OpenShift AI: Deploy and manage AI models and agents at scale
• OpenShell: Secure sandbox provisioning for agent code execution
• OpenClaw: The agentic harness used in this evaluation

For questions or to share your own benchmarking results, reach out to the PSAP Agentic team.