Build a compliance assistant with AutoRAG and Red Hat OpenShift AI

If you run AI platforms for a bank, an insurer, or anyone else living under a stack of regulations, you already know the use case everyone asks for first: let our compliance and risk teams ask questions against our own policy documents and get an answer they can trust, with a citation they can check.

Retrieval-augmented generation (RAG) is the right tool for that job. When a question is asked, instead of answering from memory, the model looks the answer up in your documents and includes a citation.

The part slowing teams down isn't the pattern, it's the configuration. A RAG pipeline includes a number of variables: chunk size, overlap, how many chunks to retrieve, which embedding model to use, and how the search itself works. There's no universal best setting. The right combination depends on your documents and your questions, so it can't ship as a default. It has to be discovered, per corpus, and most teams discover it by guessing from a tutorial, eyeballing a few answers, and shipping. In a regulated environment, that's not a comfortable place to be.

AutoRAG, an open source tool for finding the optimal RAG pipeline for your data, is a technology preview feature in Red Hat OpenShift AI that replaces guesswork with measurement. You point it at your documents, give it a set of questions with known answers, and it runs an optimization: it sweeps configurations, scores each, and ranks them on a leaderboard. The winning pattern is immediately usable, with an endpoint your applications can call. We built a demo to show the whole path end-to-end, from a folder of policy documents to a compliance assistant answering questions with citations.

Figure 1: Watch the full AutoRAG demo

The scenario: A compliance assistant for a European bank

We built the demo around "Fed Aura Capital Bank," a fictitious European bank with a very realistic problem: its people spend their days answering questions, the answers for which live across overlapping regulatory frameworks.

The knowledge base is a compendium of 6 fictional but realistic internal policies, including around 10,000 words covering:

• A Digital Operational Resilience Act (DORA) implementation policy (European Union (EU) 2022/2554, Threat Intelligence-based Ethical Red Teaming in the European Union (TIBER-EU))
• Markets in Financial Instruments Directive II (MiFID II) product governance, including Markets in Crypto-Assets (MiCA) and crypto custody
• A Corporate Sustainability Reporting Directive (CSRD) Environmental, Social, and Governance (ESG) data collection playbook (EU Taxonomy, Sustainable Finance Disclosure Regulation (SFDR), Partnership for Carbon Accounting Financials (PCAF))
• Third-party Information and Communication Technology (ICT) risk management (DORA Chapter V, European Banking Authority (EBA) guidelines)
• A risk appetite statement (Capital Requirements Directive V (CRD V), Basel IV)
• An Anti-Money Laundering (AML)/Know Your Customer (KYC) policy (Sixth Anti-Money Laundering Directive (AMLD6), Anti-Money Laundering Regulation (AMLR), the Travel Rule)

The questions come from the people who use documents like these. A compliance officer asks about DORA obligations for third-party ICT contracts. A relationship manager in Frankfurt asks whether the bank can offer crypto custody to retail clients. An AML investigator asks about Travel Rule requirements for transfers above EUR 1,000. Each needs a specific answer grounded in a specific policy section, not a plausible paragraph. All documents, figures, and regulatory references in the demo are fictional, but the shape of the problem is common to every regulated institution.

Run an AutoRAG optimization in gen AI studio

Here's what's used in the project. In the OpenShift AI console, under gen AI studio, you create an AutoRAG optimization run. You pick a generation model (the demo uses Llama 3.1 8B Instruct served on vLLM) and an embedding model, then connect your data source, in this case an object storage bucket holding the policy compendium.

Then comes the step that makes this AutoRAG rather than just RAG. You upload a labeled evaluation dataset, a set of query and expected-response pairs drawn from your own documents. The demo ships with 25 of them. You choose the metric you want to optimize for, such as faithfulness, which measures whether the answer sticks to what the retrieved documents say. For a compliance use case, that's exactly the property you care about most.

From there, AutoRAG does the work you'd otherwise do by hand over days of trial and error. Behind the scenes, it explores different chunk sizes, embedding models, and retrieval settings, runs each configuration against your evaluation dataset, and scores the results using a large language model (LLM) to judge based on your chosen metric. The candidate patterns land on a ranked leaderboard, and you can open any of them to inspect exactly what it does: the chunking method, the embedding model, the retrieval strategy, the vector store it's bound to, and its scores.

This is the same principle we've written about before when hand-rolling this loop on a laptop: the best RAG configuration is the output of an evaluation, not something you have before you run it. The difference is that here, the platform runs the loop for you, with a real LLM judge and a wider search space, and the winning pattern isn't a configuration file you still have to implement. It's already backed by an indexed vector store and callable through an endpoint. You can validate it right in gen AI studio, chatting with the pattern against your own documents before any application code exists.

Figure 2: AutoRAG scoring configurations against the evaluation dataset

Connect the optimized pattern to your application

Once you've picked your pattern, the console gives you code snippets in Python, Go, Node.js, and curl with everything an application needs: the endpoint URL, an API key, and the vector store ID for your indexed documents.

The demo application is a small Flask app using the standard OpenAI Software Development Kit (SDK), calling the Responses API against the optimized pattern's endpoint. Retrieval runs through the file_search tool against the vector store, so the application code doesn't reimplement any RAG logic at all. It sends the question, the pattern retrieves the relevant policy chunks with the optimized configuration, the model generates a grounded answer, and the app renders the answer alongside the retrieved sources.

Because the endpoint is OpenAI-compatible, anything already written against the API shape can point at it. There's no proprietary client to adopt and no rewrite when you move an application onto the platform.

Figure 3: The Fed Aura Compliance Navigator answering a DORA question with cited policy sections

Why automated RAG optimization matters

The demo is a compliance assistant, but the workflow underneath it isn't specific to compliance. It's the same path for any team putting enterprise documents behind an AI application: connect the data, define what a good answer looks like, and let the platform find the configuration that gets there.

That changes a few things depending on where you sit.

If you're responsible for AI engineering across teams, retrieval tuning stops being a per-project research effort and becomes a standard workflow. Every use case goes through the same optimization run, produces the same leaderboard, and hands off the same way, so the second and tenth RAG application on your platform cost far less than the first.

If you're building the application, you don't need to become a retrieval specialist to ship something reliable. The optimized pattern arrives as an OpenAI-compatible endpoint, and the demo app shows how little code sits on top of it: a standard SDK call to the Responses API, with retrieval handled by the pattern through the file_search tool.

And if part of your job is answering for how these systems behave, the evaluation isn't a side benefit—it's the record. The dataset, the metric, the scored configurations, and the selected pattern are all artifacts you can point to when someone asks how you know the system retrieves the right document. When the source documents change, you rerun the evaluation and get a new answer, backed the same way.

Get started

AutoRAG is available as a  technical preview in OpenShift AI 3.5, so it's the right time to try it against your own documents and tell us what you find. The whole demo—including the policy compendium, the 25-question evaluation dataset, the setup scripts, and the application code—is available in the fed-aura-autorag-demo repository. Upload the compendium to your own AutoRAG pipeline, run the setup script, and you'll have the same assistant answering questions in a few minutes.

Resources

• Demo GitHub: https://github.com/suhasvkashyap/fed-aura-autorag-demo