Deploying a Red Hat OpenShift Operator in an environment with internet access is typically straightforward. However, in industries like cyber security or the military sector, where security concerns often prohibit internet access, the process becomes more complex. In a disconnected or air-gapped environment, internet access is usually restricted or unavailable.
In this article, I demonstrate the process of deploying an operator in a disconnected environment. I use the recent Red Hat OpenShift AI operator for this example, because the use of artificial intelligence is becoming crucial to many environments.
Prerequisites
The steps involve saving all required images to a disk, transferring the disk to the disconnected environment, uploading the OpenShift images to a private registry, and then installing the operator
To proceed, you must have the following requirements in place:
- OpenShift is installed in a disconnected environment
- Object storage is available
- A private registry is deployed in the disconnected environment
Mirroring images from public registries to disk
To complete the first task, you use the oc-mirror
plugin, an excellent tool for managing and controlling the volume of data being mirrored.
This section requires an internet connection, because you're fetching images from public registries.
For each operator you want to install in an air-gapped environment, it's essential to carefully review the documentation. This ensures that you mirror all the necessary images and their dependencies.
The OpenShift AI Operator documentation reveals that several other operators need to be installed first. Similarly, the OpenShift Service Mesh has its own set of dependencies. Missing these dependencies won’t necessarily cause the OpenShift AI Operator installation to fail, but it could lead to issues when using it. For example, if you enable the KServe component when deploying the DataScienceCluster instance, you must also have installed Operators for Red Hat OpenShift Serverless and Red Hat OpenShift Service Mesh in order to support KServe (used by the single-model serving platform to serve large models).
The Openshift AI operator (rhods-operator
) has these requirements:
- Pipeline operator
- Serverless operator
- Servicemesh operator
- Jaeger operator
- Kiali operator
Red Hat Openshift AI needs extra images that can be found in the Git repository RHOAI disconnected install helper, which includes the list of extra images required for Openshift AI 2.11.
The oc-mirror plugin
Before using the oc-mirror
plugin, you need to complete some configuration steps. Fortunately, there is comprehensive documentation available to guide you through this process.
Install the oc-mirror plugin
- To download the
oc-mirror
CLI plugin, navigate to the downloads page of the OpenShift Cluster Manager. Under the OpenShift disconnected installation tools section, click Download for OpenShift Client (oc) mirror plugin and save the file Extract the archive:
$ tar xvzf oc-mirror.tar.gz
$ chmod +x oc-mirrorDo not rename the
oc-mirror
file- To install the
oc-mirror
CLI plugin, place the file in your PATH. For example:$ sudo mv oc-mirror /usr/local/bin/
- Verify that the plugin for
oc-mirror
is installed:$ oc mirror help
Configure the oc-mirror credentials
Next, you must configure the oc-mirror credentials. As usual, read the documentation before attempting this.
- Download your
registry.redhat.io
pull secret from Red Hat OpenShift Cluster Manager Convert your pull secret in JSON format:
$ cat ./pull-secret | jq . > <path>/<pull_secret_file_in_json>
Copy the JSON file to the .docker directory:
$ mkdir ~/.docker
$ cp <path>/<pull_secret_file_in_json> ~/.docker/config.json
Validate the oc-mirror configuration
Validate the oc-mirror
configuration:
- Verify that the plugin for oc-mirror is installed:
$ oc mirror help
- Note that there is no dash (-) between
oc
andmirror
. - If the command returns help options, then
oc-mirror
is installed.
- Note that there is no dash (-) between
Log in to
registry.redhat.io
and look for acredentials are valid
message:$ podman login registry.redhat.io
Authenticating with existing credentials for registry.redhat.io
Existing credentials are valid. Already logged in to registry.redhat.io
Create the oc-mirror ISC (ImageSourceConfiguration) configuration file
The oc-mirror
tool uses an ISC configuration file to define what needs to be mirrored. This configuration allows you to specify the catalog source name and the versions of the operators you want to mirror.
To learn how to automate the creation of ISC files, read my article on developers.redhat.com.
Find operator channel and release information
You need to provide the oc-mirror
plugin with the operator's channel and version. To get that information, get a list of the available catalogs:
$ oc mirror list operators --catalogs --version=4.16 Available OpenShift OperatorHub catalogs: OpenShift 4.16: registry.redhat.io/redhat/redhat-operator-index:v4.16 registry.redhat.io/redhat/certified-operator-index:v4.16 registry.redhat.io/redhat/community-operator-index:v4.16 registry.redhat.io/redhat/redhat-marketplace-index:v4.16
Find the available packages within the selected catalog
In this scenario, all operators you're mirroring are from the Red Hat operator catalogs.
$ oc mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.16 NAME DISPLAY NAME DEFAULT CHANNEL 3scale-operator Red Hat Integration - 3scale threescale-2.14 advanced-cluster-management Advanced Cluster Management for Kubernetes release-2.9 amq-broker-rhel8 Red Hat Integration - AMQ Broker for RHEL 8 (Multiarch) 7.11.x amq-online Red Hat Integration - AMQ Online stable amq-streams AMQ Streams stable
Find channels for the selected package
Next, get the channels for the package:
$ oc mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.16 --package=rhods-operator NAME DISPLAY NAME DEFAULT CHANNEL rhods-operator Red Hat OpenShift AI fast PACKAGE CHANNEL HEAD rhods-operator alpha rhods-operator.2.11.0 rhods-operator beta rhods-operator.2.4.0 rhods-operator embedded rhods-operator.2.7.0 rhods-operator eus-2.8 rhods-operator.2.8.4 rhods-operator fast rhods-operator.2.11.0 rhods-operator stable rhods-operator.2.10.0 rhods-operator stable-2.10 rhods-operator.2.10.0 rhods-operator stable-2.8 rhods-operator.2.8.4
Find release information in the selected channel
Look in the output for the versions available in the selected channel.
VERSIONS
…
…
2.8.1
1.32.0
2.10.0
2.11.0
Repeat this process for each operator until you've gathered this information for each operator you want to mirror.
Build the oc-mirror ISC configuration file
If this is the first time you're creating an ISC file, use the oc mirror
command to create a template.
$ oc mirror init > imageset-config.yaml
Edit the new ISC file and update it using the good catalog and the operators release information you gathered in the previous steps. Here's an example of an ISC configuration file for OpenShift AI 2.11:
$ cat redhat-op-v4.16-config_ai.yaml kind: ImageSetConfiguration apiVersion: mirror.openshift.io/v1alpha2 storageConfig: local: path: /path/to/disk-rh-ai/metadata mirror: operators: - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.16 targetCatalog: redhat-catalog-v4.16 packages: - name: "jaeger-product" channels: - name: "stable" minVersion: "1.57.0-7" - name: "kiali-ossm" channels: - name: "stable" minVersion: "1.73.8" - name: "openshift-pipelines-operator-rh" channels: - name: "latest" minVersion: "1.15.1" - name: "rhods-operator" channels: - name: "fast" minVersion: "2.11.0" - name: "serverless-operator" channels: - name: "stable" minVersion: "1.33.1" - name: "servicemeshoperator" channels: - name: "stable" minVersion: "2.5.2-0" additionalImages: - name: quay.io/integreatly/prometheus-blackbox-exporter@sha256:35b9d2c1002201723b7f7a9f54e9406b2ec4b5b0f73d114f47c70e15956103b5 - name: quay.io/modh/caikit-nlp@sha256:0cde6c26e02ec398aea959a1a1bcdc615b86821adb41989e81d03de01124545c - name: quay.io/modh/caikit-tgis-serving@sha256:4e907ce35a3767f5be2f3175a1854e8d4456c43b78cf3df4305bceabcbf0d6e2 … … …
Finally, create a directory for the image's TAR file, and then run the command:
$ mkdir disk-rh-ai $ oc mirror --verbose 3 --config=redhat-op-v4.16-config_ai.yaml file://disk-rh-ai
Once complete, the TAR file is in the disk-rh-ai
directory you created:
$ ls -ltr disk-rh-ai/ drwxr-xr-x. 3 admin admin 17 Aug 1 10:56 oc-mirror-workspace -rw-r--r--. 1 admin admin 176802807808 Aug 1 11:29 mirror_seq1_000000.tar drwxr-x---. 3 admin admin 21 Aug 1 11:29 metadata
Move the TAR file to the disconnected environment
The operational procedures and security requirements can vary significantly from one organization to another. For example, in certain highly restricted environments, you must copy the TAR file to a portable disk, which you take to your disconnected environment's security team for review.
Mirroring the operator from disk to private registry
Once the data has been moved to the disconnected environment, you can begin pushing the images in your private registry. Because you've used the oc-mirror
plugin to push all necessary container images, you must configure the tools accordingly, with the credential information pointing to your private registry instead of the Red Hat public registry.
Here is an example of the oc-mirror
credential configuration for a private registry:
$ cat ~/.docker/config.json { "auths": { "quaylab.redhat.com:8443": { "auth": "aW5pdDpVUjiAYzJiOGcxTVc01djQcGh6PQ==", "email": "ebeaudoi@example.com" } }
Upload the container’s images to your private registry
Create a directory to copy the TAR file, and to serve as the destination for “oc mirror
” output:
$ mkdir disk-rh-ai $ cp mirror_seq1_000000.tar disk-rh-ai/ $ cd disk-rh-ai
Execute the oc mirror
command to upload the container’s images to your private registry
$ oc mirror --verbose 3 \ --from=./mirror_seq1_000000.tar \ docker://quaylab.redhat.com:8443/redhat-v416
Once complete, you get the following directories with similar files:
$ ls -ltr oc-mirror-workspace/results-1722527218/ total 60 drwxr-xr-x. 2 admin admin 6 Aug 1 11:46 charts drwxr-xr-x. 2 admin admin 6 Aug 1 12:47 release-signatures -rw-r--r--. 1 admin admin 50289 Aug 1 12:48 mapping.txt -rwxr-xr-x. 1 admin admin 264 Aug 1 12:48 catalogSource-cs-redhat-catalog-v4-16.yaml -rwxr-xr-x. 1 admin admin 2384 Aug 1 12:48 imageContentSourcePolicy.yaml
Configure the Openshift catalog
Keeping the Openshift default catalogs in a disconnected environment would just trigger unnecessary errors, so disable the default OperatorHub catalog sources:
$ oc patch OperatorHub cluster --type json -p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]'
Create a new catalog referring to your private registry using the YAML files generated by the oc-mirror plugin output.
Log in with cluster-admin privileges
You must be logged in to your OpenShift instance as a user with cluster-admin privileges. To confirm:
$ oc login -u admin -p redhat https://api.quaylab.redhat.com:6443 $ oc whoami admin
Create the ICSP file using Openshift CLI
Now you can create the ICSP file:
$ oc apply -f oc-mirror-workspace/results-1722527218/imageContentSourcePolicy.yaml … imagecontentsourcepolicy.operator.openshift.io/generic-0 created imagecontentsourcepolicy.operator.openshift.io/operator-0 created
Create the Red Hat catalog using the Openshift CLI
Next, create the Red Hat catalog:
$ oc apply -f oc-mirror-workspace/results-1722527218/catalogSource-cs-redhat-catalog-v4-16.yaml … catalogsource.operators.coreos.com/cs-redhat-catalog-v4-16 created
Verify that you can see the catalog in the mirrored operators in Operators > OperatorHub:
![deploying-red-hat-openshift-operators-in-disconnected-environment-img1](/rhdc/managed-files/styles/wysiwyg_full_width/private/deploying-red-hat-openshift-operators-in-disconnected-environment-img1.png.webp?itok=-3otGZeQ)
Configure the OpenShift internal registry
An OpenShift internal registry is a dependency for OpenShift Data Science and OpenShift AI in releases prior to 2.10. For a test cluster, you can use emptyDir. Do not use emptyDir for a production cluster.
$ oc patch configs.imageregistry.operator.openshift.io cluster \ --type merge --patch '{"spec":{"storage":{"emptyDir":{}}}}' \ config.imageregistry.operator.openshift.io/cluster $ oc patch configs.imageregistry.operator.openshift.io cluster \ --type merge --patch '{"spec":{"managementState":"Managed"}}' \ config.imageregistry.operator.openshift.io/cluster
Install the AI operators
I suggest using the OpenShift UI to install the operators. Once you have the OpenShift cluster UI open, follow these steps:
- Click on Operators > OperatorHub
- Select the operator to install
- Keep the default settings and click Install
Repeat this process for each operator:
- Kiali operator
- Jaeger operator
- Pipeline operator
- Serverless operator
- Servicemesh operator
- Openshift AI
![deploying-red-hat-openshift-operators-in-disconnected-environment-img2](/rhdc/managed-files/styles/wysiwyg_full_width/private/deploying-red-hat-openshift-operators-in-disconnected-environment-img2.png.webp?itok=K-Yd_G4i)
Create the RHOAI instance
After the OpenShift AI Operator is successfully installed, you must create a DataScienceCluster instance.
- In the web console, click Operators > Installed Operators, and then select Red Hat OpenShift AI Operator
- Click the Data Science Cluster tab
Click Create DataScienceCluster
In the UI, select YAML view to see the installed components. You have two options:
- Managed: The Operator actively manages the component, installs it, and tries to keep it active. The Operator upgrades the component only if it is safe to do so
- Removed: The Operator actively manages the component but does not install it. If the component is already installed, the Operator tries to remove it
- Click the Create button to continue
Access the Red Hat AI dashboard
At this point,the Openshift AI Data Science cluster instance has been successfully deployed. The OpenShift AI dashboard has its own UI.
To access the OpenShift AI dashboard, click the Grid icon application launcher at the top of the screen
- Right-click on Red Hat OpenShift AI and copy the URL of your OpenShift AI instance
Paste the URL into a web browser to navigate to the dashboard
Conclusion
Working in a disconnected environment presents significant challenges and adds complexity to the application development process. This mode of operation requires enhanced control over how your applications interact with the internet. Using the right tools is crucial for managing data efficiently and minimizing your workload. In this blog, I've demonstrated how to use the oc-mirror
plugin, which I believe is the best approach for this scenario. It provides enhanced control over the volume of data you need to handle, streamlines the process, and improves efficiency.
If you're interested in knowing more about Red Hat AI, click here. If you want to know more about Openshift in disconnected environment, click here.
Sobre el autor
Navegar por canal
Automatización
Las últimas novedades en la automatización de la TI para los equipos, la tecnología y los entornos
Inteligencia artificial
Descubra las actualizaciones en las plataformas que permiten a los clientes ejecutar cargas de trabajo de inteligecia artificial en cualquier lugar
Nube híbrida abierta
Vea como construimos un futuro flexible con la nube híbrida
Seguridad
Vea las últimas novedades sobre cómo reducimos los riesgos en entornos y tecnologías
Edge computing
Conozca las actualizaciones en las plataformas que simplifican las operaciones en el edge
Infraestructura
Vea las últimas novedades sobre la plataforma Linux empresarial líder en el mundo
Aplicaciones
Conozca nuestras soluciones para abordar los desafíos más complejos de las aplicaciones
Programas originales
Vea historias divertidas de creadores y líderes en tecnología empresarial
Productos
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Servicios de nube
- Ver todos los productos
Herramientas
- Training y Certificación
- Mi cuenta
- Soporte al cliente
- Recursos para desarrolladores
- Busque un partner
- Red Hat Ecosystem Catalog
- Calculador de valor Red Hat
- Documentación
Realice pruebas, compras y ventas
Comunicarse
- Comuníquese con la oficina de ventas
- Comuníquese con el servicio al cliente
- Comuníquese con Red Hat Training
- Redes sociales
Acerca de Red Hat
Somos el proveedor líder a nivel mundial de soluciones empresariales de código abierto, incluyendo Linux, cloud, contenedores y Kubernetes. Ofrecemos soluciones reforzadas, las cuales permiten que las empresas trabajen en distintas plataformas y entornos con facilidad, desde el centro de datos principal hasta el extremo de la red.
Seleccionar idioma
Red Hat legal and privacy links
- Acerca de Red Hat
- Oportunidades de empleo
- Eventos
- Sedes
- Póngase en contacto con Red Hat
- Blog de Red Hat
- Diversidad, igualdad e inclusión
- Cool Stuff Store
- Red Hat Summit