Deploying a Red Hat OpenShift Operator in an environment with internet access is typically straightforward. However, in industries like cyber security or the military sector, where security concerns often prohibit internet access, the process becomes more complex. In a disconnected or air-gapped environment, internet access is usually restricted or unavailable.
In this article, I demonstrate the process of deploying an operator in a disconnected environment. I use the recent Red Hat OpenShift AI operator for this example, because the use of artificial intelligence is becoming crucial to many environments.
Prerequisites
The steps involve saving all required images to a disk, transferring the disk to the disconnected environment, uploading the OpenShift images to a private registry, and then installing the operator
To proceed, you must have the following requirements in place:
- OpenShift is installed in a disconnected environment
- Object storage is available
- A private registry is deployed in the disconnected environment
Mirroring images from public registries to disk
To complete the first task, you use the oc-mirror
plugin, an excellent tool for managing and controlling the volume of data being mirrored.
This section requires an internet connection, because you're fetching images from public registries.
For each operator you want to install in an air-gapped environment, it's essential to carefully review the documentation. This ensures that you mirror all the necessary images and their dependencies.
The OpenShift AI Operator documentation reveals that several other operators need to be installed first. Similarly, the OpenShift Service Mesh has its own set of dependencies. Missing these dependencies won’t necessarily cause the OpenShift AI Operator installation to fail, but it could lead to issues when using it. For example, if you enable the KServe component when deploying the DataScienceCluster instance, you must also have installed Operators for Red Hat OpenShift Serverless and Red Hat OpenShift Service Mesh in order to support KServe (used by the single-model serving platform to serve large models).
The Openshift AI operator (rhods-operator
) has these requirements:
- Pipeline operator
- Serverless operator
- Servicemesh operator
- Jaeger operator
- Kiali operator
Red Hat Openshift AI needs extra images that can be found in the Git repository RHOAI disconnected install helper, which includes the list of extra images required for Openshift AI 2.11.
The oc-mirror plugin
Before using the oc-mirror
plugin, you need to complete some configuration steps. Fortunately, there is comprehensive documentation available to guide you through this process.
Install the oc-mirror plugin
- To download the
oc-mirror
CLI plugin, navigate to the downloads page of the OpenShift Cluster Manager. Under the OpenShift disconnected installation tools section, click Download for OpenShift Client (oc) mirror plugin and save the file Extract the archive:
$ tar xvzf oc-mirror.tar.gz
$ chmod +x oc-mirrorDo not rename the
oc-mirror
file- To install the
oc-mirror
CLI plugin, place the file in your PATH. For example:$ sudo mv oc-mirror /usr/local/bin/
- Verify that the plugin for
oc-mirror
is installed:$ oc mirror help
Configure the oc-mirror credentials
Next, you must configure the oc-mirror credentials. As usual, read the documentation before attempting this.
- Download your
registry.redhat.io
pull secret from Red Hat OpenShift Cluster Manager Convert your pull secret in JSON format:
$ cat ./pull-secret | jq . > <path>/<pull_secret_file_in_json>
Copy the JSON file to the .docker directory:
$ mkdir ~/.docker
$ cp <path>/<pull_secret_file_in_json> ~/.docker/config.json
Validate the oc-mirror configuration
Validate the oc-mirror
configuration:
- Verify that the plugin for oc-mirror is installed:
$ oc mirror help
- Note that there is no dash (-) between
oc
andmirror
. - If the command returns help options, then
oc-mirror
is installed.
- Note that there is no dash (-) between
Log in to
registry.redhat.io
and look for acredentials are valid
message:$ podman login registry.redhat.io
Authenticating with existing credentials for registry.redhat.io
Existing credentials are valid. Already logged in to registry.redhat.io
Create the oc-mirror ISC (ImageSourceConfiguration) configuration file
The oc-mirror
tool uses an ISC configuration file to define what needs to be mirrored. This configuration allows you to specify the catalog source name and the versions of the operators you want to mirror.
To learn how to automate the creation of ISC files, read my article on developers.redhat.com.
Find operator channel and release information
You need to provide the oc-mirror
plugin with the operator's channel and version. To get that information, get a list of the available catalogs:
$ oc mirror list operators --catalogs --version=4.16 Available OpenShift OperatorHub catalogs: OpenShift 4.16: registry.redhat.io/redhat/redhat-operator-index:v4.16 registry.redhat.io/redhat/certified-operator-index:v4.16 registry.redhat.io/redhat/community-operator-index:v4.16 registry.redhat.io/redhat/redhat-marketplace-index:v4.16
Find the available packages within the selected catalog
In this scenario, all operators you're mirroring are from the Red Hat operator catalogs.
$ oc mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.16 NAME DISPLAY NAME DEFAULT CHANNEL 3scale-operator Red Hat Integration - 3scale threescale-2.14 advanced-cluster-management Advanced Cluster Management for Kubernetes release-2.9 amq-broker-rhel8 Red Hat Integration - AMQ Broker for RHEL 8 (Multiarch) 7.11.x amq-online Red Hat Integration - AMQ Online stable amq-streams AMQ Streams stable
Find channels for the selected package
Next, get the channels for the package:
$ oc mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.16 --package=rhods-operator NAME DISPLAY NAME DEFAULT CHANNEL rhods-operator Red Hat OpenShift AI fast PACKAGE CHANNEL HEAD rhods-operator alpha rhods-operator.2.11.0 rhods-operator beta rhods-operator.2.4.0 rhods-operator embedded rhods-operator.2.7.0 rhods-operator eus-2.8 rhods-operator.2.8.4 rhods-operator fast rhods-operator.2.11.0 rhods-operator stable rhods-operator.2.10.0 rhods-operator stable-2.10 rhods-operator.2.10.0 rhods-operator stable-2.8 rhods-operator.2.8.4
Find release information in the selected channel
Look in the output for the versions available in the selected channel.
VERSIONS
…
…
2.8.1
1.32.0
2.10.0
2.11.0
Repeat this process for each operator until you've gathered this information for each operator you want to mirror.
Build the oc-mirror ISC configuration file
If this is the first time you're creating an ISC file, use the oc mirror
command to create a template.
$ oc mirror init > imageset-config.yaml
Edit the new ISC file and update it using the good catalog and the operators release information you gathered in the previous steps. Here's an example of an ISC configuration file for OpenShift AI 2.11:
$ cat redhat-op-v4.16-config_ai.yaml kind: ImageSetConfiguration apiVersion: mirror.openshift.io/v1alpha2 storageConfig: local: path: /path/to/disk-rh-ai/metadata mirror: operators: - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.16 targetCatalog: redhat-catalog-v4.16 packages: - name: "jaeger-product" channels: - name: "stable" minVersion: "1.57.0-7" - name: "kiali-ossm" channels: - name: "stable" minVersion: "1.73.8" - name: "openshift-pipelines-operator-rh" channels: - name: "latest" minVersion: "1.15.1" - name: "rhods-operator" channels: - name: "fast" minVersion: "2.11.0" - name: "serverless-operator" channels: - name: "stable" minVersion: "1.33.1" - name: "servicemeshoperator" channels: - name: "stable" minVersion: "2.5.2-0" additionalImages: - name: quay.io/integreatly/prometheus-blackbox-exporter@sha256:35b9d2c1002201723b7f7a9f54e9406b2ec4b5b0f73d114f47c70e15956103b5 - name: quay.io/modh/caikit-nlp@sha256:0cde6c26e02ec398aea959a1a1bcdc615b86821adb41989e81d03de01124545c - name: quay.io/modh/caikit-tgis-serving@sha256:4e907ce35a3767f5be2f3175a1854e8d4456c43b78cf3df4305bceabcbf0d6e2 … … …
Finally, create a directory for the image's TAR file, and then run the command:
$ mkdir disk-rh-ai $ oc mirror --verbose 3 --config=redhat-op-v4.16-config_ai.yaml file://disk-rh-ai
Once complete, the TAR file is in the disk-rh-ai
directory you created:
$ ls -ltr disk-rh-ai/ drwxr-xr-x. 3 admin admin 17 Aug 1 10:56 oc-mirror-workspace -rw-r--r--. 1 admin admin 176802807808 Aug 1 11:29 mirror_seq1_000000.tar drwxr-x---. 3 admin admin 21 Aug 1 11:29 metadata
Move the TAR file to the disconnected environment
The operational procedures and security requirements can vary significantly from one organization to another. For example, in certain highly restricted environments, you must copy the TAR file to a portable disk, which you take to your disconnected environment's security team for review.
Mirroring the operator from disk to private registry
Once the data has been moved to the disconnected environment, you can begin pushing the images in your private registry. Because you've used the oc-mirror
plugin to push all necessary container images, you must configure the tools accordingly, with the credential information pointing to your private registry instead of the Red Hat public registry.
Here is an example of the oc-mirror
credential configuration for a private registry:
$ cat ~/.docker/config.json { "auths": { "quaylab.redhat.com:8443": { "auth": "aW5pdDpVUjiAYzJiOGcxTVc01djQcGh6PQ==", "email": "ebeaudoi@example.com" } }
Upload the container’s images to your private registry
Create a directory to copy the TAR file, and to serve as the destination for “oc mirror
” output:
$ mkdir disk-rh-ai $ cp mirror_seq1_000000.tar disk-rh-ai/ $ cd disk-rh-ai
Execute the oc mirror
command to upload the container’s images to your private registry
$ oc mirror --verbose 3 \ --from=./mirror_seq1_000000.tar \ docker://quaylab.redhat.com:8443/redhat-v416
Once complete, you get the following directories with similar files:
$ ls -ltr oc-mirror-workspace/results-1722527218/ total 60 drwxr-xr-x. 2 admin admin 6 Aug 1 11:46 charts drwxr-xr-x. 2 admin admin 6 Aug 1 12:47 release-signatures -rw-r--r--. 1 admin admin 50289 Aug 1 12:48 mapping.txt -rwxr-xr-x. 1 admin admin 264 Aug 1 12:48 catalogSource-cs-redhat-catalog-v4-16.yaml -rwxr-xr-x. 1 admin admin 2384 Aug 1 12:48 imageContentSourcePolicy.yaml
Configure the Openshift catalog
Keeping the Openshift default catalogs in a disconnected environment would just trigger unnecessary errors, so disable the default OperatorHub catalog sources:
$ oc patch OperatorHub cluster --type json -p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]'
Create a new catalog referring to your private registry using the YAML files generated by the oc-mirror plugin output.
Log in with cluster-admin privileges
You must be logged in to your OpenShift instance as a user with cluster-admin privileges. To confirm:
$ oc login -u admin -p redhat https://api.quaylab.redhat.com:6443 $ oc whoami admin
Create the ICSP file using Openshift CLI
Now you can create the ICSP file:
$ oc apply -f oc-mirror-workspace/results-1722527218/imageContentSourcePolicy.yaml … imagecontentsourcepolicy.operator.openshift.io/generic-0 created imagecontentsourcepolicy.operator.openshift.io/operator-0 created
Create the Red Hat catalog using the Openshift CLI
Next, create the Red Hat catalog:
$ oc apply -f oc-mirror-workspace/results-1722527218/catalogSource-cs-redhat-catalog-v4-16.yaml … catalogsource.operators.coreos.com/cs-redhat-catalog-v4-16 created
Verify that you can see the catalog in the mirrored operators in Operators > OperatorHub:
![deploying-red-hat-openshift-operators-in-disconnected-environment-img1](/rhdc/managed-files/styles/wysiwyg_full_width/private/deploying-red-hat-openshift-operators-in-disconnected-environment-img1.png.webp?itok=-3otGZeQ)
Configure the OpenShift internal registry
An OpenShift internal registry is a dependency for OpenShift Data Science and OpenShift AI in releases prior to 2.10. For a test cluster, you can use emptyDir. Do not use emptyDir for a production cluster.
$ oc patch configs.imageregistry.operator.openshift.io cluster \ --type merge --patch '{"spec":{"storage":{"emptyDir":{}}}}' \ config.imageregistry.operator.openshift.io/cluster $ oc patch configs.imageregistry.operator.openshift.io cluster \ --type merge --patch '{"spec":{"managementState":"Managed"}}' \ config.imageregistry.operator.openshift.io/cluster
Install the AI operators
I suggest using the OpenShift UI to install the operators. Once you have the OpenShift cluster UI open, follow these steps:
- Click on Operators > OperatorHub
- Select the operator to install
- Keep the default settings and click Install
Repeat this process for each operator:
- Kiali operator
- Jaeger operator
- Pipeline operator
- Serverless operator
- Servicemesh operator
- Openshift AI
![deploying-red-hat-openshift-operators-in-disconnected-environment-img2](/rhdc/managed-files/styles/wysiwyg_full_width/private/deploying-red-hat-openshift-operators-in-disconnected-environment-img2.png.webp?itok=K-Yd_G4i)
Create the RHOAI instance
After the OpenShift AI Operator is successfully installed, you must create a DataScienceCluster instance.
- In the web console, click Operators > Installed Operators, and then select Red Hat OpenShift AI Operator
- Click the Data Science Cluster tab
Click Create DataScienceCluster
In the UI, select YAML view to see the installed components. You have two options:
- Managed: The Operator actively manages the component, installs it, and tries to keep it active. The Operator upgrades the component only if it is safe to do so
- Removed: The Operator actively manages the component but does not install it. If the component is already installed, the Operator tries to remove it
- Click the Create button to continue
Access the Red Hat AI dashboard
At this point,the Openshift AI Data Science cluster instance has been successfully deployed. The OpenShift AI dashboard has its own UI.
To access the OpenShift AI dashboard, click the Grid icon application launcher at the top of the screen
- Right-click on Red Hat OpenShift AI and copy the URL of your OpenShift AI instance
Paste the URL into a web browser to navigate to the dashboard
Conclusion
Working in a disconnected environment presents significant challenges and adds complexity to the application development process. This mode of operation requires enhanced control over how your applications interact with the internet. Using the right tools is crucial for managing data efficiently and minimizing your workload. In this blog, I've demonstrated how to use the oc-mirror
plugin, which I believe is the best approach for this scenario. It provides enhanced control over the volume of data you need to handle, streamlines the process, and improves efficiency.
If you're interested in knowing more about Red Hat AI, click here. If you want to know more about Openshift in disconnected environment, click here.
저자 소개
채널별 검색
오토메이션
기술, 팀, 인프라를 위한 IT 자동화 최신 동향
인공지능
고객이 어디서나 AI 워크로드를 실행할 수 있도록 지원하는 플랫폼 업데이트
오픈 하이브리드 클라우드
하이브리드 클라우드로 더욱 유연한 미래를 구축하는 방법을 알아보세요
보안
환경과 기술 전반에 걸쳐 리스크를 감소하는 방법에 대한 최신 정보
엣지 컴퓨팅
엣지에서의 운영을 단순화하는 플랫폼 업데이트
인프라
세계적으로 인정받은 기업용 Linux 플랫폼에 대한 최신 정보
애플리케이션
복잡한 애플리케이션에 대한 솔루션 더 보기
오리지널 쇼
엔터프라이즈 기술 분야의 제작자와 리더가 전하는 흥미로운 스토리
제품
- Red Hat Enterprise Linux
- Red Hat OpenShift Enterprise
- Red Hat Ansible Automation Platform
- 클라우드 서비스
- 모든 제품 보기
툴
체험, 구매 & 영업
커뮤니케이션
Red Hat 소개
Red Hat은 Linux, 클라우드, 컨테이너, 쿠버네티스 등을 포함한 글로벌 엔터프라이즈 오픈소스 솔루션 공급업체입니다. Red Hat은 코어 데이터센터에서 네트워크 엣지에 이르기까지 다양한 플랫폼과 환경에서 기업의 업무 편의성을 높여 주는 강화된 기능의 솔루션을 제공합니다.