Recently one of our Customers asked how to move from a standalone RHACM setup to an Active/Passive Hub one given that the Hub Backup and Restore feature went GA on Red Hat Advanced Cluster Management 2.5.
The Customer Scenario
The Customer had a 3 data-centers setup, each one with its own RHACM Cluster Hub responsible of managing the local OCP clusters. They planned their setup with disaster recovery in mind so there were no object name collisions. The initial DR plan was to manually import OpenShift clusters of the failed Cluster Hub into one of the other two.
All the policies were synchronized on all the Cluster Hubs via GitOps, while this approach permitted sort of hot standby capability - importing clusters would've been enough to start applying policies on them - it caused a lot of policies with unknown state since the clusters where they were supposed to be applied weren't present.
Another problem was the loss of cluster creation data in case of Cluster Hub failure: our Customer leveraged RHACM to create OpenShift clusters on VMware infrastructure running on their 3 data-centers, this data would not be automatically imported into another Cluster Hub.
Given all of that, Customer was really keen to adopt the new Business Continuity model offered by RHACM 2.5 Backup and Restore feature. We'll not discuss the setup of the feature here - you can find a great explanation here - we'll focus instead on the procedure adopted to move from 3 standalone RHACM Cluster Hubs to one Active Cluster Hub and two Passive Cluster Hubs.
Moving Standalone Cluster Hubs to Active/Passive Procedure
RHACM Backup and Restore feature was leveraged by Red Hat Consulting to move the managed clusters from the soon-to-be Passive Hubs into the designated Primary Hub by following this procedure:
-
Hub Backup was configured on all the RHACM cluster hubs, each Hub initially pointed to a dedicated prefix on the object storage bucket to have a valid backup for each of the hubs separately.
Hub1 example
DataProtectionApplication
:apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
name: dpa-hub1
spec:
configuration:
velero:
defaultPlugins:
- openshift
- aws
restic:
enable: false
backupLocations:
- name: default
velero:
provider: aws
default: true
objectStorage:
bucket: my-bucket
prefix: hub1 # <<<< Set a dedicated prefix for each of the hubs
config:
region: us-east-1
profile: "default"
credential:
name: cloud-credentials
key: cloud
snapshotLocations:
- name: default
velero:
provider: aws
config:
region: us-west-2
profile: "default" -
Once all the Hub backups are completed, delete the
BackupSchedule
CR to stop the Hub backups. -
Delete the
DataProtectionApplication
from the Primary Hub and recreate it using the prefix of the first Cluster Hub you wish to make Passive.Hub1 example
DataProtectionApplication
pointing to Hub2 prefix:apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
name: dpa-hub1
spec:
configuration:
velero:
defaultPlugins:
- openshift
- aws
restic:
enable: false
backupLocations:
- name: default
velero:
provider: aws
default: true
objectStorage:
bucket: my-bucket
prefix: hub2 # <<<< points to hub2 backup location
config:
region: us-east-1
profile: "default"
credential:
name: cloud-credentials
key: cloud
snapshotLocations:
- name: default
velero:
provider: aws
config:
region: us-west-2
profile: "default" -
Verify that the velero pod is running and check if the
BackupStorageLocation
is available:oc -n open-cluster-management-backup get backupstoragelocation
NAME PHASE LAST VALIDATED AGE DEFAULT
dpa-hub1-1 Available 58s 2d trueYou can also verify in the velero pod logs that the backups are discovered and imported.
-
Detach all the managed clusters from the Cluster Hub that you are making Passive, wait for the process to complete.
-
On the Primary Hub, create a one-shot
Restore
Custom Resource to import the managed clusters previously attached to the hub you are making Passive:apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Restore
metadata:
name: import-hub2-clusters
namespace: open-cluster-management-backup
spec:
cleanupBeforeRestore: None # IMPORTANT: prevent the restore to delete anything from the Primary Hub during the restore
veleroManagedClustersBackupName: latest
veleroCredentialsBackupName: latest
veleroResourcesBackupName: latestATTENTION: it is really important to set the
cleanupBeforeRestore
value toNone
to prevent accidental deletion of restored objects from the Primary Hub.Check the
Restore
status by runningoc -n open-cluster-management-backup describe restore import-hub2-clusters
This
Restore
can be safely deleted once completed with the commandoc -n open-cluster-management-backup delete restore import-hub2-clusters
-
Repeat the steps from 3 to 6 to move other Cluster Hubs to a Passive role.
-
When you are done moving the managed clusters from the Passive Hubs to the Primary Hub, you can complete the configuration of the Active/Passive feature.
Set a prefix on your
DataProtectionApplication
Custom Resource that will be used for both Active and Passive Hubs:apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
name: dpa-hub1
spec:
configuration:
velero:
defaultPlugins:
- openshift
- aws
restic:
enable: false
backupLocations:
- name: default
velero:
provider: aws
default: true
objectStorage:
bucket: my-bucket
prefix: active-passive # <<<< final location of the backups, passive hubs will restore from here
config:
region: us-east-1
profile: "default"
credential:
name: cloud-credentials
key: cloud
snapshotLocations:
- name: default
velero:
provider: aws
config:
region: us-west-2
profile: "default"Create the same
DataProtectionApplication
Custom Resource on all the Cluster Hubs. -
On the Primary Hub create the
BackupSchedule
Custom Resource to start taking backups to the new location. -
On the Passive Hubs create a one-shot
Restore
with thecleanupBeforeRestore
set toCleanupAll
to prepare the former standalone Hub to receive restores from the Primary Hub without any left-over:apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Restore
metadata:
name: restore-with-cleanupall
namespace: open-cluster-management-backup
spec:
cleanupBeforeRestore: CleanupAll # This will cleanup ALL the resources from the Passive hub
veleroManagedClustersBackupName: skip
veleroCredentialsBackupName: latest
veleroResourcesBackupName: latestOnly the resources that would be backed up will be deleted by this
Restore
, the task will honor backup exclusion label.This
Restore
, once completed, can be safely deleted with the commandoc -n open-cluster-management-backup delete restore restore-with-cleanup
-
On the Passive Hubs create the
Restore
Custom Resource to periodically restore passive data from the Primary Hub: cluster_v1beta1_restore_passive_sync.yaml
Common pitfalls
-
Pay attention to the name of the objects, if you find name collisions deal with them prior to switch to Active/Passive.
-
Policies applied to the Cluster Hub itself are backed up automatically, if those are different across your Cluster Hubs, exclude them using the label
velero.io/exclude-from-backup: "true"
Conclusions
Applying the procedure described earlier allowed the Customer to move to the new Business Continuity feature offered by Red Hat Advanced Cluster Management 2.5.
저자 소개
채널별 검색
오토메이션
기술, 팀, 인프라를 위한 IT 자동화 최신 동향
인공지능
고객이 어디서나 AI 워크로드를 실행할 수 있도록 지원하는 플랫폼 업데이트
오픈 하이브리드 클라우드
하이브리드 클라우드로 더욱 유연한 미래를 구축하는 방법을 알아보세요
보안
환경과 기술 전반에 걸쳐 리스크를 감소하는 방법에 대한 최신 정보
엣지 컴퓨팅
엣지에서의 운영을 단순화하는 플랫폼 업데이트
인프라
세계적으로 인정받은 기업용 Linux 플랫폼에 대한 최신 정보
애플리케이션
복잡한 애플리케이션에 대한 솔루션 더 보기
오리지널 쇼
엔터프라이즈 기술 분야의 제작자와 리더가 전하는 흥미로운 스토리
제품
- Red Hat Enterprise Linux
- Red Hat OpenShift Enterprise
- Red Hat Ansible Automation Platform
- 클라우드 서비스
- 모든 제품 보기
툴
체험, 구매 & 영업
커뮤니케이션
Red Hat 소개
Red Hat은 Linux, 클라우드, 컨테이너, 쿠버네티스 등을 포함한 글로벌 엔터프라이즈 오픈소스 솔루션 공급업체입니다. Red Hat은 코어 데이터센터에서 네트워크 엣지에 이르기까지 다양한 플랫폼과 환경에서 기업의 업무 편의성을 높여 주는 강화된 기능의 솔루션을 제공합니다.