Recently one of our Customers asked how to move from a standalone RHACM setup to an Active/Passive Hub one given that the Hub Backup and Restore feature went GA on Red Hat Advanced Cluster Management 2.5.
The Customer Scenario
The Customer had a 3 data-centers setup, each one with its own RHACM Cluster Hub responsible of managing the local OCP clusters. They planned their setup with disaster recovery in mind so there were no object name collisions. The initial DR plan was to manually import OpenShift clusters of the failed Cluster Hub into one of the other two.
All the policies were synchronized on all the Cluster Hubs via GitOps, while this approach permitted sort of hot standby capability - importing clusters would've been enough to start applying policies on them - it caused a lot of policies with unknown state since the clusters where they were supposed to be applied weren't present.
Another problem was the loss of cluster creation data in case of Cluster Hub failure: our Customer leveraged RHACM to create OpenShift clusters on VMware infrastructure running on their 3 data-centers, this data would not be automatically imported into another Cluster Hub.
Given all of that, Customer was really keen to adopt the new Business Continuity model offered by RHACM 2.5 Backup and Restore feature. We'll not discuss the setup of the feature here - you can find a great explanation here - we'll focus instead on the procedure adopted to move from 3 standalone RHACM Cluster Hubs to one Active Cluster Hub and two Passive Cluster Hubs.
Moving Standalone Cluster Hubs to Active/Passive Procedure
RHACM Backup and Restore feature was leveraged by Red Hat Consulting to move the managed clusters from the soon-to-be Passive Hubs into the designated Primary Hub by following this procedure:
-
Hub Backup was configured on all the RHACM cluster hubs, each Hub initially pointed to a dedicated prefix on the object storage bucket to have a valid backup for each of the hubs separately.
Hub1 example
DataProtectionApplication
:apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
name: dpa-hub1
spec:
configuration:
velero:
defaultPlugins:
- openshift
- aws
restic:
enable: false
backupLocations:
- name: default
velero:
provider: aws
default: true
objectStorage:
bucket: my-bucket
prefix: hub1 # <<<< Set a dedicated prefix for each of the hubs
config:
region: us-east-1
profile: "default"
credential:
name: cloud-credentials
key: cloud
snapshotLocations:
- name: default
velero:
provider: aws
config:
region: us-west-2
profile: "default" -
Once all the Hub backups are completed, delete the
BackupSchedule
CR to stop the Hub backups. -
Delete the
DataProtectionApplication
from the Primary Hub and recreate it using the prefix of the first Cluster Hub you wish to make Passive.Hub1 example
DataProtectionApplication
pointing to Hub2 prefix:apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
name: dpa-hub1
spec:
configuration:
velero:
defaultPlugins:
- openshift
- aws
restic:
enable: false
backupLocations:
- name: default
velero:
provider: aws
default: true
objectStorage:
bucket: my-bucket
prefix: hub2 # <<<< points to hub2 backup location
config:
region: us-east-1
profile: "default"
credential:
name: cloud-credentials
key: cloud
snapshotLocations:
- name: default
velero:
provider: aws
config:
region: us-west-2
profile: "default" -
Verify that the velero pod is running and check if the
BackupStorageLocation
is available:oc -n open-cluster-management-backup get backupstoragelocation
NAME PHASE LAST VALIDATED AGE DEFAULT
dpa-hub1-1 Available 58s 2d trueYou can also verify in the velero pod logs that the backups are discovered and imported.
-
Detach all the managed clusters from the Cluster Hub that you are making Passive, wait for the process to complete.
-
On the Primary Hub, create a one-shot
Restore
Custom Resource to import the managed clusters previously attached to the hub you are making Passive:apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Restore
metadata:
name: import-hub2-clusters
namespace: open-cluster-management-backup
spec:
cleanupBeforeRestore: None # IMPORTANT: prevent the restore to delete anything from the Primary Hub during the restore
veleroManagedClustersBackupName: latest
veleroCredentialsBackupName: latest
veleroResourcesBackupName: latestATTENTION: it is really important to set the
cleanupBeforeRestore
value toNone
to prevent accidental deletion of restored objects from the Primary Hub.Check the
Restore
status by runningoc -n open-cluster-management-backup describe restore import-hub2-clusters
This
Restore
can be safely deleted once completed with the commandoc -n open-cluster-management-backup delete restore import-hub2-clusters
-
Repeat the steps from 3 to 6 to move other Cluster Hubs to a Passive role.
-
When you are done moving the managed clusters from the Passive Hubs to the Primary Hub, you can complete the configuration of the Active/Passive feature.
Set a prefix on your
DataProtectionApplication
Custom Resource that will be used for both Active and Passive Hubs:apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
name: dpa-hub1
spec:
configuration:
velero:
defaultPlugins:
- openshift
- aws
restic:
enable: false
backupLocations:
- name: default
velero:
provider: aws
default: true
objectStorage:
bucket: my-bucket
prefix: active-passive # <<<< final location of the backups, passive hubs will restore from here
config:
region: us-east-1
profile: "default"
credential:
name: cloud-credentials
key: cloud
snapshotLocations:
- name: default
velero:
provider: aws
config:
region: us-west-2
profile: "default"Create the same
DataProtectionApplication
Custom Resource on all the Cluster Hubs. -
On the Primary Hub create the
BackupSchedule
Custom Resource to start taking backups to the new location. -
On the Passive Hubs create a one-shot
Restore
with thecleanupBeforeRestore
set toCleanupAll
to prepare the former standalone Hub to receive restores from the Primary Hub without any left-over:apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Restore
metadata:
name: restore-with-cleanupall
namespace: open-cluster-management-backup
spec:
cleanupBeforeRestore: CleanupAll # This will cleanup ALL the resources from the Passive hub
veleroManagedClustersBackupName: skip
veleroCredentialsBackupName: latest
veleroResourcesBackupName: latestOnly the resources that would be backed up will be deleted by this
Restore
, the task will honor backup exclusion label.This
Restore
, once completed, can be safely deleted with the commandoc -n open-cluster-management-backup delete restore restore-with-cleanup
-
On the Passive Hubs create the
Restore
Custom Resource to periodically restore passive data from the Primary Hub: cluster_v1beta1_restore_passive_sync.yaml
Common pitfalls
-
Pay attention to the name of the objects, if you find name collisions deal with them prior to switch to Active/Passive.
-
Policies applied to the Cluster Hub itself are backed up automatically, if those are different across your Cluster Hubs, exclude them using the label
velero.io/exclude-from-backup: "true"
Conclusions
Applying the procedure described earlier allowed the Customer to move to the new Business Continuity feature offered by Red Hat Advanced Cluster Management 2.5.
Sobre os autores
Navegue por canal
Automação
Últimas novidades em automação de TI para empresas de tecnologia, equipes e ambientes
Inteligência artificial
Descubra as atualizações nas plataformas que proporcionam aos clientes executar suas cargas de trabalho de IA em qualquer ambiente
Nuvem híbrida aberta
Veja como construímos um futuro mais flexível com a nuvem híbrida
Segurança
Veja as últimas novidades sobre como reduzimos riscos em ambientes e tecnologias
Edge computing
Saiba quais são as atualizações nas plataformas que simplificam as operações na borda
Infraestrutura
Saiba o que há de mais recente na plataforma Linux empresarial líder mundial
Aplicações
Conheça nossas soluções desenvolvidas para ajudar você a superar os desafios mais complexos de aplicações
Programas originais
Veja as histórias divertidas de criadores e líderes em tecnologia empresarial
Produtos
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Red Hat Cloud Services
- Veja todos os produtos
Ferramentas
- Treinamento e certificação
- Minha conta
- Suporte ao cliente
- Recursos para desenvolvedores
- Encontre um parceiro
- Red Hat Ecosystem Catalog
- Calculadora de valor Red Hat
- Documentação
Experimente, compre, venda
Comunicação
- Contate o setor de vendas
- Fale com o Atendimento ao Cliente
- Contate o setor de treinamento
- Redes sociais
Sobre a Red Hat
A Red Hat é a líder mundial em soluções empresariais open source como Linux, nuvem, containers e Kubernetes. Fornecemos soluções robustas que facilitam o trabalho em diversas plataformas e ambientes, do datacenter principal até a borda da rede.
Selecione um idioma
Red Hat legal and privacy links
- Sobre a Red Hat
- Oportunidades de emprego
- Eventos
- Escritórios
- Fale com a Red Hat
- Blog da Red Hat
- Diversidade, equidade e inclusão
- Cool Stuff Store
- Red Hat Summit