Updating clusters can be complex. When there's a problem, you have to determine which alerts are unimportant, which logs to focus on, and work to pinpoint the real issue behind the issue. To make the update process smoother and more transparent, the Red Hat OpenShift team has developed a new tool: oc adm update status
. This new command helps you obtain detailed information about the state of your cluster during an update. And the functionality, initially available in the oc command-line tool, is set to evolve into a full-fledged API provided by the cluster itself.
Technology preview in Red Hat OpenShift 4.16
The oc adm upgrade status
command debuts as a technology preview in Red Hat OpenShift 4.16. This new subcommand is entirely client-side in its current version; you only need the 4.16 oc binary. It's safe to use in production environments as it only reads data without modifying any configurations.
How to Use oc adm upgrade status
To use this new feature, you must first enable it by setting the OC_ENABLE_CMD_UPGRADE_STATUS
environment variable to true
:
export OC_ENABLE_CMD_UPGRADE_STATUS=true
Then run the command during an update:
$ oc adm upgrade status
= Control Plane =
Assessment: Progressing
Completion: 12%
Duration: 12m5s
Operator Status: 33 Healthy
Control Plane Nodes
NAME ASSESSMENT PHASE VERSION EST MESSAGE
ip-10-0-30-217.us-east-2.compute.internal Outdated Pending 4.14.0 ?
ip-10-0-53-40.us-east-2.compute.internal Outdated Pending 4.14.0 ?
ip-10-0-92-180.us-east-2.compute.internal Outdated Pending 4.14.0 ?
= Worker Upgrade =
= Worker Pool =
Worker Pool: worker
Assessment: Excluded
Completion: 0%
Worker Status: 3 Total, 3 Available, 0 Progressing, 3 Outdated, 0 Draining, 3 Excluded, 0 Degraded
Worker Pool Nodes
NAME ASSESSMENT PHASE VERSION EST MESSAGE
ip-10-0-20-162.us-east-2.compute.internal Excluded Paused 4.14.0 -
ip-10-0-4-159.us-east-2.compute.internal Excluded Paused 4.14.0 -
ip-10-0-99-40.us-east-2.compute.internal Excluded Paused 4.14.0 -
= Update Health =
SINCE LEVEL IMPACT MESSAGE
- Warning Update Stalled Outdated nodes in a paused pool 'worker' will not be updated
Run with --details=health for additional description and links to related online documentation
The command provides the following information:
- Control plane: Shows the progress, completion percentage, and health status of operators
- Worker upgrade: Details on worker nodes, including their current state and any issues affecting their upgrade process
- Update health: Offers insights into the health of the update process, highlighting any critical issues and providing actionable advice
Here's example output when the control plane nodes are updated first during an OpenShift update:
= Control Plane =
Assessment: Progressing
Completion: 97%
Duration: 1h58m50s
Operator Status: 28 Healthy, 1 Unavailable, 4 Available but degraded
Control Plane Nodes
NAME ASSESSMENT PHASE VERSION EST MESSAGE
ip-10-0-30-217.us-east-2.compute.internal Outdated Pending 4.14.0-rc.3 ?
ip-10-0-53-40.us-east-2.compute.internal Outdated Pending 4.14.0-rc.3 ?
ip-10-0-92-180.us-east-2.compute.internal Outdated Pending 4.14.0-rc.3 ?
This next example shows the worker nodes updating after the control plane has updated during an OpenShift update:
= Worker Upgrade =
= Worker Pool =
Worker Pool: worker
Assessment: Degraded
Completion: 39%
Worker Status: 59 Total, 46 Available, 5 Progressing, 36 Outdated, 12 Draining, 0 Excluded, 7 Degraded
Worker Pool Nodes
NAME ASSESSMENT PHASE VERSION EST MESSAGE
build0-gstfj-ci-prowjobs-worker-b-9lztv Degraded Draining 4.16.0-ec.2 ? failed to drain node: <node> after 1 hour ...
build0-gstfj-ci-prowjobs-worker-b-bg9f5 Degraded Draining 4.16.0-ec.2 ? failed to drain node: <node> after 1 hour ...
build0-gstfj-ci-prowjobs-worker-b-mrxwn Degraded Draining 4.16.0-ec.2 ? failed to drain node: <node> after 1 hour ...
build0-gstfj-ci-tests-worker-b-4h7pn Degraded Draining 4.16.0-ec.2 ? failed to drain node: <node> after 1 hour ...
build0-gstfj-ci-tests-worker-b-jv5bg Degraded Draining 4.16.0-ec.2 ? failed to drain node: <node> after 1 hour ...
build0-gstfj-ci-tests-worker-b-kj6gk Degraded Draining 4.16.0-ec.2 ? failed to drain node: <node> after 1 hour ...
build0-gstfj-ci-tests-worker-c-dcz9p Degraded Draining 4.16.0-ec.2 ? failed to drain node: <node> after 1 hour ...
build0-gstfj-ci-tests-worker-c-jq5rk Unavailable Updated 4.16.0-ec.3 - Node is unavailable
build0-gstfj-ci-tests-worker-c-2kz4m Progressing Draining 4.16.0-ec.2 +30m
build0-gstfj-ci-tests-worker-c-55hpj Progressing Draining 4.16.0-ec.2 +30m
...
Omitted additional 49 Total, 22 Completed, 46 Available, 3 Progressing, 27 Outdated, 3 Draining, 0 Excluded, and 0 Degraded nodes.
Pass along --details=nodes to see all information.
You can see insights about an ongoing update:
= Update Health =
SINCE LEVEL IMPACT MESSAGE
14m4s Info None Upgrade is proceeding well
Of course, sometimes not everything goes as planned. Here's an example of the kind of output you get when there's a problem.:
= Update Health =
SINCE LEVEL IMPACT MESSAGE
20m24s Error API Availability Cluster Operator machine-config is unavailable (MachineConfigControllerFailed)
- Error Update Stalled Node build0-gstfj-ci-prowjobs-worker-b-9lztv is degraded
- Error Update Stalled Node build0-gstfj-ci-prowjobs-worker-b-bg9f5 is degraded
- Error Update Stalled Node build0-gstfj-ci-prowjobs-worker-b-mrxwn is degraded
- Warning Update Speed Node build0-gstfj-ci-tests-worker-c-jq5rk is unavailable
Run with --details=health for additional description and links to related online documentation
Use the --details=health
option to see individual insights expanded with further information about the update issue:
$ oc adm upgrade status --detailed=health
...
= Update Health =
Message: Node build0-gstfj-ci-prowjobs-worker-b-9lztv is degraded
Since: -
Level: Error
Impact: Update Stalled
Reference: https://docs.openshift.com/container-platform/latest/post_installation_configuration/machine-configuration-tasks.html#understanding-the-machine-config-operator
Resources:
nodes: build0-gstfj-ci-prowjobs-worker-b-9lztv
Description: failed to drain node: build0-gstfj-ci-prowjobs-worker-b-9lztv after 1 hour. Please see machine-config-controller logs for more information
Message: Cluster Operator machine-config is unavailable (MachineConfigControllerFailed)
Since: 20m24s
Level: Error
Impact: API Availability
Reference: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/ClusterOperatorDown.md
Resources:
clusteroperators.config.openshift.io: machine-config
Description: Cluster not available for [{operator 4.14.0-rc.3}]: ControllerConfig.machineconfiguration.openshift.io "machine-config-controller" is invalid: [status.controllerCertificates[0].notAfter: Required value ... some validation rules were not checked because the object was invalid; correct the existing errors to complete validation]
Future roadmap
Our roadmap includes transitioning this functionality to a cluster-provided API in OpenShift 4.17, allowing for broad consumption and integration across different tools and platforms. This initial release in the client helps us gather feedback to refine the feature.
We encourage everyone involved in cluster updates to try this new feature and share your feedback, helping us better serve your needs in future releases.
À propos de l'auteur
Subin Modeel is a principal technical product manager at Red Hat.
Parcourir par canal
Automatisation
Les dernières nouveautés en matière d'automatisation informatique pour les technologies, les équipes et les environnements
Intelligence artificielle
Actualité sur les plateformes qui permettent aux clients d'exécuter des charges de travail d'IA sur tout type d'environnement
Cloud hybride ouvert
Découvrez comment créer un avenir flexible grâce au cloud hybride
Sécurité
Les dernières actualités sur la façon dont nous réduisons les risques dans tous les environnements et technologies
Edge computing
Actualité sur les plateformes qui simplifient les opérations en périphérie
Infrastructure
Les dernières nouveautés sur la plateforme Linux d'entreprise leader au monde
Applications
À l’intérieur de nos solutions aux défis d’application les plus difficiles
Programmes originaux
Histoires passionnantes de créateurs et de leaders de technologies d'entreprise
Produits
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Services cloud
- Voir tous les produits
Outils
- Formation et certification
- Mon compte
- Assistance client
- Ressources développeurs
- Rechercher un partenaire
- Red Hat Ecosystem Catalog
- Calculateur de valeur Red Hat
- Documentation
Essayer, acheter et vendre
Communication
- Contacter le service commercial
- Contactez notre service clientèle
- Contacter le service de formation
- Réseaux sociaux
À propos de Red Hat
Premier éditeur mondial de solutions Open Source pour les entreprises, nous fournissons des technologies Linux, cloud, de conteneurs et Kubernetes. Nous proposons des solutions stables qui aident les entreprises à jongler avec les divers environnements et plateformes, du cœur du datacenter à la périphérie du réseau.
Sélectionner une langue
Red Hat legal and privacy links
- À propos de Red Hat
- Carrières
- Événements
- Bureaux
- Contacter Red Hat
- Lire le blog Red Hat
- Diversité, équité et inclusion
- Cool Stuff Store
- Red Hat Summit