Bugzilla and the zombie apocalypse: understanding the life cycle of a bugzilla

Bugzilla is an open source project, details: https://www.bugzilla.org/. The instance running at bugzilla.redhat.com is heavily used for Red Hat Enterprise Linux, Fedora and other projects. While customers interface with us from Red Hat Support mostly via Customer Portal cases, bugzilla is used as an interface between Red Hat Support, Red Hat engineering and partners.

In this article, I illustrate the most important states of a bugzilla case. ‘bugzilla’ is actually the name of the upstream project, but we are also using ‘bugzilla’ or even just ‘bz’ to refer to single bugzilla cases/tickets.

While not aiming at explaining each detail, I will try to give an overview, using a specific bugzilla.

Creation of a bugzilla

A new bugzilla is usually created when

• either a problem was observed

• or a feature is getting requested

Most Red Hat Enterprise Linux bugzillas are opened by Red Hat Support associates after communication with customers. The bugzilla I will use here is reporting that the Red Hat Enterprise Linux 6 kernel cannot indicate that we have been attacked by zombies, so that feature is requested. The bugzilla can be accessed here: bz746698.

Let’s look at some of the fields of this bz. The missing feature was reported for Red Hat Enterprise Linux 6.4, and the affected component is the ‘kernel’ - when an issue gets investigated it might turn out that a different component needs to be modified, so the component can be changed afterward. Some issues also require changes in multiple components. For these, multiple bugzillas get created, and a dependency is set up via the ‘Depends On’ and ‘Blocks’ fields. An example for this is bz1291286.

‘severity’ contains the importance for the requestor, with Customer Portal cases this is almost identical to the case severity. Priority is internally used.

The ‘Flags’ box is visible for users and has a condensed view on some important things. Our issue here was aimed at the Red Hat Enterprise Linux 6.3 release, together with the bz status of CLOSED ERRATA we know that the fix went into Red Hat Enterprise Linux .3GA. Single departments were here providing statements regarding the bz: as developers were sufficiently sure that a fix can be developed, they did provide devel_ack+. When our quality assurance colleagues (aka Quality Assurance (QA)) confirmed a way to verify the fix and that they can allocate resources for the verification, they provided qa_ack+. Eventually, product management signaled with pm_ack+ that resources should be allocated to fix this issue.

For a newly created bz, these flags have a question mark, so are not set.

Further progress

Scrolling a bit down on our zombie-bz, we see that the bz got directly assigned to a colleague, and a first patch was attached to the bz. Further comments wonder whether this bz should be a blocker: if a bz has blocker status, then the version it is aimed at (here Red Hat Enterprise Linux 6.3) cannot be released without the blocker bz getting solved. For example, if a bug in the installer is preventing deployment to hard discs, the bz would probably become a ‘blocker’.

Discussion about QA spins up. QA should verify that the fix really resolves the issue. In some cases, this requires special setups or hardware only a partner or customer has, in other cases this verification can be automated. In our case, verification of notifications regarding a zombie attack is a bit special, of course.

A lively discussion unfolds on how to resolve the issue best. People commenting are by default added to the CC field of the bz and receive an email when further comments are done.

We also start wondering if this issue exists on other Red Hat Enterprise Linux releases: in such cases we clone a bugzilla for the other release, for example, Red Hat Enterprise Linux 5, and then investigate if a fixed is also there required.

Priority and severity of the bz get adjusted at that point - appropriate for a zombie outbreak! A patch is developed, but we figure that we cannot make it into the 6.2 minor release anymore and aim at 6.3.

More on bugzilla states

Bz state ASSIGNED means that we have named a person to look into this, the assignee. If a patch is developed for the kernel component, then is discussed/reviewed with colleagues, reflected in state POSTED. If the patch has seen enough review it gets committed into the source tree and the bz moves to state MODIFIED. The Fedora Project has more details on bz states, most also applicable to Red Hat Enterprise Linux.

For serious issues without a workaround, we can also release fixes for critical issues in the currently open z-stream, at the time of writing 6.9.z and 7.4.z are the open z-streams. For such requests, the bz aimed at the next minor release can get cloned for the z-stream and get evaluated whether the errata should be developed. Our zombie bz got cloned to 6.1.z. The cloned bz was considered, but as this is a new feature, the bz was closed. Z-stream is for stability, features just get introduced in new minor releases.

One of our approaches is ‘upstream first’, the idea to bring code changes upstream first and then ship it as part of Red Hat Enterprise Linux. Discussing changes in upstream with a wide audience can take time, but the benefits are huge: chances of problems with a patch get lower, and customers can also use other distributions which contain the upstream code. Discussion in upstream can take time, all of this may be while a discovered and disclosed urgent issue is leaving customers vulnerable to a possible zombie invasion - so sometimes we also implement a remedy and later convert to an upstream solution. Here for the zombie bz, the patch did not make it upstream, it’s just in Red Hat Enterprise Linux 6.3 and later. Red Hat Enterprise Linux 7 has its roots in upstream/Fedora, so was a clean start without the zombie patch.

In comment 63 we see that the bugzilla gets CLOSED ERRATA, following the link we see that the fix becomes available with kernel-2.6.32-279.el6. As for documentation, kbase Why is the kernel "tainted" and how are the taint values deciphered? is mapped to the bz, documenting a new tainted state TAINT_BIT_BY_ZOMBIE.

Now, let’s test the zombie warning functionality!

Grab a free test system with Red Hat Enterprise Linux 6.3 or a later Red Hat Enterprise Linux 6, or something based on Red Hat Enterprise Linux like CentOS or Scientific Linux. Let’s first have quick look at the documentation, it has a new kernel-parameter:

[root@rhel6.9 ~]# yum install kernel-doc
Loaded plugins: product-id, security, subscription-manager
Updating certificate-based repositories.
[..]
[root@rhel6.9 ~]# grep OMGZOMBIES \
/usr/share/doc/kernel-doc-2.6.32/Documentation/kernel-parameters.txt
    OMGZOMBIES   BRRRAAAAAAIIIIINNNNNSSSSSSSS
[root@rhel6.9 ~]#

This is also available via the Red Hat Code Browser.

Now that we see that our kernel understands the OMGZOMBIES option, let’s add it:

[root@rhel6.9 ~]# vi /etc/grub.conf

We are changing this line:

kernel /vmlinuz-2.6.32[..] rd_NO_DM

Into this:

kernel /vmlinuz-2.6.32[..] rd_NO_DM OMGZOMBIES

After rebooting the system, we see the zombie warning in the kernel ring buffer:

[root@rhel6u9 ~]# dmesg|tail -5
[BRRAAIIIINNNSSSSS!] type=1305 audit(1508986586.506:57627): auid=4294967295 ses=4294967295 op="remove rule" key=(null) list=4 res=1
[BRRAAIIIINNNSSSSS!] type=1305 audit(1508986586.506:57628): audit_enabled=0 old=1 auid=4294967295 ses=4294967295 res=1
[BRRAAIIIINNNSSSSS!] readahead-collector: starting delayed service auditd
[BRRAAIIIINNNSSSSS!] readahead-collector: sorting
[BRRAAIIIINNNSSSSS!] readahead-collector: finished
[root@rhel6u9 ~]#

Final thoughts

We have looked at the most basic parts and steps in the life of a bugzilla as it is used for Red Hat Enterprise Linux, and then verified with multiple tools that the fix is available. While it is not intended to be used directly by Red Hat Enterprise Linux customers, knowledge about the states is useful.

• While bugzilla is used to track all kinds of issues and requests, we have seen that it can even help implementing notifications about a zombie attack! I am honored to be among colleagues where these things are possible.

• Bugzilla also offers many search options - but the fastest way to search tends to be via search engines, for example, using syntax ‘site:bugzilla.redhat.com search terms’.

• The ‘shadowman’ kernel option is also rumored to have an effect.

Christian Horn is a Red Hat TAM based in Tokyo, working with partners and customers. In his work as Linux Engineer/Architect in Germany since 2001, later as Red Hat TAM in Germany and Japan, Virtualization, operations and performance tuning are among the returning topics of this daily work.

A Red Hat Technical Account Manager (TAM) is a specialized product expert who works collaboratively with IT organizations to strategically plan for successful deployments and help realize optimal performance and growth. The TAM is part of Red Hat’s world-class Customer Experience and Engagement organization and provides proactive advice and guidance to help you identify and address potential problems before they occur. Should a problem arise, your TAM will own the issue and engage the best resources to resolve it as quickly as possible with minimal disruption to your business.

Connect with TAMs at a Red Hat Convergence event near you! Red Hat Convergence is a free, invitation-only event offering technical users an opportunity to deepen their Red Hat product knowledge and discover new ways to apply open source technology to meet their business goals. These events travel to cities around the world to provide you with a convenient, local one-day experience to learn and connect with Red Hat experts and industry peers.