How we use Linux Test Project to test and improve Linux

The Linux Test Project (LTP) is a general-purpose, integrated test suite designed to help organizations using and developing Linux better understand what things work and what still needs work. It is comprised of regression and conformance tests designed to confirm the behavior of the Linux kernel and glibc. Its tools and test suites aim to verify the Linux kernel and related subsystems.

In short, the Linux Test Project (LTP) is aimed at testing and improving Linux. Its goal is to deliver a suite of automated testing tools for Linux and publish the results of the tests they run. For example, we use LTP tests on Red Hat Enterprise Linux (RHEL) to improve the Linux kernel and system libraries.

Image

What is RHTS-LTP?

LTP has a consistent development rhythm; it releases a new stable version of the test suite every four months. The main branch of upstream LTP is always in development. RHEL releases a minor stream every six months and provides a major update every three years. That means the RHEL production environment needs a diverse, safe, high-efficiency, and reliable test suite. For this reason, Red Hat maintains an internal version of LTP called Red Hat Test Suite-Linux Test Project (RHTS-LTP).

Understanding the workflow between the upstream and Red Hat

RHTS-LTP doesn't operate in isolation. It builds upon what LTP provides and submits patches back upstream. From an engineering perspective, here is the general workflow:

1. Choose a stable package version from upstream as the baseline
2. Backport new patches from the latest branch, which includes:
   • Updating Common Vulnerabilities and Exposures (CVE) and emergency test cases
   • Bug fixes for the framework or test defect
3. Submit the patch to upstream for common problem solving
   • Bug fix, new regression test, and so on

Image

Backporting patches

The development process, in terms of the upstream project, looks like this:

ltp-full-20210121 --> ltp-full-20210524 --> ltp-full-20210927 --> ltp-full-next

Only the stable version is downloaded to RHTS-LTP. So, ltp-full-20210927 along with its patches gets used in the internal test. Usually patches that fix critical common issues, RHEL regression bugs, or the test framework itself are backported.

[ Register for the free online course Red Hat Enterprise Linux Technical Overview to learn basic practical techniques for Linux and system administration tasks. ]

The stable version of the LTP test source is released every four months. After conducting centralized testing and fixing problems with some of the mainstream Linux distributions, the test release version gets tagged with the latest date. Then it's uploaded as a compressed tarball by the project maintainers, and users can download it from the LTP GitHub.

Before RHTS-LTP gets built in the final step, some internal configurations (important parameters, firmware, and so on) are applied to make it fit for the RHEL environment or a specific testing hardware system. After that, many RHEL-only patches get applied to build the LTP binary.

Triaging known issues

The point of testing is to identify errors, and errors get fixed only if they're reported. The most important part of RHTS-LTP is the problem-triage process, which helps internal users discern known from unknown issues. Red Hat maintains an internal LTP known-issues list so that testing can ignore false positives or fix the errors causing them.

The LTP known-issues list is based on various RHEL releases to ensure LTP's compatibility in Beaker tasks, including RHEL 5, RHEL 6, RHEL 7, RHEL 8, RHEL 9, and upstream versions. When a case of RHTS-LTP fails during running, the known-issue list is checked to identify whether the LTP tool is defective or if the RHEL build requires fixes.

This definitely benefits internal users testing minor releases and major releases, especially for various product lines, to locate problems promptly.

Image

Examining RHTS-LTP's mechanics

RHTS-LTP consists of four parts:

• Stable LTP package
• Upstream patches
• Internal configuration
• Known-issue filtering

We always choose the latest stable version of LTP and then add our custom changes to build a dedicated test suite for RHEL.

Here's the complete LTP maintenance process in diagram form:

Image

Supporting minor derivative versions

In addition to a special configuration and known-issues handling, RHTS-LTP also provides a shared library to support more derivative test versions for dedicated use in different functional teams. All of these use the common LTP stable version, with the same known issues, although the focus is on different components. These include:

• RHTS-Ltp-lite-test
• RHTS-Ltp-filesystem-test
• RHTS-Ltp-generic-test
• RHTS-Ltp-openposix-test
• RHTS-Ltp-git-test

Image

LTP also deploys for CentOS Stream testing and future various kernel support in Red Hat (including ark-kernel, mainline-kernel, and cki-testing). Almost every new forked branch is based on this with secondary development work. The internal maintainers exchange information through the include-library and work closely to enable efficient troubleshooting.

Testing together

Testing is a vital part of quality assurance, and we all want Linux to consistently be at its highest quality. This makes LTP and RHTS-LTP important parts of Linux development.

Linux isn't just a huge project; it's a diverse and widespread one. Contributions to testing come from the community. Whether that contribution is in the form of a patch or a specialized configuration, it has a definite positive impact on the upstream branch. It promotes the healthy growth of LTP and better testing in software development. With CentOS Stream working closer with RHEL than ever before, community participation is even easier than it was in the past. Visit the LTP wiki to learn how you can get involved.