Store and analyze your test-suite logs with this open source tool

When you have lots of test suites to run, you want somewhere safe to keep all the logs and some way to analyze them. Bunsen is a small free, open source software toolkit that aims to address both tasks with an absolute minimum of infrastructure.

[ Download A system administrator's guide to IT automation. ]

Keep logs in Git

Most developers and sysadmins run test suites to check software. If you have many configurations, versions, and systems to test, you end up with a giant pile of log files. You may keep them on a filesystem somewhere, but that's not very efficient. A better way is to use Git, which is brilliantly suited to the compact storage of related textual content.

To do so, push all log files related to the test suite to a dedicated Git repo. Add some metadata about the build process if you like—source version info, host environment settings, whatever—into additional little text files. Push them all into one git commit and tag it.

The Bunsen toolkit's t-upload-git-push script automates these steps. You can delete the test suite directory since your logs are now safe.

You can get all the text files back out at any time with just a git checkout or git show. You can use the usual local and hosted tooling you already have to manipulate the repo. In the meantime, the logs are protected and thoroughly (99%) compressed.

Future Bunsen tools will help manage (and especially age-out) content.

[ Get A sysadmin's guide to Bash scripting. ]

Analyze the logs with SQLite

Once the test-run data is in Git, you can analyze it beyond just eyeballing logs. Parsing all the logs takes time, so it's best to keep analysis results in a separate data store that still allows high-performance search and computation operations.

One of the simplest ways is in an SQLite database. The Bunsen toolkit provides several analysis scripts written in Python 3, all named i-* (incremental) or g-* (global). They do one job each: i-dejagnu-parser processes DejaGNU-format log files, such as those produced by GCC and GDB. Another parses autoconf config.log files to see how the compiler development environment looks at the time. Other scripts parse various file formats or summarize previous analyses into higher-level results. All these scripts consume data from Git and add data to tables in the SQLite database.

A front-end script called pipeline runs all the analysis passes for any new test runs in Git. It also deletes data for any test runs you have removed from Git. Since the SQLite database contains only mechanically derived data, it can be deleted anytime, and the Bunsen pipeline script can recompute it from scratch. The relational data is encoded tightly, but it can be similar in size to the compressed Git repo.

[ Check out The ultimate CI/CD resource guide ]

Report with SQLite and the command line

Once the analysis is complete, it's time to inspect the results. You can even review the results during analysis; SQLite doesn't mind.

You could use normal SQLite tools to query the raw tables and related views, but that's an act of bravery because these are not well-documented yet. A better choice is using Bunsen command-line tools, all named r-* (for "reporting"). Here is a command summary:

• r-find-testruns: Searches the database for test runs with desired metadata values
• r-show-testrun: Summarizes all the statistics about one test run
• r-diff-testruns: Compares a bunch of them to identify regressions

These reporting scripts use a mixture of SQL and procedural Python to compute results. They can produce output in plain human-readable text or machine-readable JSON forms.

Report with HTTP

You may be thinking: This is 2022; surely there is a web interface, too. Yes, there is. The r-httpd-browse script offers a paper-thin Flask framework wrapper on the other Bunsen reporting scripts. It lets you click around the SQLite database and the related Git files and invoke several reporting operations. Here are a couple of screenshots.

Image

Image

Try it live with Sourceware Buildbots

A Buildbot software installation runs automated tests for many guest projects on sourceware.org, the Red Hat-sponsored community free software hosting site. The test results are available on the Sourceware Buildbot index page.

We test projects that include GNU toolchain components such as GCC and GDB and other tools like Valgrind and SystemTap, built and tested on various donated worker systems. Since all these projects use automated test suites, the Buildbot workers can naturally save those files and deposit them in a Bunsen Git repo on sourceware.org.

You are welcome to inspect and clone the Bunsen repository. We regularly run a Bunsen analysis pipeline and keep a hefty SQLite file up to date with all the fresh analyses.

While that file is not directly accessible, the HTTP frontend is. Direct your web browser at any little Bunsen-burner icon on the Sourceware Buildbot index page, and you'll see all the data available for the corresponding set of test-suite runs.

[ Want to test your sysadmin skills? Take a skills assessment today. ]

Try it in containers

Bunsen is a Python 3 toolkit that you can build easily on a Fedora or RHEL8+EPEL system. But to make it even simpler to try out, Bunsen has a container image with the package prebuilt:

% podman run --rm -i -t quay.io/bunsen-testsuite-analysis/bunsen /bin/sh
sh-5.1% ls /usr/local/bin
R-show-testcases   	i-dejagnu-summarizer  r-dejagnu-summary
[...]
i-automake-summarizer   pipeline   	  t-sourceware-mails-import
i-dejagnu-parser   	r-dejagnu-diff-logs     t-upload-git-push

With a few bound container volumes and an exposed TCP port, this container image will run a Bunsen pipeline analysis pass and then start the HTTP frontend for near-instant browsing of a test-run Git repo you may already have.

If you find the tool useful or you need something it doesn't do yet, please get in touch with the team at bunsen@sourceware.org. Let's build it together.