Who is Testing Your Cloud?

Co-Authored with Dan Sheppard, Product Manager, Rackspace

With test driven development, continuous integration/continuous deployment and devops practices now the norm, most organizations understand the importance of testing their applications.

But what about the cloud those applications are going to live on? Too many companies miss this critical step, leading to gaps in their operations, which can lead to production issues, API outages, inability to upgrade, problems when trying to upgrade and general instability of the cloud.

It all begs the question: "Do you even test?"

At Rackspace, our industry leading support teams use a proactive approach to operations, and that begins with detailed and comprehensive testing, so that not only your applications but your cloud is ready for your production workload.

Critical Collaboration

For Rackspace Private Cloud Powered by Red Hat, we collaborate closely with Red Hat; we test the upstream OpenStack code as well as the open sourced projects we leverage for our deployment, such as Ceph and Red Hat OpenStack Platform Director. This is done in a variety of ways, like sharing test cases upstream with the community via Tempest, creating and tracking bugs, and creating bug fixes upstream.

The Rackspace and Red Hat team also work together on larger scale and performance tests at the OpenStack Innovation Center, which we launched last year in conjunction with Intel to advance the capabilities of OpenStack.

Recent tests have included performance improvements in relation to offloading VXLAN onto network cards, scaled upgrade testing from Red Hat OpenStack Platform version 7 to version 8, and testing of scaled out Ceph deployments. Data from this testing will be made available to the community as the detailed results are analyzed.

Building on the upstream testing, the Rackspace operations team leverages Rally and Tempest to execute 1,300 test cases prior to handing the cloud over to the customer. This testing serves as a "1,300 point inspection" of the cloud to give you the confidence that your cloud is production ready and a report of this testing is handed over to you with a guide to help you get started with your new cloud. These test cases serve to validate and demonstrate the functionality of the OpenStack APIs, with specific scripts testing things such as (just to name a few):

• administration functions
• creating instances and cinder volumes
• creating software defined networks
• testing keystone functions and user management

Upgrades Made Easy

One of the key requirements for enterprises is the ability to upgrade software without impacting the business.

These upgrades have been challenging in OpenStack in the past, but thanks to the Rackspace/Red Hat collaboration, we can now make those upgrades with limited downtime to your guests on the Rackspace Private Cloud Powered by Red Hat.

To deliver this, the Rackspace team runs the latest version of OpenStack code through our lab and executes the 1,300 point inspection. When we are satisfied with that, we test upgrading our lab to the latest version and execute our 1,300 point test again, thus confirming that the new version of OpenStack meets your requirements and that the code is safe for your environment.

Our team doesn’t stop there.

So that the code deploys properly to your cloud, our operations team executes a 500-script regression test at the start of a scheduled upgrade window. Then our team upgrades your cloud and executes the regression test again. The final step in the scheduled upgrade window is to compare our pre- and post-regression test results to validate that the upgrade was successful.

Since the launch of Rackspace Private Cloud Powered by Red Hat, the Red Hat and Rackspace team has been working to refine that process by incorporating Red Hat’s Distributed Continuous Integration project.

Distributed Continuous Integration User Interface

Extended Testing

With Distributed Continuous Integration, Red Hat extends the testing process related to building Red Hat OpenStack Platform to Rackspace’s in-house environment. Instead of waiting for a general availability release of Red Hat OpenStack Platform to start testing Rackspace scenarios, pre-release versions are delivered and tested following a CI process. Test results are automatically shared with Red Hat’s experts and, along with Rackspace, new features are debugged and improved taking into consideration new scenarios.

Using DCI to test pre-released versions of Red Hat OpenStack Platform helps ensure we’re ready for the new general release just after launch. Why? Because we have been running incremental changes of the software in preparation for general availability.

DCI also helps existing Rackspace private cloud customers, by allowing the Rackspace operations team to test code changes from Red Hat while they’re being developed, allowing us a shorten the feedback loop back to Red Hat engineering, and giving us a supported CI/CD environment for your cloud at a scale not possible without a considerable investment in talent and resources.

So, if you are one of the 81 percent of senior IT professionals leveraging or planning to leverage OpenStack, ask your team, "How do we test our OpenStack?" — then give Rackspace a call to talk about a better way.