Open Source Stats--But What Do the Numbers Mean?

I recently sent a report to project management containing some numbers that purport to describe the status of the RDO project.

I got a long and thoughtful response from one of the managers—we'll call him Mark—and it seems worthwhile sharing some of his insights. To summarize, what he said was, don't bother collecting stats if they don't tell a story.

1. Focus on the Goals

Listing a bunch of numbers without context—even with pretty graphs—doesn't tell us anything unless you relate them to goals that we're trying to achieve.

Several weeks ago I presented a "stakeholder review" to this same audience. Any statistics that I present in the future should be directly related to a goal in that review, or they are just meaningless numbers, and possibly a distraction, and, worse still, might cause people to work towards growing the wrong metric. (Google for "be careful what you measure" and read any of those articles for more commentary on this point.)

2. Focus on the People

One of the stats that I provided was about how certain words and phrases feature in the questions on ask.openstack.org. Mark looked beyond the numbers and saw three people who are very active on that website, two of whom are not obviously engaged in the RDO community itself. Why not? How can we help them? How can they help us? What's their story? Why are we ignoring them?

3. Focus on the Blips

In February, our Twitter mentions, retweets, visits, and so on, went through the roof. Why? And why didn't we do that same thing again in March?

As it turns out, in February there were two conferences that contributed to this. But, specifically, we captured a lot of video at those events, and the Twitter traffic was all around those videos. So clearly we should be doing more of that kind of content, right?

4. Ignore the Stuff That Doesn't Seem to Mean Anything

We track "downloads" of RDO, which roughly speaking means every time someone runs the quickstart and it grabs the RPM. Except RDO is on a mirror network, so that number is false—or, at best, it reflects what the trends might be across the rest of the mirror network. So we have no idea what this metric means. So why are we bothering to track it? Just stop.

5. Ask Not-the-Usual-Suspects

This last one wasn't one of Mark's observations, but is what I'm taking from this interaction. We tend to ask the same people the same questions year after year, and then are surprised that we get the same answers.

By taking this data to a new audience, I got new answers. Seems obvious, right? But it's the kind of obvious thing we overlook all the time. Mark provided insight that I've been overlooking because I'm staring so hard at the same things every day.

By the way, I've presented Mark's insight very bluntly here, because it's important to be clear and honest about the places where we're not doing our job as well as we can be. Mark's actual response was much kinder and less judgmental, because Mark is always kind and supportive.