What to measure

Technology is very good at producing new things to measure: the trick is knowing what to watch. Unless you’re careful, you can quickly end up drowning in a sea of meaningless statistics, thinking you’re making progress but missing the important trends and impact of your project. Once again, having a clear purpose for your project is critical; your task now is to turn that purpose into a set of key metrics that between them reveal whether you have achieved it.

What you measure about your technology will be determined by how you want people to use it. Do you want to engage a small group of people to do a range of things repeatedly, or a large group of people to do something specific once? Silicon Valley startup investor and entrepreneur Dave McClure explores these issues in his excellent presentation Startup Metrics for Pirates: AARRR!!! He suggests five areas of user behaviour that you can usefully measure:

• Acquisition: users come to the site from various channels (eg. new visitors per month)
• Activation: users enjoy 1st visit: ‘happy’ user experience (eg. 7% ofvisitors sign up)
• Retention: users come back, visit site multiple times (eg. 10% of users come back)
• Referral: users like product enough to refer others (eg. average user recruits 0.2 others)
• Revenue: users conduct some monetisation behaviour (eg. average value per user > £5)

Dave also describes three kinds of users: visitors, contributors and distributors. Put simply, visitors passively consume your content; contributors create new content; and distributors tell other people about your content. Depending on your project purpose and objectives, you’ll need to choose which of these are more important to you.

For example, you could measure the number of unique visitors to your website (using a web stats package like Google Analytics), but this might not mean much if your goal is to get people creating content. You could measure how many of those users come back, which might give you an idea of how much value people are getting from your platform, but that’s not so useful if your tool is designed to help people perform a one-off function, like signing a petition. If your goal is to encourage awareness of a cause, you might be interested in pageviews for your own site, but also references to your project elsewhere on the internet, or how many of your users invite friends to join your site. You can even measure how many followers you get on Twitter, or the number of mentions for your project in Google.

Agree a set of 5-10 key performance indicators (KPIs) that collectively indicate whether you are delivering on your objectives or not. Try to get a good balance of KPIs that measure: the scale of your impact; the nature of your engagement with users; the loyalty of your users; the speed of user growth; the utility and usability of key features of your site.

Make sure you set realistic levels for each of these, so that you have something to aim for that you are happy to share with staff, funders and project sponsors. Run them through a load of scenarios to see if they cover all the situations you’d like to know about, such as huge success scenarios, terrible failures, and those tricky bits in between where things may look like they’re working but in fact you’re missing vital parts of the puzzle. Get these KPIs right and it makes it much easier to set priorities for future activities, hold open conversations with funders, and feel confident that your project is progressing in the right direction.

Be kind to yourselves though: set conservative targets. Just because these tools have the potential to achieve huge impact doesn’t mean they will. Slow and steady growth is fine, and you’ll often get sudden jumps in usage without any indication why. There aren’t any magic buttons to press: the more you put people in control, the harder it is to ‘do things’ to them and force progress towards your targets. The nature of these technologies means surrendering many of the obvious control mechanisms for making the project succeed.

Targets are useful if they motivate you and keep you focussed on what’s important, but if they become a set of broken promises and impossible dreams, then you could feel like you’ve failed when really you haven’t. Tailor your targets to fit what you can control, and focus on creating the right conditions for growth rather than forcing it to happen to a fixed timeline.