What are you using to measure outcomes?

challenges

(Richard Millington) #1

A quick question related to my @ALLIES book.

I’m looking for some interesting stories about what people measure and why. Specifically interested in people who can answer any of the following questions:

  1. What are you measuring at the moment?

  2. Have you changed what you measure in response to internal pressures (e.g. to focus on outcomes over-activity?)

  3. If you could measure anything you wanted, what would it be?

Any other interesting thoughts about measurement here would be much appreciated too. The chapter i’m working on is short of examples (even anonymous) examples at the moment.


(Rebecca Braglio) #2

At the moment, I’m doing some really granular type of measurement on individual user activity (I’m working on another Ambassador program). Our communities are all on wordpress, which makes getting metrics realllly challenging. Lots of manual work.

So, I have to be creative. I’m looking at each of our properties for individuals who have the potential to be ambassadors - I’m measuring the # who have changed their avatar, dates of registration and consistency in visits/activity, and “quality” of action. For example, I’m putting more weight on a user who answers a question or responds with encouragement over a user who just posts a status update.

I’m not sure how “scientific” or reliable this will be, but I’m trying!


(Kristen Gastaldo) #3

One of my main metrics is % of forum topics that are responded to. I’d love to do it within 24 hours, but it’s a manual process with Jive (partial issue with Jive, partial issue with our migration). I do it on Monday for the entire week before.

What I do with this data? Brag and nag, mostly. One of our communities is really active and has an 85% response rate. The other has become what basically feels like an unsupported product (until the next version comes out next year), and has about a 20% rate. So the good rate is often used to brag about our open source developer community, and was recently called out as a factor with analyst ratings. And the bad rate I use to try to drum up internal support for that community.


(Lana Lee) #4

Right now, to enlist internal engagement, I track page views (external and anonymous/not logged in). Internal teams want to know how many eyeballs I’m getting for posts because I’m trying to push SMEs to create webinars or write posts in the Community.

One example is that I asked a support SME to host a couple of webinars which were on topics that she received lots of tickets on. We post the recording in the Community and also upload the PPT presentation. Last quarter, it received 300 views and I used that metric to entice our training team to create promotional videos about their certification program and upcoming training courses. That metric made them very excited and they jumped on board :slight_smile:


(Richard Millington) #5

Hey everyone, thanks for this so far.

I actually meant how are you measuring the value of the community?

How do you know if the community is doing what it’s supposed to do?


(David DeWald) #6

In the past in a closed, Support based community we used Case Deflection as a measure.

We assigned an agreed upon value (via percentages) to the different ways that users could interact with the content.
For example, we would say that when Question was marked with an response as “Correct” it was 100% of the time a case deflection; whereas someone viewing that same question and answer in the future was a deflection 2% of the time.

With a known value for the cost of a case we could calculate a value for the number of cases “deflected”, subtracting out the cost of the community (monthly) to come up with a “Cost Savings” value.

This is an imperfect solution. There is no real way to determine the number of “views” - and other interactions - that were “real” case deflections. And there was no vetting if the questions were actually possible “support” cases.


(Lana Lee) #7

Hm…I ask for a customer/partner interview every chance I get - even if it’s in an email. I ask them for feedback. I’m starting to send messages to users who help others to show my appreciation and that opens up the line of communication.

At events, I ask if people are registered for the community to see how many buckets I can put people in. So far it’s the following:

  1. registered, participating and find it helpful - no qualms
  2. registered, but not participating > I follow up and ask them why. Usually it’s something beyond my control.
  3. registered, don’t find it helpful > I follow up to ask them what their biggest issue is and what I can do to help make the community better.
  4. not registered, no interest
  5. not registered, not seeing the value > I follow up to ask them what their biggest problems are and work with them to find a way to alleviate that “pain”

In all of the cases where I follow up, people are very eager to tell me what they’re looking for from the community; most of the time, they want to connect with other experts. Of course, I can’t ask everyone, but I try to get a good sampling.


(Robert McIntosh) #8

This is an ambition rather than a completed task as yet, because we are a new community, but the community is a place for members of a co-operative membership organisation that is in wine retail.

Our ultimate measure is “member satisfaction” with the overall Society, and the objective of the community is to add to the engagement between the Society and members, and between members with each other so that they are ultimately happier with their shared ownership.

In this case we will be evaluating their level of satisfaction through surveys, and I hope to demonstrate that the community is delivering BOTH overall increases in satisfaction (because more members are finding better wines and getting more from their membership) and that individual members who are participating in the community are more satisfied (their own satisfaction scores increase over time, and they are more satisfied compared to members who do not participate in the community).

There are other measures below this, regarding engagement with the community (inactive/reader/creator/moderator), and measures of revenue generated via referral traffic, but these are not as important as the higher level measure.

It is still early in the process as we have only just launched, but this is the goal we have set ourselves. Hope this helps