Melissa dives into the world of experimentation, answering subscribers’ questions about metrics and signals for internal applications, measuring the success of company transformations, and the best way to track experiments.
Show Notes
In this Dear Melissa segment, Melissa dives into the world of experimentation, answering subscribers’ questions about metrics and signals for internal applications, measuring the success of company transformations, and the best way to track experiments.
Q: Do you have any recommendations for metrics signals for internal applications, particularly where we are trying to change behaviors over the long term? How can an organization measure the success of a product-led company transformation?
Q: How do you track experiments and the results in a central way?
Q: How do I set a timeframe for measuring success and performance of a product before I pivot or iterate? How do I know when to kill a product, if after a couple more tries the initial idea didn't work?
Resources Melissa Perri on LinkedIn | Twitter MelissaPerri.com
Episode Transcript
Sounds like a dashboard type thing to me. And this is a little bit different because I think about dashboards as driving behavior. A lot of people like to think of them as like a status update, or let me just throw some data up there so people can see what's going on. But a dashboard is not useful unless it's causing somebody to change their behavior, right?
So what do I need to see as an engineer when I log in? All right, I'm logging in. What do you think it's going to do if they see case studies on internal migrations? How should it make them change, right? What do you want to do about learnings on incidents? Should it prevent those incidents from happening later?
So I'd want to see a decrease in incidents. Like how do you display that information? How do you do it in a way that drives them to take action? And then also, how do you measure the results of that? Like each one of these things is going to have a different outcome. You have to start from what's the outcome?
How do I measure it in my system? How do I instrument my system to actually measure it?
And then before that, get some qualitative feedback that they're actually using those things to help move that way. Whoa, whoa, whoa, whoa, whoa. That's a lot of questions. All right, let's dive in. Hello, and welcome to another episode of Dear Melissa. Today, we've got a great episode for you on experimentation, metrics and signals, and how to measure success. All fantastic questions on this. And remember, if you want me to answer your questions on Dear Melissa, go to dearmelissa.com and submit them. It could be about anything. It could be about careers.
It could be about breaking into product management. It could be about dealing with difficult stakeholders, whatever you want to talk about. That's what I'm here for. So don't be shy. Go send your questions, and I'm looking forward to answering them on a future episode. Okay, so let's dive in. First question.
Dear Melissa, do you have any recommendations for metric signals for internal applications, particularly where we are trying to change behaviors over the long term? For example, if we implement a product process to enable shared learnings on incidents, near misses, and case studies of internal migrations, how could I measure if this is a good or useful thing that's genuinely helping people be better engineers? What if I had an ambitious goal to create a product to teach the organization life cycles thinking, and with it, cause some cultural changes?
Realistically, I suspect this would take two to three years to show in data. What would the signals be? Taking a step back and looking at even bigger problems, how could an organization measure the success of a product-led company transformation? Wow, lots to unpack in this, and we will be right back after this message to answer this question.
All right, we are back, and we are talking about metrics and signals for internal applications. Lots of different questions in here. So let's break it down. Let's break it into a couple parts. All right, first question. We're talking about what are the metrics and signals for internal applications? So when I think about internal applications, I immediately go to workflows and behavior change.
So internal applications are usually there to help people do their jobs better, run the company, things like that. So how do you measure the success of someone doing their job? Can your application help them do it faster, better, or cheaper? I'd start there. Then I would think about the elements in your flow that contribute to that.
So can you calculate out what faster and cheaper means by seeing how much time a workflow saves? Then I'd measure the adoption of that workflow, and then that would help me predict how much time, which really comes down to cost, that can save. So that's how I kind of think of internal applications.
A lot of that, just making things better, making them faster. Another internal application could be giving people information or data they didn't have before to make better quality decisions. And then you want to measure the success of the decisions and how they're using that data. You're going to need some kind of combination here, though, of quantitative and qualitative thinking is really what this comes down to.
So quantitative metrics are going to be the actions that people take. But the qualitative thinking, right? Like the sitting down with them, doing the interviews, trying to figure out how they get the results, what it means, how it affects their lives. That's going to tell you what the quantitative things should be that you're measuring it.
So you have to do a little bit of research before you can actually make metrics and signals here. You have to basically start to prove or start to draw a relationship through. When somebody does X, Y, and Z in my platform, it ends up making them a better engineer because I can prove that with, I don't know, the quality of code, the less bugs that they have, the faster time to production, like something like that.
You qualify what makes a better engineer. And then you work backwards to figure out what do they need to do in my system? What actions do they need to take that would tell me they're a better engineer? And then can I measure this on a quarterly, monthly, daily basis? Whatever you want to do to actually prove that.
That's how I would think about that. So now, what you were talking about, though, in your second part of this, is what if we implement a product process to enable shared learning on incidents, near misses, case studies, or of internal migrations, right? Sounds like a dashboard type thing to me. And this is a little bit different because I think about dashboards as driving behavior.
A lot of people like to think of them as like a status update or let me just throw some data up there so people can see what's going on. But a dashboard is not useful unless it's causing somebody to change their behavior, right? So what do I need to see as an engineer when I log in? All right, I'm logging in.
What do you think it's going to do if they see case studies on internal migrations?
How should it make them change, right? What do you want to do about learnings on incidents? Should it prevent those incidents from happening later? So I'd want to see a decrease in incidents. Like, how do you display that information? How do you do it in a way that drives them to take action? And then also, how do you measure the results of that?
Like each one of these things is going to have a different outcome. You have to start from, what's the outcome? How do I measure it in my system? How do I instrument my system to actually measure it? And then before that, get some qualitative feedback that they're actually using those things to help move that way.
Another thing that you asked about too, taking a step back and looking at the bigger problem, how can an organization measure the success of a product-led company transformation? This is a great question. So I do it by looking at behavior change, like I said, throughout the organization. So I want to see that there is a product strategy tied to the company strategy, and we're talking about it and revisiting it constantly.
So I'm looking at your strategy cadences. Do you have them? What's discussed in them? What are the outcomes that come from it? So again, ticking the box would be, we have a strategy cadence, but okay, what's quality of it? Are decisions actually being made? Then I look at, did the products drive growth?
That to me is like the ultimate goal of a product-led company transformation. We talk about product transformations a lot, but why do it? You're doing it so that one, you can scale technology products to help enable your company to grow and meet customer demands and needs in a better way. There's an outcome associated with that.
Did our products drive growth? Did they drive customer success? Did they drive people staying with us in retention? You have to draw that. And it comes back down to product strategy drivers at the end of the day. That to me is like the ultimate success of a product-led transformation. But can you measure things along the way?
Yeah.
I want to see a chief product officer in the hot seat. I want to see somebody reporting into a CEO that oversees product. I want to see us measuring the success of our products and making decisions off it. Do we have instrumentation in our systems that say like, yes, what we released was successful or not?
So there's also a book out there by Nicole Forsgren. It's called Accelerate. It's got a bunch of DevOps metrics in there too. I'm going to defer to the expert here because she did a PhD on this. She has researched the hell out of it. And she has shown in this book and proven through numerous companies running this thing over and over and over again, how to shift a culture and what metrics you should measure to see the success in that.
So at the end of the day, these are the things I looked for. I just rattled them off, but I would say pick up the book, Accelerate, read through it, measure the things that she tells you to measure because that's what's going to tell you if you're really leading a cultural and product-led transformation in the right way. All right.
Next question.
How do you track experiments and the results in a central way? It feels like we learn from experiments at the time, but those learnings aren't necessarily carried forward over time as a team changes or shared with other teams working on related problems. This is a great question. I get this a lot. I see this a lot in companies that are trying experiments for the first time, but they haven't really embraced a true learning culture.
And a true learning culture means we actually take those learnings and act on it, right? We're not just experimenting for the sake of experimentation. Now, there is a real problem with showing results, sharing results, tracking experiments, making sure that people understand, you know, what we're doing and they're taking those learnings and they're integrating it into their own things.
So that's why I really advocate for a central research platform in companies. We did end up building something like this at Athena Health, which was really awesome. The design team led the charge around this to track user research, to disseminate it, to make sure that people could go back and pull out interviews and see what they were doing, even if they weren't the ones running the research.
Something like that is needed for both experiments and for any type of research. If you were small, you can use Airtable for this. I've seen a lot of companies do that. I've done that myself with companies. You can organize your research in that. You can filter it by topics and people that you talk to in companies and all that wonderful stuff.
As for a larger user research platform that captures all of this and disseminates it, I'm not... As for somebody that collects all of this data in one way, you know, a platform that can really do this, I know a lot that aid you in doing interviews and focus groups. Forsta, the company that I sit on the board of, that's what they do from a market research perspective.
So you can do interviews, focus groups on it, quantitative analytics, and they grab all of that in one central place for you. There's other user research platforms that allow you to do that as well, but they have their tools on there for interviews or tracking data, and then they pull that back in to store that. So for example, UserZoom does usability testing.
They do interviews through UserZoom, so you can find your users and do that. They do surveys, that type of stuff. But they have their collection methodologies on there. It's not just a repository for your user research. So that's why I'm kind of gravitating towards Airtable with your question, because you can do whatever you want and then dump the interview results in there.
But you may want to invest in some kind of user research platform, market research platform that allows you to get in touch with users in different ways. These will have repositories on there as well, tracking this data, synthesizing it, being able to share it, but you'll use their tools to collect it. So that's something to think about, maybe investing in a platform like that.
All right, last question. Dear Melissa, how do I set a timeframe for measuring success and performance of a product before I pivot or iterate? How do I know that if after a couple more tries the initial idea didn't work, am I going in the right direction or should I kill that product? This is a tricky one.
You don't want to spin your wheels forever, but you also don't want to give up too early. And I've actually been in this situation before. I think what it comes down to is, are you testing on the right variable that's going to drive change? So what I've found is that if you're experimenting and it's not working, usually you don't understand the problem well.
And that happened to me once. I was working at a company that, basically an e-commerce company. We were iterating on the homepage, trying to drive conversion rate, changing things like images, changing the text on it, and nothing was working. It turns out that that wasn't the problem. It had nothing to do with the design or the images or these little tweaks.
It had to do with the entire workflow of the conversion funnel. So the problem was much bigger. It was less like little tiny things that we were testing. It was more big things, big underlying issues that we had with the platform. And that's what you're going to want to take a step back on and think about if what you're trying is not working.
Now, let's say for a timeframe, it really depends on how much data you can get. So if you are like A-B testing on a consumer website that gets hundreds and thousands of people a day, you can run a bunch of tests in probably an hour, right? And know if something's working or not. You get statistical significance.
But if you are in a B2B company and you have to run a pilot test with like five to 10 people, it might take a little bit longer to make a determination. You may get instantaneous feedback that nobody's using it. Easy thing to do, easy decision to make. I mean, you can just figure out, okay, gotta change this, gotta do something else.
Or you may not learn for a while what's really working, what's not, because they have to use it longer. Or maybe they don't have that problem every single day. So they only use it once a week and you can only collect feedback once a week. These are things to think about. So how often do users actually experience the problem that I can go back and actually measure the success of my experiments in those moments?
That's gonna give you a timeframe.
And then also how many people can I get quickly to start to get a large enough number to tell me if I'm going the right direction or not. When you do more qualitative experiments, you're gonna have smaller numbers of people. When you do more quantitative measurements, you're gonna want larger numbers of people.
So that's something to think about as well. Now, when do you decide like whether to kill it or spin it or do something else? You have to put a line in the sand based on that data. That's what I would say. If you tried something, it's been three months, somebody experienced this problem once a week, you've got a group of 10 people going through it, it's not getting traction.
Yeah, it's definitely time to try something new. And maybe you iterate on it a little bit and you try it again and it still doesn't work, right? What you should be doing though here is getting so much feedback from those users using it. You wanna talk to them constantly. You wanna be sitting in front of them, watching them use it so that you can pinpoint the problems they're actually having with it.
Because that's gonna tell you to pivot or iterate. A lot of times we launch experiments out to the world, we never keep tabs on them. And then we are like, okay, what do we change? What went wrong? Can't tell, you can't tell unless you're there. So sit with them, look at it. That's gonna tell you whether you should try something new or if they're just really not interested, that's where you're gonna say, okay, I gotta pivot to something else.
So there's no easy answer. There's no one timeframe, but those are the things I would think about. I would go with, who is my audience? Who am I testing this with? How much data can I get from them? When did they experience this problem? And that's how I would set my timeframes to something comfortable, to let them use it for a while, to get enough feedback.
And then I would really focus on getting the qualitative feedback if you were working in a B2B environment or something where I can only get a small group of users on it. And then if you're working in a consumer environment where I can throw a lot of people at a test, do that and make decisions quickly. And if you were iterating and iterating and just tweaking things and it's not working, usually the underlying problem or the overall solution is not what it should be.
And I would go step back a couple of steps, try to figure out what that is and then proceed from there. All right, that concludes this week's Dear Melissa questions. Thank you so much for listening. And again, if you have any questions for us, go to dearmelissa.com. Otherwise, you can join us next week. We'll have another guest on here. And if you love the podcast, we really appreciate a review on Apple Podcasts or Spotify or wherever you're listening to podcasts to tell other people to come and listen to this as well. Thanks very much.
