Using a simplified Analytic Hierarchy Process with Scrum

I was having a discussion tonight with Mrs. Geek and we were chatting about prioritizing and estimating in projects. Yeah, it’s an odd relationship when you talk about this at home but then that’s what makes her Mrs. Geek. Anyways, she’s very scope focused and has done estimating and management on mega-projects but was never exposed to Scrum. I was trying to explain how Scrum works from a prioritization perspective and we got into this conversation.

As we got into it I explained how Scrum generally worked with prioritizing features. Not a lot of info is out there on *how* to prioritize, just that it’s part of the process. The Product Owner (PO) is meant to prioritize this list, however a lot of times while they may know what’s important to them, if it’s one person doing the prioritization you might miss some different perspectives.

Most people practicing Scrum say the PO does know his thing and will do the prioritization correctly, however a lot of times when new teams are adopting Scrum they may not have an approach other than something very traditional. For example Dmitri Zimine has a blog entry here on how to prioritize the work, but his technique is like most might think. Put the list up on a wall and assign values to each item (1–9, a-z, whatever). This is fine and produces a list but it’s hard to say if this is prioritzed very well or not.

Also there are many examples out there that show Product Backlog items like “Setup development environment” or “Provide Extract from Database to External Sources”. How is a Product Owner who has little or no knowledge of say the IT infrastructure or their processes supposed to prioritize this? To an IT guy, this is most important (and required to say do other things like oh, test the application) but for others that are more business oriented they might see this as low priority.

So let’s start with the traditional approach. Put together a list of items (our Product Backlog) and having the Product Owner prioritize it. Here’s our prioritized list:

Our PO thinks that the ability to organize and sort articles isn’t that important and more importantly is the ability to notify members with membership data (perhaps to publish a newsletter). We’re not sure based on these items what some of this means, or the method they went through to arrive at this order but we have to go with it as that’s what the customer wants right? I find the process to create this list to be flawed. While it’s great that the PO has put together the priorities, he does so by only looking one dimensionally at the values. Also whether they knew it or not, they’re probably not really looking at the entire list each time they try to prioritize each item.

For example, once you’ve gone through the first 5 items in our list you have their weights setup and assigned. Subconsciously I think you now dismiss those items for future planning. Sure, you’ve weighed those items against others (like Membership to Authoring) but when it comes to prioritizing Authoring, you ignore the inverse comparison of Authoring to Membership as you’ve already prioritized Membership. That’s cheating the process a bit and not giving the Authoring feature a fair chance. It only has to compare to what the left over items are and we know that inverse comparisons are not always just as simple as the opposite of what you’ve already looked at.

Enter a technique called the Analytic Hierarchy Process (AHP). This is an organization approach that focuses on the relative importance of objects to each other. This will take us from looking at our Backlog in one dimension to looking at it in two, and the outcome is a more well defined set of numbers that tells us what the priorities really are. In a nutshell, it does so by comparing each item to another item, then adding up the values for each item to arrive at a weight (or ranking, priority, whatever you want to call it).

Now, as Scrum is a simple process unto itself I wasn’t too happy when I read through the various explanations on AHP. Edwardo Miranda has a great white paper called “Improving Subjective Estimates Using Paired Comparisons” on how the process works. However once I saw those crazy mathematic symbols explaining the calculations behind it, my blurry college and university years came rushing back and I almost blacked out. James McCaffrey has a short MSDN article on using the process which didn’t get into as deep as Edwardo’s explanation, but I still thought it was too complex for Scrum.

Like I said, this whole conversation started tonight as we were talking about project estimates and priorities. Then it hit me that you could take the uber-simple process of tossing items into a list but apply AHP to it to get something a step above, but not as freaky as learning thermo dynamics or theoretical physics again. We went through this process while discussing the subject tonight so let’s take a look at how we can come up with a better list of priorities than what we have above.

Break out Excel (or do it on a white board for that matter) and set up your items as rows and columns in a matrix. Something like this (I’m using numbers for each item in the top row so it’s easier to read here, but you might want to use the full names set at an angle in your spreadsheet):

The order isn’t important here so it can just be your raw Product Backlog. Now go to each cell and compare the Feature to the other Feature in each column along the row (comparing the same item is redundant so it’s ignored and indicated here with 0). For example we’ll start with Setup Development Environment (Item #1). Is it more important, less important, or just as important as the ability to sign up for memberships (Item #2).

Give it 1 if it’s less important, 5 if it’s the same, and 10 if it’s more important. I find the 1–5–10 combo to be easy to use and remember. It’s either obvious it’s the same or more/less important. While you can assign say a number from 1–10, it starts creating long winded discussions like “Should it be a 6 or a 7?”. That’s detrimental to the exercise and you really don’t want to go down that rat hole.

A couple of things to note as you go through the items. You can compare Features from any angle you want. Sometimes it might be the business value, other times it might be risk or cost or whatever. You have to decide what’s best for the overall product. Also make it simple but don’t spend more than a couple of minutes for each item. Don’t try to keep track in your head of what the last compare was as it might taint what you’re really trying to compare against. Just focus on each item to each item then move on. Plain and simple.

Once you’ve done the first comparison, do the next column over and keep going to the end. Then start the next row and repeat the process. You’ll end up with a grid that looks like this:

Looks good and yes, it does take some time. For 10 items, you’re doing 90 comparisons but don’t cheat. Do the actual comparison. Think about if it’s really more or less important (or the same) and be honest. Just like you do an estimate on a task, think about it from whatever angle that makes sense. It does give you perspective on things and lets you look at your list in a different light.  The heart of the application, the core functionality that’s most important will bubble up eventually. For example in our scenario here, the PO felt that item #2 (sign up for memberships) was more important than item #4 (notifying members with membership data) but about the same as using credit cards for membership. Maybe that was a revenue decision, maybe it was functionality. Doesn’t matter as long as there was some thought behind it. Also try not to look at the other values. You won’t be doing yourself any service if you just replicate the inverse of the other values as that’s no different than doing a one-dimensional list. If you’re capturing it in Excel, just hide the first row when you move onto the next.

Now that you’re done, add up the values across the columns. This is our weight for each Feature. Now sort the list based on the value for that total and you’ll have your prioritized list. It’s that simple. Here’s our result from this exercise:

The “OR” column is our original ranking we did just by looking at all the items and coming up with the priorities. The “R” column is the weights calculated by adding up all the item to item compares.

Notice that after going through this, we find out that the all-important Setup Development Environment Feature is way down on the priority list, the Library Catalog for Articles that was the least important item now is somewhere in the middle, and the look and feel (when all is said and done) really is the least important item and should be done last. These are valid numbers as the PO went through and decided, on an item by item basis, what was important at the time.

Now this isn’t to say that this is gold. For example, we may need to setup the development environment in order to deliver the ability to sign up for memberships. This is where I belie the entire team, not just the PO, should prioritize the items (unless the PO really knows everything, which is rare). The PO should be telling the team why something is important while the team works out what’s risky or perhaps required for some items to be delivered.

A little negotiation goes a long way, but the main thing is that you get to see what’s really important here and why. Along with the why, you get to see the what. There are a lot of things that are 80 here but when you compare one item to the next, there are subtle differences. This will help when you have all those #1 items to deliver in Sprint 1 and don’t know which one to start on first.

Okay, this is a mythical example using made up values and such, but it really does work. It provides you a better weighting of what’s important and does put some pseudo-science behind it. As I said, AHP in it’s traditional form can be complex and what I’ve presented here is a simple version of it. In trying to keep with the tradition of Scrum being simple, I hope this echos that. You can, as you see fit, add layers to this process (or choose not to adopt it at all). For example, you can take those weights and apply them to estimates (costs) in your system to figure out what the best bang for the buck is (thanks to Mike Cohn for that tip).

Use what works for you but when you’re done the idea is that you might end up with a more accurate view of the prioritized world before you begin your Sprint.

P.S. Completely unrelated (but I found it while I was checking some references for this article) I noticed Mountain Goat Software, the site Mike Cohn runs got a major face lift. Looks a lot nicer now and easier to navigate. Check it out here as there’s lots of great articles and resources there on Scrum.