Tracking Bugs
“Can you clarify - how issues are assigned to developers? Does a tester assign issues directly to members of development team, or they assign them to PM who will assign those issues to team members? Or issues are automatically distributed between development groups based on member's workload? What kind of software do you use for issue tracking?”
Here are some answers:
What bug tracking software do we use?
We use a home-grown bug tracking system that we currently call “Product Studio” to track and assign bugs and issues to team members. What is cool is that this code-base and the team that built it moved to the Visual Studio group last year, and a new and even richer version of the software will now ship as part of our Visual Studio Team System (VSTS) product suite – which means that starting next year you can use the exact same system we use internally at Microsoft for your own projects.
It stores all bug details in a database and provides rich querying and reporting statistics on issues. Below is a screenshot of what the current ASP.NET Dev Team bug query looks like as of a few minutes ago within it:
(click the picture above to see a full size version of it)
The columns are configurable – I usually like to track AssignedTo, OpenDate, ChangeDate (meaning last time the bug was updated), its IssueType (code defect, performance issue, spec issue, etc), and then the bug title with my main query.
You can then double click on any entry within the query list to pull up the full details about a bug. For example, here is one from ASP.NET 2.0:
(click the picture above to see a full size version of it)
The fields are configurable, and you can add additional fields to the system if you want to track custom settings. The key fields we usually use on my team to track bugs are:
1) Repro Steps. This field defines crisp steps in numeric order that enable anyone to easily reproduce the issue being reported.
2) Details. This field provides additional information discovered during the investigation of the bug, and comments/information that we want to make sure never get lost regardless of who picks it up to fix.
3) Priority. This field indicates the relevant priority of fixing this bug. P0 means drop everything and fix it immediately while P4 means it is a minor fit-finish style bug that could be ignored for awhile.
4) FixBy. This field indicates the project milestone the bug should be fixed by.
5) Status. This field indicates one of three settings: Active (meaning the bug is open), Resolved (meaning the bug has been fixed but not verified), and Closed (meaning the bug has been fixed and verified and is no longer an issue).
6) Path. This field indicates the feature area of the product (with the top-level path value indicating the product itself). This enables leads to do queries based on their feature areas as opposed to simply who the issues are assigned to (which is useful when 1 person owns multiple feature areas).
7) Build. This field indicates what build drop the issue exists on. A standard build number might look like: “Lab22d.40531”. The Lab22d part indicates what branch and build-lab the build was produced from. The 5 digit version string then indicates the specific build from that lab (this is an always increasing number). One trick we use on the .NET Framework and Visual Studio is to use a year/month/day format for our build numbers. So 40531 indicates the May 31st, 2004 build. This means that numbers are sometimes non-consecutive as months change (for example: the June 1st build after 40531 would be 40601), but it makes it very easy to figure out how old a build is. If you have VS Whidbey installed on your machine right now, you can go to the About Menu item to see this in action.
8) AssignedTo. This field indicates who on the team currently owns the issue.
Pretty much all teams at Microsoft now use the same bug tracking software above. One really nice benefit of this is that it means that bugs can be easily linked and assigned across different product teams.
What is the workflow we use when opening and fixing bugs?
Different teams within the company use different workflows when handling issues. My team owns three products (Visual Web Developer, ASP.NET and IIS) – and this workflow even differs slightly across these three feature teams.
At a high-level, though, the standard workflow is that anyone on the team who finds an issue creates and opens a bug themselves. This could be a tester, developer or program manager. As part of this process the finder enters detailed repro steps to allow anyone else to reproduce the problem themselves.
When it comes to assigning the issue to someone, they will either assign it to the developer/tester/pm who will handle it directly, or alternatively they will assign it to “Active”. “Active” means that they either do not know who should handle it, or are waiting for a lead or triage team to take a look at it, triage its priority, and re-assign it as appropriate to the right person to fix. Note that as we get closer to the end-game of a milestone or software cycle, we will require that all bugs get opened against “Active” so as to ensure that it always goes through a triage team to ensure that the right priority and calls on the issue get made.
Individual developers, testers and PMs on the team periodically refresh their own queries within product studio to see if any new issues have been assigned to them (they just run a query that filters by their name).
Bugs do sometimes bounce back and forth between a tester, developer and program manager. If there is a functionality question (example: what is the right error message to display?, should it really work like this?, are we missing something?, etc.), then the bug usually gets assigned to the PM to clarify the right behavior and/or update the spec, and then reassign back to a developer. If the bug can’t be reproduced, it will often be assigned back to the original owner for more data or resolved as “non-repro”.
All changes during a bug’s lifetime are automatically tracked by the bug system. There is a “history” tab that anyone can click to see when any field data was modified (and what the before/after value and the name of the person who changed it was for each step). This helps encourage the “right thing” to happen with bugs – and helps avoid cases where people sit on bugs forever and then at the last moment before a milestone date assigns it to someone else (note: this still does sometimes happen, but at least it is easy to spot because of the history log and timestamp).
When a bug is fixed on a developer’s local box, the developer making the change will mark the bug’s substatus as “fix available” and keep it assigned to themselves. Once they check-in the change to the team branch (which usually happens every other day or two after they’ve queued up a few fixes on their local machine – so as to optimize the 2-3 hours they’ll need to spend waiting for checkin suites to pass), they’ll resolve the bug (changing the status from “Active” to “Resolved”) and assign the bug back to the original bug opener or a tester who owns that area of the product.
The original bug opener or tester will then wait until an official build has come out that contains the bug fix. They will then walk through the repro steps again to make sure the bug has truly been fixed. If it has, they will “close” the bug by changing the status from “resolved” to “closed”. If the issue still exists, they will “re-activate” the bug by setting the status back to “active” and assigning it back to the developer.
As we get closer to a key milestone date like ZBB, we’ll monitor the number and activity on the bugs intensely. The nice thing is that because everything is stored in the central bug system, anyone on the team can do this from any machine – allowing fast and frequent communication to take place.
I was in fact running through my standard queries right before writing this blog post (that was what reminded me to write this entry). Looks like in the last 40 minutes since I started the post our ASP.NET Dev team bug count dropped from the 203 bugs in the screenshot above to 189 as of right now (and Visual Web Developer went from 93 to 86). Cool stuff.