NetBSD Wiki/ bugtracking

Bugtracking in NetBSD

Currently NetBSD uses gnats for bugtracking. Gnats is a horrible legacy tool. There is a whole page of stuff we dislike about gnats.

It has been clear for a long time (years) that we need to migrate to some other bugtracker. Various tools have been proposed, usually without much thought being applied.

The way this usually works is that the topic comes up and then a dozen people say "Let's use $MY_FAVORITE_TOOL! It's great!" Then a shouting match ensues and the people who call for requirements analysis or even a list of criteria to pick one tool over another get slagged for standing in the way of 'progress'. These arguments inevitably take place with little understanding of either the project's needs or the problems that a bugtracker needs to handle for us. Little or no information is generated, and nothing happens, except that the particpants tend to get alienated and demotivated.

In order to avoid going around this barn again and again I'm creating this page to try to document some of the genuine issues, as well as conclusions that have been drawn in the past about requirements and paths forward.

Problems with bugtracking in NetBSD

This section lists and discusses some of the challenges that arise handling NetBSD's bug reports. This is not meant to be a list of gripes about gnats -- there's another page for that (above) -- but a list of things that are still issues no matter what bugtracker we're using.

We already have a bug database.

Any plan for moving forward has to be able to import the existing bug database without losing information. This is basically not negotiable -- we cannot throw away the existing bug data. (The alternative to importing the existing data is to keep gnats running indefinitely in parallel with a new system. Besides being confusing, this does nothing to solve the problems with gnats itself.)

The bug database is large.

NetBSD's existing bug database currently contains almost 48,000 bugs. This is not especially large compared to other large projects (consider the likely size of the Windows bug database, if you will...) but it is large compared to most projects. A lot of bugtrackers will creak, groan, and tip over if asked to track this many bugs. Or, they may be able to store and retrieve the bugs fine but the user interface for handling them just fails to scale to the database size.

Many people who propose their favorite tool have used it for assorted projects but never actually tried using it on a large bug database. Some of these tools turn out to work ok on large databases, and others don't.

There are also currently some 5400 open bug reports; some bugtrackers that scale adequately to 50,000 bugs in the database turn out to not be able to handle having so many of them open at once.

Another consequence of the database size is that the schema conversion for any migration must be automatic. It is not feasible to hand-edit or even review all bugs, or even all open bugs, as part of a transition to a new system. This creates problems for many otherwise decent choices for a new bugtracker, as most bugtrackers (just like gnats itself) have their own hardcoded assumptions about the schema and about things like what states bugs can be in, and no two are the same.

The bug database is broad and not readily subdivided.

There is a wide range of software in NetBSD, and an even wider range in pkgsrc, and we get bug reports on all of it. There are plenty of identifiable units in this, such as specific pkgsrc packages, but many bugs can't be linked directly to one of these.

Furthermore, the existing database is only divided into broad categories (kern, bin, pkg, etc.) and even these don't work all that well sometimes. (And the deployed base of send-pr scripts causes new PRs to come in with only this much classification, something that can be changed only slowly.)

The result of this is that it's hard to find things in the bug database by looking around. You can browse the database based on metadata (or you could if gnats sucked less, currently it's hard) but we don't have the metadata needed to do this effectively. You can also search the database based on metadata (even gnats can do this) but it doesn't really produce useful results for the same reason. This will remain true if we just switch to a different bugtracker; to make progress on this problem we need more and better metadata.

Many large bug databases (CPAN's bug database was recently floated as an example) can be clearly subdivided into individual projects or subprojects and don't have this problem. You can just look at bugs for the (sub)project you're interested in and the number of those is manageable.

This problem is not unique to NetBSD (FreeBSD shares it, for example) but as far as I know it's not common outside of OS projects because most other projects are not broad in the same way.

Search doesn't work too well.

Because of the nature of the names of Unix entities (programs, drivers, virtually everything), searching for them in a large text corpus like the bug database doesn't work too well. This problem is exacerbated if you're trying to find bugs filed against programs that often appear incidentally in bug reports, like make or sh.

This is not just a consequence of gnats issues; search won't work all that well no matter what we do. (Try typing "sh site:gnats.netbsd.org" into Google. When Google can't do it, no bugtracker is going to do better.)

This means that text search really does not work as an alternative to be able to find things by browsing or via metadata.

Some observations about the problems

The most basic problem we have is finding stuff. Back when I first started tackling the bug database, I found that the best way to make progress was not to search (either for text or metadata) or to browse but to ask for a randomly selected open PR. This basically constitutes a total failure of the bugtracker: it was completely unable to provide useful information of any kind.

I (dholland) have since learned some tricks and have also accumulated an external index for the database; this means I can get stuff out of it now, at least sometimes, but most developers are in the position I was then: the bug database is a completely useless black hole. Several developers have recently said so; also we have the same problem that FreeBSD observed in their database some time back, which is that new bugs come in and get seen, and maybe they get fixed, but if they don't get fixed fairly soon they get forgotten and hang around indefinitely.

This is partly a consequence of gnats issues, and this is why gnats must go. However, as described above it isn't entirely because of gnats: finding stuff by navigating (or searching) metadata is hard because we don't have adequate metadata, and finding stuff by searching for text is hard because it's a fundamentally hard text-retrieval problem.

Therefore, if we want to actually improve the situation, any migration plan needs to include a way to get more metadata into the database. This metadata will mostly need to be hand-applied; this is expensive but not insurmountable (for 5400 open existing PRs) and not that big a deal for incoming new PRs... provided the new bugtracker has adequate support for arbitrary metadata, which many don't.

Note that in addition to the above analysis we also have some supporting results. Based on the analysis I (dholland) started maintaining an annotated browseable index (aka the "buglists" pages) of the bug database. This basically amounted to additional per-bug metadata of several kinds, organized in a fashion that allowed generating an index as a tree of web pages. Unfortunately because it was a gimcrack thing only I could update it, and unfortunately it also needed to be synchronized with the gnats database by hand, with the result that when my available time dried up it went out of date and is now pretty much useless.

However, while it existed people used it and it helped them. Before we had it the number of open PRs had been steadily increasing over time. (The occasional hackathon brought down the count from time to time, but never persistently.) During the time we had it, the number of open PRs remained more or less stable at around 4800-4900. Now we don't have it again and we're up to 5400 open PRs. The influx has not changed much since I got behind on it; if anything it's dropped. What this means is that the rate PRs are getting fixed has dropped, and that's because it's become impossible to find anything again.

It seems to me that one of the chief things we want from a new bugtracker is to be able to provide something like this browseable index. Therefore it must be able to support the kinds of metadata that the buglists tree was using.

If we move to a bugtracker without this support, it may solve some of the more glaring problems with gnats, but it isn't going to help us find stuff in the bug database and it isn't going to do anything to help make the large backlog of unfixed bugs go away.

Metadata types

After working the bug database for some years and also after maintaining the buglists pages, I've come to the conclusion that we need the following types of metadata:

And also, importantly, we need arbitrarily many such fields, not a fixed set concocted at the time the database is set up.

Tags

A tag field contains zero or more entries from a list of allowable choices. This is a well-understood concept and most bug databases support tags in one way or another; however, I don't think most support arbitrarily many different tags fields with their own sets of allowable tags.

Two questions immediately arise from this description: why do we need to restrict tags to allowable values, and why do we need multiple tags fields instead of just one?

The first question is easy: when you have a database of 50,000 things you need to place some controls on what gets entered or you eventually end up with trash. This is just a fact of life with databases when they get big enough. We have enough problems without having to deal with misspelled tags and typos.

The second follows partly from the first and partly from ensuing human interface concerns: if you just have one tags field, the number of possible tag values grows without bound as more and more tags get added, and before too long the list becomes itself hard to work with. Grouping tags into logical sets (e.g. all releng tags for which bugs are critical for which releases in one field) makes it much easier to search for them, and also much easier to browse the database looking for bugs that aren't tagged but should be.

Also this makes it possible to have developer-only tag fields, private personal tags, and so forth, without undue complications.

Enumerations

An enumeration field contains exactly one entry from a list of allowable choices. This differs from a tag field in certain obvious ways. We already have some of these in gnats, but they're hardcoded fields rather than being instances of a general metadata type.

If we're going to have arbitrary metadata fields at all (rather than a fixed set of predefined fields) we more or less need enumeration fields to be able to migrate the existing database.

Also some things that one might abuse tags for if one only had tags are perhaps better handled as enumerations; e.g. no bug should be both "critical" for a release and also "would be nice" for the same release.

That said, a bugtracker that only has tags fields is probably adequate (though not entirely desirable) because one can abuse tags instead.

Hierarchical taxonomy

A hierarchical taxonomy is a scheme for identifying (and thus, finding) things based on a nested series of choices. The hierarchical taxonomy most people are most familiar with is probably the scheme for species in biology. That (especially in its more modern forms) is more complex than we need but the basic principle is the same: at the top you have everything, and then you pick one of several kinds of things and then you're dealing with a restricted subset, and so forth until you get down to a manageable number of things to look at at once that all have similar properties.

The buglists pages supported tags as well but were fundamentally based on a hierarchical classification of bugs based on where in the system they occur. This (as noted above) has been extremely useful in practice.

There is another hierarchical taxonomy that I'd like to deploy, but which I wasn't about to try to do without better tools: classifying bugs based on their symptoms.

If we were going to pick just one of these metadata features that's the most important, it would be this. The problem is: most bugtrackers do not support hierarchical taxonomies. In fact, so far no existing bugtracker has been found to do so.

This is an extremely important point, because it means that we need to find a way to do it. This is going to involve writing code, either new code or an extension to some bugtracker we otherwise like.

Free-form text

Free-form text fields have two uses: one is for enumerations where the set of things being enumerated is too large to be manageable as an enumeration (e.g. pkgsrc packages, or NetBSD version numbers, or programs in /usr/bin) and the other is for text that we think text retrieval tools will be able to process usefully.

I have no examples of the latter kind on hand but I expect some will appear; there are several of the former kind that we definitely want to be able to support. One is the pkgsrc package (by pkgpath) a PR is about; not all pkgsrc PRs apply to only one package, but most do. Another (for base system PRs) is the name of the man page most closely associated with what's broken. This has been found (by me and also by FreeBSD) to be a useful way of organizing things. The version number field is probably another one.

Untagged vs. inappropriate

Note that the database needs to be able to distinguish between "this metadata does not apply to this PR", as is the case for the pkgsrc package field and a bug report on make, from "this PR has not been tagged with this metadata yet", which in the near to medium term will be the case for all new incoming PRs.

Some other points

The fact that send-pr comes with the system and doesn't require signing up for anything (or anything other than a more-or-less working mail configuration) has long been a strength of the project. This has been cited many times by many people, and it's a feature we want to retain. There is not, in practice, a problem with people dumping useless PRs without valid return addresses; it happens occasionally but not enough to worry about.

Subscribing to PRs (so you get notices of changes) is important; once you find PRs you generally have to be able to follow them too. This is something most bugtrackers other than gnats do ok, so it isn't a big issue, but should nonetheless be noted.

Merging duplicate PRs is a nice feature but it's not critical; the chief reason not being able to do it is annoying right now is that gnats doesn't handle subscribing intelligently. If you can crossreference the duplicates and everyone involved can subscribe as needed, merging becomes less significant.

Keeping track of which PRs are blocking which other PRs is often cited as a desirable or even critical feature in a bugtracker. This is probably true in general, but for us it doesn't matter that much: because the bug database (and the system) is broad, most bug reports are independent of one another and blocking dependencies rarely arise.

Conclusions on requirements

These are not set in stone but reflect my best estimate of the situation and what does and doesn't matter. This is weighted some towards backend issues, particularly in connection with gnats.

Please don't edit this randomly; talk it over first.

Hard requirements

Very strongly desired based on problem analysis:

Desired based on problem analysis:

Very strongly desired because we have existing workflows and habits:

Desired because we have existing workflows and habits:

Very strongly desired because we're tired of gnats:

Desired because we're tired of gnats:

Some other stuff that would be nice:

Things that are less important:

Things we don't care that much about:

The (old) plan

Given all the above, some years ago a plan was formulated and even provisionally approved. This has not materialized owing to (partly) a lack of time and (partly) a near-total lack of response to requests for feedback or input or assistance. At this point some other plan may be better, but nothing has really changed much in the meantime.

There are two key points in the material above:

There is another point that is not obvious to those who haven't dealt with gnats at length:

Gnats contains a fair amount of code, but most of that code is storage code (not user interface or analysis or other valuable material) and it doesn't do a particularly good job of it. If we moved the data to a real database, the amount of work needed to replace the functionality of gnats is small: there are half a dozen or so access programs that gnats comes with, of which the only one that does anything nontrivial is edit-pr, and a half a dozen or so more programs that we wrote that we can update as needed.

Therefore, the plan was to move the data to a real database, replace the access programs, and deploy the results. This was not expected to take long; it hasn't happened because even the small amount of time required hasn't been available. Just doing this much would not itself help a lot, but it would leave us in a much better position for further improvements.

First, with the data moved to a real database and all the stuff accessing it under our control, instead of being legacy gnats code nobody wants to touch, we'd be in a position to adjust the schema: make incremental changes with the end goal of working it into something that can be imported safely into some other bugtracker, make changes with the goal of improving the metadata, or whatever else seemed useful.

Second, in the course of doing this we could eliminate the worst of the problems with gnats. And we'd be in a position to fix up more such problems as desired.

Third, we'd now have backend support for the buglists so it wouldn't need hand-syncing (which has been expensive) and so other people could help with the tagging. This would free up more of my time to actually fix bugs (or do other stuff) instead of administer.

The original plan was to build a new system (which got called "swallowtail" after the Irish jig because swallows are insectivorous), beat things into a better state than they ever could be with gnats, and then take stock and decide whether to work on swallowtail or plan a migration to something else.

In the course of trying to figure out which gnats bogosity was the most critical to deal with, multiple versions of swallowtail got planned out (more or less) and the possible subsequent migration part got mostly forgotten, but that was always intended to be one of the options on the table.

The (new) plan

Nothing has actually changed much since the old plan was formulated.

It is possible that accurate schema conversion is not going to be as painful as we thought at the time; but this is unproven. Importing the gnats data into postgres has been done but is pretty easy; nobody has actually transformed the data into a schema some existing bugtracker can use.

Nor has anybody seriously looked into adding hierarchical taxonomy support to any existing bugtracker.

When/if that gets done, and if the results are positive, it might make sense to forget about swallowtail and move directly to that system.

However, rushing to adopt something new without considering any of this, which has been the apparent goal of recent argumentation, is foolish.

Add a comment
Last edited at lunch time on Thursday, June 26th, 2014
Preferences | Logout