# Bugtracking in NetBSD

Currently NetBSD uses gnats for bugtracking.
Gnats is a horrible legacy tool.
There is a whole page of [[stuff we dislike about gnats]].

It has been clear for a long time (years) that we need to migrate to
some other bugtracker.
Various tools have been proposed, usually without much thought being
applied.

The way this usually works is that
the topic comes up and then a dozen people say "Let's use
$MY_FAVORITE_TOOL! It's great!"
Then a shouting match ensues and the people who call for requirements
analysis or even a list of criteria to pick one tool over another get
slagged for standing in the way of 'progress'.
These arguments inevitably take place with little understanding of
either the project's needs or the problems that a bugtracker needs to
handle for us.
Little or no information is generated, and nothing happens, except
that the particpants tend to get alienated and demotivated.

In order to avoid going around this barn again and again I'm creating
this page to try to document some of the genuine issues, as well as
conclusions that have been drawn in the past about requirements and
paths forward.


## Problems with bugtracking in NetBSD

This section lists and discusses some of the challenges that arise
handling NetBSD's bug reports.
This is not meant to be a list of gripes about gnats -- there's
another page for that (above) -- but a list of things that are still
issues no matter what bugtracker we're using.

### We already have a bug database.

Any plan for moving forward has to be able to import the existing bug
database without losing information.
This is basically not negotiable -- we cannot throw away the existing
bug data.
(The alternative to importing the existing data is to keep gnats
running indefinitely in parallel with a new system.
Besides being confusing, this does nothing to solve the problems with
gnats itself.)

### The bug database is large.

NetBSD's existing bug database currently contains almost 48,000 bugs.
This is not especially large compared to other large projects
(consider the likely size of the Windows bug database, if you will...)
but it is large compared to _most projects_.
A lot of bugtrackers will creak, groan, and tip over if asked to track
this many bugs.
Or, they may be able to store and retrieve the bugs fine but the user
interface for handling them just fails to scale to the database size.

Many people who propose their favorite tool have used it for assorted
projects but never actually tried using it on a large bug database.
Some of these tools turn out to work ok on large databases, and others
don't.

There are also currently some 5400 open bug reports; some bugtrackers
that scale adequately to 50,000 bugs in the database turn out to not
be able to handle having so many of them open at once.

Another consequence of the database size is that the schema conversion
for any migration must be automatic.
It is not feasible to hand-edit or even review all bugs, or even all
open bugs, as part of a transition to a new system.
This creates problems for many otherwise decent choices for a new
bugtracker, as most bugtrackers (just like gnats itself) have their
own hardcoded assumptions about the schema and about things like what
states bugs can be in, and no two are the same.

### The bug database is broad and not readily subdivided.

There is a wide range of software in NetBSD, and an even wider range
in pkgsrc, and we get bug reports on all of it.
There are plenty of identifiable units in this, such as specific
pkgsrc packages, but many bugs can't be linked directly to one of
these.

Furthermore, the existing database is only divided into broad
categories (kern, bin, pkg, etc.) and even these don't work all that
well sometimes.
(And the deployed base of send-pr scripts causes new PRs to come in
with only this much classification, something that can be changed only
slowly.)

The result of this is that it's hard to find things in the bug
database by looking around.
You can browse the database based on metadata (or you could if gnats
sucked less, currently it's hard) but we don't have the metadata
needed to do this effectively.
You can also search the database based on metadata (even gnats can do
this) but it doesn't really produce useful results for the same
reason.
This will remain true if we just switch to a different bugtracker;
to make progress on this problem we need more and better metadata.

Many large bug databases (CPAN's bug database was recently floated as
an example) can be clearly subdivided into individual projects or
subprojects and don't have this problem.
You can just look at bugs for the (sub)project you're interested in
and the number of those is manageable.

This problem is not unique to NetBSD (FreeBSD shares it, for example)
but as far as I know it's not common outside of OS projects because
most other projects are not broad in the same way.

### Search doesn't work too well.

Because of the nature of the names of Unix entities (programs,
drivers, virtually everything), searching for them in a large text
corpus like the bug database doesn't work too well.
This problem is exacerbated if you're trying to find bugs filed
against programs that often appear incidentally in bug reports, like
make or sh.

This is not just a consequence of gnats issues; search won't work all
that well no matter what we do.
(Try typing "sh site:gnats.netbsd.org" into Google.
When Google can't do it, no bugtracker is going to do better.)

This means that text search really does not work as an alternative to
be able to find things by browsing or via metadata.


## Some observations about the problems

The most basic problem we have is _finding stuff_.
Back when I first started tackling the bug database, I found that the
best way to make progress was not to search (either for text or
metadata) or to browse but to ask for a randomly selected open PR.
This basically constitutes a total failure of the bugtracker: it was
completely unable to provide useful information of any kind.

I ([[dholland]]) have since learned some tricks and have also accumulated
an external index for the database; this means I can get stuff out of
it now, at least sometimes, but most developers are in the position I
was then: the bug database is a completely useless black hole.
Several developers have recently said so; also we have the same
problem that FreeBSD observed in their database some time back, which
is that new bugs come in and get seen, and maybe they get fixed, but
if they don't get fixed fairly soon they get forgotten and hang around
indefinitely.

This is partly a consequence of gnats issues, and this is why gnats
must go.
However, as described above it isn't entirely because of gnats:
finding stuff by navigating (or searching) metadata is hard because we
don't have adequate metadata, and finding stuff by searching for text
is hard because it's a fundamentally hard text-retrieval problem.

Therefore, if we want to actually improve the situation, any migration
plan needs to include a way to get more metadata into the database.
This metadata will mostly need to be hand-applied; this is expensive
but not insurmountable (for 5400 open existing PRs) and not that big a
deal for incoming new PRs... provided the new bugtracker has adequate
support for arbitrary metadata, which many don't.

Note that in addition to the above analysis we also have some
supporting results.
Based on the analysis I ([[dholland]]) started maintaining an annotated
browseable index (aka the "buglists" pages) of the bug database.
This basically amounted to additional per-bug metadata of several
kinds, organized in a fashion that allowed generating an index as a
tree of web pages.
Unfortunately because it was a gimcrack thing only I could update it,
and unfortunately it also needed to be synchronized with the gnats
database by hand, with the result that when my available time dried up
it went out of date and is now pretty much useless.

However, while it existed people used it and it helped them.
Before we had it the number of open PRs had been steadily increasing
over time.
(The occasional hackathon brought down the count from time to time,
but never persistently.)
During the time we had it, the number of open PRs remained more or
less stable at around 4800-4900.
Now we don't have it again and we're up to 5400 open PRs.
The influx has not changed much since I got behind on it; if anything
it's dropped.
What this means is that the rate PRs are getting fixed has dropped,
and that's because it's become impossible to find anything again.

It seems to me that one of the chief things we want from a new
bugtracker is to be able to provide something like this browseable
index.
Therefore it must be able to support the kinds of metadata that the
buglists tree was using.

If we move to a bugtracker without this support, it may solve some of
the more glaring problems with gnats, but it isn't going to help us
_find_ stuff in the bug database and it isn't going to do anything to
help make the large backlog of unfixed bugs go away.


## Metadata types

After working the bug database for some years and also after
maintaining the buglists pages, I've come to the conclusion that
we need the following _types_ of metadata:

* fields containing tags
* fields containing one of an enumerated list of choices
* fields containing a classification according to a hierarchical taxonomy
* fields containing free-form text

And also, importantly, we need arbitrarily many such fields, not a
fixed set concocted at the time the database is set up.

### Tags

A tag field contains zero or more entries from a list of allowable
choices.
This is a well-understood concept and most bug databases support tags
in one way or another; however, I don't think most support arbitrarily
many different tags fields with their own sets of allowable tags.

Two questions immediately arise from this description: why do we need
to restrict tags to allowable values, and why do we need multiple tags
fields instead of just one?

The first question is easy: when you have a database of 50,000 things
you need to place some controls on what gets entered or you eventually
end up with trash.
This is just a fact of life with databases when they get big enough.
We have enough problems without having to deal with misspelled tags
and typos.

The second follows partly from the first and partly from ensuing human
interface concerns: if you just have one tags field, the number of
possible tag values grows without bound as more and more tags get
added, and before too long the list becomes itself hard to work with.
Grouping tags into logical sets (e.g. all releng tags for which bugs
are critical for which releases in one field) makes it much easier to
search for them, and also much easier to browse the database looking
for bugs that aren't tagged but should be.

Also this makes it possible to have developer-only tag fields, private
personal tags, and so forth, without undue complications.

### Enumerations

An enumeration field contains exactly one entry from a list of
allowable choices.
This differs from a tag field in certain obvious ways.
We already have some of these in gnats, but they're hardcoded fields 
rather than being instances of a general metadata type.

If we're going to have arbitrary metadata fields at all (rather than a
fixed set of predefined fields) we more or less need enumeration
fields to be able to migrate the existing database.

Also some things that one might abuse tags for if one only had tags
are perhaps better handled as enumerations; e.g. no bug should be both
"critical" for a release and also "would be nice" for the same
release.

That said, a bugtracker that only has tags fields is probably adequate
(though not entirely desirable) because one can abuse tags instead.

### Hierarchical taxonomy

A hierarchical taxonomy is a scheme for identifying (and thus,
finding) things based on a nested series of choices.
The hierarchical taxonomy most people are most familiar with is
probably the scheme for species in biology.
That (especially in its more modern forms) is more complex than we
need but the basic principle is the same: at the top you have
everything, and then you pick one of several kinds of things and then
you're dealing with a restricted subset, and so forth until you get
down to a manageable number of things to look at at once that all have similar properties.

The buglists pages supported tags as well but were fundamentally based
on a hierarchical classification of bugs based on where in the system
they occur.
This (as noted above) has been extremely useful in practice.

There is another hierarchical taxonomy that I'd like to deploy, but
which I wasn't about to try to do without better tools: classifying
bugs based on their symptoms.

If we were going to pick just one of these metadata features that's
the most important, it would be this.
The problem is: most bugtrackers do not support hierarchical
taxonomies.
In fact, so far no existing bugtracker has been found to do so.

This is an extremely important point, because it means that we need to
find a way to do it.
This is going to involve writing code, either new code or an extension
to some bugtracker we otherwise like.

### Free-form text

Free-form text fields have two uses: one is for enumerations where the
set of things being enumerated is too large to be manageable as an
enumeration (e.g. pkgsrc packages, or NetBSD version numbers, or
programs in /usr/bin) and the other is for text that we think text
retrieval tools will be able to process usefully.

I have no examples of the latter kind on hand but I expect some will
appear; there are several of the former kind that we definitely want
to be able to support.
One is the pkgsrc package (by pkgpath) a PR is about; not all pkgsrc
PRs apply to only one package, but most do.
Another (for base system PRs) is the name of the man page most closely
associated with what's broken.
This has been found (by me and also by FreeBSD) to be a useful way of
organizing things.
The version number field is probably another one.

### Untagged vs. inappropriate

Note that the database needs to be able to distinguish between "this
metadata does not apply to this PR", as is the case for the pkgsrc
package field and a bug report on make, from "this PR has not been
tagged with this metadata yet", which in the near to medium term will
be the case for all new incoming PRs.


## Some other points

The fact that send-pr comes with the system and doesn't require
signing up for anything (or anything other than a more-or-less working
mail configuration) has long been a strength of the project.  This has
been cited many times by many people, and it's a feature we want to
retain.  There is not, in practice, a problem with people dumping
useless PRs without valid return addresses; it happens occasionally
but not enough to worry about.

Subscribing to PRs (so you get notices of changes) is important; once
you find PRs you generally have to be able to follow them too.
This is something most bugtrackers other than gnats do ok, so it isn't
a big issue, but should nonetheless be noted.

Merging duplicate PRs is a nice feature but it's not critical; the
chief reason not being able to do it is annoying right now is that
gnats doesn't handle subscribing intelligently.
If you can crossreference the duplicates and everyone involved can
subscribe as needed, merging becomes less significant.

Keeping track of which PRs are blocking which other PRs is often cited
as a desirable or even critical feature in a bugtracker.
This is probably true in general, but for us it doesn't matter that
much: because the bug database (and the system) is broad, most bug
reports are independent of one another and blocking dependencies
rarely arise.


## Conclusions on requirements

These are not set in stone but reflect my best estimate of the
situation and what does and doesn't matter.
This is weighted some towards backend issues, particularly in
connection with gnats.

Please don't edit this randomly; talk it over first.

Hard requirements

* Must be able to import the existing bug database.
* Doing so must not lose information.
* Must be able to accept incoming email from deployed send-pr scripts.
* Must handle confidential PRs in a way that does not make them
accessible to non-NetBSD people.
* Must be able to accept and file commit messages.
* It is not necessary to sign up to file a problem report.
* Nothing may be written in php.

Very strongly desired based on problem analysis:

* Support for arbitrary metadata fields not precooked in the database.
* Support for hierarchical taxonomies.
* Support for systems of tags.
* A decent workflow for retrieving incoming PRs and tagging them with
the desired new metadata.
* Support for free-form text metadata fields (for pkgsrc)

Desired based on problem analysis:

* Support for enumerated metadata fields.

Very strongly desired because we have existing workflows and habits:

* Command-line access (search, update, administer)
* Web access (search)

Desired because we have existing workflows and habits:

* Web access (update, maybe also administer)

Very strongly desired because we're tired of gnats:

* Proper handling of incoming MIME attachments.
* Some mechanism to prevent commit messages from accidentally
spamming the database.
* A way to file comments on a PR from a web browser.
* A web-based search form that works usefully.
* Crosslinks in the web interface to allow browsing.
* Command-line search that doesn't involve query-pr's nasty little
query "language".
* A nondegenerate way to subscribe to PRs, for both developers and
ordinary folks, at least by email and preferably also via RSS.
* A mail ingester that returns broken PR submissions instead of
filing sometimes-mangled versions for manual attention.
* A mail ingester that honors the confidential field of incoming PRs
properly.
* At least slightly automated handling of email bounces.
* A way to update email addresses without hand-editing a bajillion
PRs one at a time.

Desired because we're tired of gnats:

* A way to file comments on a PR directly from the command line.
* Something like a newsreader for working the bug database.
* Feedback nag mail that comes out such that replying directly to it
does something useful.
* A way to configure the contents of responsible nag mail to sort by
personal priority or other criteria.
* A way to turn off mail for bouncing addresses.
* A way to move misfiled comments from one PR to another.
* A way to mark bugs not merely as "closed" but one of fixed,
  invalid, obsolete, or "won't fix".

Some other stuff that would be nice:

* Being able to vote PRs up and down from the web interface.
* A smartphone app for working the database.

Things that are less important:

* Merging multiple PRs on the same subject.
* Explicit crosslinks when one PR is blocking progress on another.

Things we don't care that much about:

* Padded cells for juvenile developers.
* Click-and-drool support for developers without basic clues.


## The (old) plan

Given all the above, some years ago a plan was formulated and
even provisionally approved.
This has not materialized owing to (partly) a lack of time and
(partly) a near-total lack of response to requests for feedback or
input or assistance.
At this point some other plan may be better, but nothing has really
changed much in the meantime.

There are two key points in the material above:

* Schema conversion (to just about anything) without losing
information is going to be hard.
* Nothing that already exists off the shelf is going to handle the
most important thing we/I want anyway.

There is another point that is not obvious to those who haven't dealt
with gnats at length:

* gnats does very little.

Gnats contains a fair amount of code, but most of that code is storage
code (not user interface or analysis or other valuable material) and
it doesn't do a particularly good job of it.
If we moved the data to a real database, the amount of work needed to
replace the functionality of gnats is small: there are half a dozen or
so access programs that gnats comes with, of which the only one that
does anything nontrivial is edit-pr, and a half a dozen or so more
programs that we wrote that we can update as needed.

Therefore, the plan was to move the data to a real database, replace
the access programs, and deploy the results.
This was not expected to take long; it hasn't happened because even
the small amount of time required hasn't been available.
Just doing this much would not itself help a lot, but it would leave
us in a much better position for further improvements.

First, with the data moved to a real database and all the stuff
accessing it under our control, instead of being legacy gnats code
nobody wants to touch, we'd be in a position to adjust the schema:
make incremental changes with the end goal of working it into
something that can be imported safely into some other bugtracker,
make changes with the goal of improving the metadata, or whatever else
seemed useful.

Second, in the course of doing this we could eliminate the worst of
the problems with gnats.
And we'd be in a position to fix up more such problems as desired.

Third, we'd now have backend support for the buglists so it wouldn't
need hand-syncing (which has been expensive) and so other people could
help with the tagging.
This would free up more of my time to actually fix bugs (or do other
stuff) instead of administer.

The original plan was to build a new system (which got called
"swallowtail" after the Irish jig because swallows are insectivorous),
beat things into a better state than they ever could be with gnats,
and then take stock and decide whether to work on swallowtail or plan
a migration to something else.

In the course of trying to figure out which gnats bogosity was the
most critical to deal with, multiple versions of swallowtail got
planned out (more or less) and the possible subsequent migration part
got mostly forgotten, but that was always intended to be one of the
options on the table.


## The (new) plan

Nothing has actually changed much since the old plan was formulated.

It is possible that accurate schema conversion is not going to be as
painful as we thought at the time; but this is unproven.
Importing the gnats data into postgres has been done but is pretty
easy; nobody has actually transformed the data into a schema some
existing bugtracker can use.

Nor has anybody seriously looked into adding hierarchical taxonomy
support to any existing bugtracker.

When/if that gets done, and if the results are positive, it might make
sense to forget about swallowtail and move directly to that system.

However, rushing to adopt something new without considering any of
this, which has been the apparent goal of recent argumentation, is
foolish.