Annotation of wikisrc/bugtracking.mdwn, revision 1.1
1.1 ! dholland 1: # Bugtracking in NetBSD
! 3: Currently NetBSD uses gnats for bugtracking.
! 4: Gnats is a horrible legacy tool.
! 5: There is a whole page of [[stuff we dislike about gnats]].
! 7: It has been clear for a long time (years) that we need to migrate to
! 8: some other bugtracker.
! 9: Various tools have been proposed, usually without much thought being
! 10: applied.
! 12: The way this usually works is that
! 13: the topic comes up and then a dozen people say "Let's use
! 14: $MY_FAVORITE_TOOL! It's great!"
! 15: Then a shouting match ensues and the people who call for requirements
! 16: analysis or even a list of criteria to pick one tool over another get
! 17: slagged for standing in the way of 'progress'.
! 18: These arguments inevitably take place with little understanding of
! 19: either the project's needs or the problems that a bugtracker needs to
! 20: handle for us.
! 21: Little or no information is generated, and nothing happens, except
! 22: that the particpants tend to get alienated and demotivated.
! 24: In order to avoid going around this barn again and again I'm creating
! 25: this page to try to document some of the genuine issues, as well as
! 26: conclusions that have been drawn in the past about requirements and
! 27: paths forward.
! 30: ## Problems with bugtracking in NetBSD
! 32: This section lists and discusses some of the challenges that arise
! 33: handling NetBSD's bug reports.
! 34: This is not meant to be a list of gripes about gnats -- there's
! 35: another page for that (above) -- but a list of things that are still
! 36: issues no matter what bugtracker we're using.
! 38: ### We already have a bug database.
! 40: Any plan for moving forward has to be able to import the existing bug
! 41: database without losing information.
! 42: This is basically not negotiable -- we cannot throw away the existing
! 43: bug data.
! 44: (The alternative to importing the existing data is to keep gnats
! 45: running indefinitely in parallel with a new system.
! 46: Besides being confusing, this does nothing to solve the problems with
! 47: gnats itself.)
! 49: ### The bug database is large.
! 51: NetBSD's existing bug database currently contains almost 48,000 bugs.
! 52: This is not especially large compared to other large projects
! 53: (consider the likely size of the Windows bug database, if you will...)
! 54: but it is large compared to _most projects_.
! 55: A lot of bugtrackers will creak, groan, and tip over if asked to track
! 56: this many bugs.
! 57: Or, they may be able to store and retrieve the bugs fine but the user
! 58: interface for handling them just fails to scale to the database size.
! 60: Many people who propose their favorite tool have used it for assorted
! 61: projects but never actually tried using it on a large bug database.
! 62: Some of these tools turn out to work ok on large databases, and others
! 63: don't.
! 65: There are also currently some 5400 open bug reports; many databases
! 66: that scale adequately to 50,000 bugs in the database turn out to not
! 67: be able to handle having so many of them open at once.
! 69: Another consequence of the database size is that the schema conversion
! 70: for any migration must be automatic.
! 71: It is not feasible to hand-edit or even review all bugs, or even all
! 72: open bugs, as part of a transition to a new system.
! 73: This creates problems for many otherwise decent choices for a new
! 74: bugtracker, as most bugtrackers (just like gnats itself) have their
! 75: own hardcoded assumptions about the schema and about things like what
! 76: states bugs can be in, and no two are the same.
! 78: ### The bug database is broad and not readily subdivided.
! 80: There is a wide range of software in NetBSD, and an even wider range
! 81: in pkgsrc, and we get bug reports on all of it.
! 82: There are plenty of identifiable units in this, such as specific
! 83: pkgsrc packages, but many bugs can't be linked directly to one of
! 84: these.
! 86: Furthermore, the existing database is only divided into broad
! 87: categories (kern, bin, pkg, etc.) and even these don't work all that
! 88: well sometimes.
! 89: (And the deployed base of send-pr scripts causes new PRs to come in
! 90: with only this much classification, something that can be changed only
! 91: slowly.)
! 93: The result of this is that it's hard to find things in the bug
! 94: database by looking around.
! 95: You can browse the database based on metadata (or you could if gnats
! 96: sucked less, currently it's hard) but we don't have the metadata
! 97: needed to do this effectively.
! 98: You can also search the database based on metadata (even gnats can do
! 99: this) but it doesn't really produce useful results for the same
! 100: reason.
! 101: This will remain true if we just switch to a different bugtracker;
! 102: to make progress on this problem we need more and better metadata.
! 104: Many large bug databases (CPAN's bug database was recently floated as
! 105: an example) can be clearly subdivided into individual projects or
! 106: subprojects and don't have this problem.
! 107: You can just look at bugs for the (sub)project you're interested in
! 108: and the number of those is manageable.
! 110: This problem is not unique to NetBSD (FreeBSD shares it, for example)
! 111: but as far as I know it's not common outside of OS projects because
! 112: most other projects are not broad in the same way.
! 114: ### Search doesn't work too well.
! 116: Because of the nature of the names of Unix entities (programs,
! 117: drivers, virtually everything), searching for them in a large text
! 118: corpus like the bug database doesn't work too well.
! 119: This problem is exacerbated if you're trying to find bugs filed
! 120: against programs that often appear incidentally in bug reports, like
! 121: make or sh.
! 123: This is not just a consequence of gnats issues; search won't work all
! 124: that well no matter what we do.
! 125: (Try typing "sh site:gnats.netbsd.org" into Google.
! 126: When Google can't do it, no bugtracker is going to do better.)
! 128: This means that text search really does not work as an alternative to
! 129: be able to find things by browsing or via metadata.
! 132: ## Some observations about the problems
! 134: The most basic problem we have is _finding stuff_.
! 135: Back when I first started tackling the bug database, I found that the
! 136: best way to make progress was not to search (either for text or
! 137: metadata) or to browse but to ask for a randomly selected open PR.
! 138: This basically constitutes a total failure of the bugtracker: it was
! 139: completely unable to provide useful information of any kind.
! 141: I (dholland) have since learned some tricks and have also accumulated
! 142: an external index for the database; this means I can get stuff out of
! 143: it now, at least sometimes, but most developers are in the position I
! 144: was then: the bug database is a completely useless black hole.
! 145: Several developers have recently said so; also we have the same
! 146: problem that FreeBSD observed in their database some time back, which
! 147: is that new bugs come in and get seen, and maybe they get fixed, but
! 148: if they don't get fixed fairly soon they get forgotten and hang around
! 149: indefinitely.
! 151: This is partly a consequence of gnats issues, and this is why gnats
! 152: must go.
! 153: However, as described above it isn't entirely because of gnats:
! 154: finding stuff by navigating (or searching) metadata is hard because we
! 155: don't have adequate metadata, and finding stuff by searching for text
! 156: is hard because it's a fundamentally hard text-retrieval problem.
! 158: Therefore, if we want to actually improve the situation, any migration
! 159: plan needs to include a way to get more metadata into the database.
! 160: This metadata will mostly need to be hand-applied; this is expensive
! 161: but not insurmountable (for 5400 open existing PRs) and not that big a
! 162: deal for incoming new PRs... provided the new bugtracker has adequate
! 163: support for arbitrary metadata, which many don't.
! 165: Note that in addition to the above analysis we also have some
! 166: supporting results.
! 167: Based on the analysis I (dholland) started maintaining an annotated
! 168: browseable index (aka the "buglists" pages) of the bug database.
! 169: This basically amounted to additional per-bug metadata of several
! 170: kinds, organized in a fashion that allowed generating an index as a
! 171: tree of web pages.
! 172: Unfortunately because it was a gimcrack thing only I could update it,
! 173: and unfortunately it also needed to be synchronized with the gnats
! 174: database by hand, with the result that when my available time dried up
! 175: it went out of date and is now pretty much useless.
! 177: However, while it existed people used it and it helped them.
! 178: Before we had it the number of open PRs had been steadily increasing
! 179: over time.
! 180: (The occasional hackathon brought down the count from time to time,
! 181: but never persistently.)
! 182: During the time we had it, the number of open PRs remained more or
! 183: less stable at around 4800-4900.
! 184: Now we don't have it again and we're up to 5400 open PRs.
! 185: The influx has not changed much since I got behind on it; if anything
! 186: it's dropped.
! 187: What this means is that the rate PRs are getting fixed has dropped,
! 188: and that's because it's become impossible to find anything again.
! 190: It seems to me that one of the chief things we want from a new
! 191: bugtracker is to be able to provide something like this browseable
! 192: index.
! 193: Therefore it must be able to support the kinds of metadata that the
! 194: buglists tree was using.
! 196: If we move to a bugtracker without this support, it may solve some of
! 197: the more glaring problems with gnats, but it isn't going to help us
! 198: _find_ stuff in the bug database and it isn't going to do anything to
! 199: help make the large backlog of unfixed bugs go away.
! 202: ## Metadata types
! 204: After working the bug database for some years and also after
! 205: maintaining the buglists pages, I've come to the conclusion that
! 206: we need the following _types_ of metadata:
! 208: * fields containing tags
! 209: * fields containing one of an enumerated list of choices
! 210: * fields containing a classification according to a hierarchical taxonomy
! 211: * fields containing free-form text
! 213: And also, importantly, we need arbitrarily many such fields, not a
! 214: fixed set concocted at the time the database is set up.
! 216: ### Tags
! 218: A tag field contains zero or more entries from a list of allowable
! 219: choices.
! 220: This is a well-understood concept and most bug databases support tags
! 221: in one way or another; however, I don't think most support arbitrarily
! 222: many different tags fields with their own sets of allowable tags.
! 224: Two questions immediately arise from this description: why do we need
! 225: to restrict tags to allowable values, and why do we need multiple tags
! 226: fields instead of just one?
! 228: The first question is easy: when you have a database of 50,000 things
! 229: you need to place some controls on what gets entered or you eventually
! 230: end up with trash.
! 231: This is just a fact of life with databases when they get big enough.
! 232: We have enough problems without having to deal with misspelled tags
! 233: and typos.
! 235: The second follows partly from the first and partly from ensuing human
! 236: interface concerns: if you just have one tags field, the number of
! 237: possible tag values grows without bound as more and more tags get
! 238: added, and before too long the list becomes itself hard to work with.
! 239: Grouping tags into logical sets (e.g. all releng tags for which bugs
! 240: are critical for which releases in one field) makes it much easier to
! 241: search for them, and also much easier to browse the database looking
! 242: for bugs that aren't tagged but should be.
! 244: Also this makes it possible to have developer-only tag fields, private
! 245: personal tags, and so forth, without undue complications.
! 247: ### Enumerations
! 249: An enumeration field contains exactly one entry from a list of
! 250: allowable choices.
! 251: This differs from a tag field in certain obvious ways.
! 252: We already have some of these in gnats, but they're hardcoded fields
! 253: rather than being instances of a general metadata type.
! 255: If we're going to have arbitrary metadata fields at all (rather than a
! 256: fixed set of predefined fields) we more or less need enumeration
! 257: fields to be able to migrate the existing database.
! 259: Also some things that one might abuse tags for if one only had tags
! 260: are perhaps better handled as enumerations; e.g. no bug should be both
! 261: "critical" for a release and also "would be nice" for the same
! 262: release.
! 264: That said, a bugtracker that only has tags fields is probably adequate
! 265: (though not entirely desirable) because one can abuse tags instead.
! 267: ### Hierarchical taxonomy
! 269: A hierarchical taxonomy is a scheme for identifying (and thus,
! 270: finding) things based on a nested series of choices.
! 271: The hierarchical taxonomy most people are most familiar with is
! 272: probably the scheme for species in biology.
! 273: That (especially in its more modern forms) is more complex than we
! 274: need but the basic principle is the same: at the top you have
! 275: everything, and then you pick one of several kinds of things and then
! 276: you're dealing with a restricted subset, and so forth until you get
! 277: down to a manageable number of things to look at at once that all have similar properties.
! 279: The buglists pages supported tags as well but were fundamentally based
! 280: on a hierarchical classification of bugs based on where in the system
! 281: they occur.
! 282: This (as noted above) has been extremely useful in practice.
! 284: There is another hierarchical taxonomy that I'd like to deploy, but
! 285: which I wasn't about to try to do without better tools: classifying
! 286: bugs based on their symptoms.
! 288: If we were going to pick just one of these metadata features that's
! 289: the most important, it would be this.
! 290: The problem is: most bugtrackers do not support hierarchical
! 291: taxonomies.
! 292: In fact, so far no existing bugtracker has been found to do so.
! 294: This is an extremely important point, because it means that we need to
! 295: find a way to do it.
! 296: This is going to involve writing code, either new code or an extension
! 297: to some bugtracker we otherwise like.
! 299: ### Free-form text
! 301: Free-form text fields have two uses: one is for enumerations where the
! 302: set of things being enumerated is too large to be manageable as an
! 303: enumeration (e.g. pkgsrc packages, or NetBSD version numbers, or
! 304: programs in /usr/bin) and the other is for text that we think text
! 305: retrieval tools will be able to process usefully.
! 307: I have no examples of the latter kind on hand but I expect some will
! 308: appear; there are several of the former kind that we definitely want
! 309: to be able to support.
! 310: One is the pkgsrc package (by pkgpath) a PR is about; not all pkgsrc
! 311: PRs apply to only one package, but most do.
! 312: Another (for base system PRs) is the name of the man page most closely
! 313: associated with what's broken.
! 314: This has been found (by me and also by FreeBSD) to be a useful way of
! 315: organizing things.
! 316: The version number field is probably another one.
! 318: ### Untagged vs. inappropriate
! 320: Note that the database needs to be able to distinguish between "this
! 321: metadata does not apply to this PR", as is the case for the pkgsrc
! 322: package field and a bug report on make, from "this PR has not been
! 323: tagged with this metadata yet", which in the near to medium term will
! 324: be the case for all new incoming PRs.
! 327: ## Some other points
! 329: The fact that send-pr comes with the system and doesn't require
! 330: signing up for anything (or anything other than a more-or-less working
! 331: mail configuration) has long been a strength of the project. This has
! 332: been cited many times by many people, and it's a feature we want to
! 333: retain. There is not, in practice, a problem with people dumping
! 334: useless PRs without valid return addresses; it happens occasionally
! 335: but not enough to worry about.
! 337: Subscribing to PRs (so you get notices of changes) is important; once
! 338: you find PRs you generally have to be able to follow them too.
! 339: This is something most bugtrackers other than gnats do ok, so it isn't
! 340: a big issue, but should nonetheless be noted.
! 342: Merging duplicate PRs is a nice feature but it's not critical; the
! 343: chief reason not being able to do it is annoying right now is that
! 344: gnats doesn't handle subscribing intelligently.
! 345: If you can crossreference the duplicates and everyone involved can
! 346: subscribe as needed, merging becomes less significant.
! 348: Keeping track of which PRs are blocking which other PRs is often cited
! 349: as a desirable or even critical feature in a bugtracker.
! 350: This is probably true in general, but for us it doesn't matter that
! 351: much: because the bug database (and the system) is broad, most bug
! 352: reports are independent of one another and blocking dependencies
! 353: rarely arise.
! 356: ## Conclusions on requirements
! 358: These are not set in stone but reflect my best estimate of the
! 359: situation and what does and doesn't matter.
! 360: This is weighted some towards backend issues, particularly in
! 361: connection with gnats.
! 363: Please don't edit this randomly; talk it over first.
! 365: Hard requirements
! 366: * Must be able to import the existing bug database.
! 367: * Doing so must not lose information.
! 368: * Must be able to accept incoming email from deployed send-pr scripts.
! 369: * Must be able to accept and file commit messages.
! 370: * It is not necessary to sign up to file a problem report.
! 371: * Nothing may be written in php.
! 373: Very strongly desired based on problem analysis:
! 374: * Support for arbitrary metadata fields not precooked in the database.
! 375: * Support for hierarchical taxonomies.
! 376: * Support for systems of tags.
! 377: * A decent workflow for retrieving incoming PRs and tagging them with
! 378: the desired new metadata.
! 380: Desired based on problem analysis:
! 381: * Support for enumerated metadata fields.
! 382: * Support for free-form text metadata fields.
! 384: Very strongly desired because we have existing workflows and habits:
! 385: * Command-line access (search, update, administer)
! 386: * Web access (search)
! 388: Desired because we have existing workflows and habits:
! 389: * Web access (update, maybe also administer)
! 391: Very strongly desired because we're tired of gnats:
! 392: * Proper handling of incoming MIME attachments.
! 393: * Some mechanism to prevent commit messages from accidentally
! 394: spamming the database.
! 395: * A way to file comments on a PR from a web browser.
! 396: * A web-based search form that works usefully.
! 397: * Crosslinks in the web interface to allow browsing.
! 398: * Command-line search that doesn't involve query-pr's nasty little
! 399: query "language".
! 400: * A nondegenerate way to subscribe to PRs, for both developers and
! 401: ordinary folks, at least by email and preferably also via RSS.
! 402: * A mail ingester that returns broken PR submissions instead of
! 403: filing sometimes-mangled versions for manual attention.
! 404: * A mail ingester that honors the confidential field of incoming PRs
! 405: properly.
! 406: * At least slightly automated handling of email bounces.
! 407: * A way to update email addresses without hand-editing a bajillion
! 408: PRs one at a time.
! 410: Desired because we're tired of gnats:
! 411: * A way to file comments on a PR directly from the command line.
! 412: * Something like a newsreader for working the bug database.
! 413: * Feedback nag mail that comes out such that replying directly to it
! 414: does something useful.
! 415: * A way to configure the contents of responsible nag mail to sort by
! 416: personal priority or other criteria.
! 417: * A way to turn off mail for bouncing addresses.
! 418: * A way to move misfiled comments from one PR to another.
! 420: Some other stuff that would be nice:
! 421: * Being able to vote PRs up and down from the web interface.
! 422: * A smartphone app for working the database.
! 424: Things that are less important:
! 425: * Merging multiple PRs on the same subject.
! 426: * Explicit crosslinks when one PR is blocking progress on another.
! 428: Things we don't care that much about:
! 429: * Padded cells for juvenile developers.
! 430: * Click-and-drool support for developers without basic clues.
! 433: ## The (old) plan
! 435: Given all the above, some years ago a plan was formulated and
! 436: even provisionally approved.
! 437: This has not materialized owing to (partly) a lack of time and
! 438: (partly) a near-total lack of response to requests for feedback or
! 439: input or assistance.
! 440: At this point some other plan may be better, but nothing has really
! 441: changed much in the meantime.
! 443: There are two key points in the material above:
! 444: * Schema conversion (to just about anything) without losing
! 445: information is going to be hard.
! 446: * Nothing that already exists off the shelf is going to handle the
! 447: most important thing we/I want anyway.
! 449: There is another point that is not obvious to those who haven't dealt
! 450: with gnats at length:
! 451: * gnats does very little.
! 453: Gnats contains a fair amount of code, but most of that code is storage
! 454: code (not user interface or analysis or other valuable material) and
! 455: it doesn't do a particularly good job of it.
! 456: If we moved the data to a real database, the amount of work needed to
! 457: replace the functionality of gnats is small: there are half a dozen or
! 458: so access programs that gnats comes with, of which the only one that
! 459: does anything nontrivial is edit-pr, and a half a dozen or so more
! 460: programs that we wrote that we can update as needed.
! 462: Therefore, the plan was to move the data to a real database, replace
! 463: the access programs, and deploy the results.
! 464: This was not expected to take long; it hasn't happened because even
! 465: the small amount of time required hasn't been available.
! 466: Just doing this much would not itself help a lot, but it would leave
! 467: us in a much better position for further improvements.
! 469: First, with the data moved to a real database and all the stuff
! 470: accessing it under our control, instead of being legacy gnats code
! 471: nobody wants to touch, we'd be in a position to adjust the schema:
! 472: make incremental changes with the end goal of working it into
! 473: something that can be imported safely into some other bugtracker,
! 474: make changes with the goal of improving the metadata, or whatever else
! 475: seemed useful.
! 477: Second, in the course of doing this we could eliminate the worst of
! 478: the problems with gnats.
! 479: And we'd be in a position to fix up more such problems as desired.
! 481: Third, we'd now have backend support for the buglists so it wouldn't
! 482: need hand-syncing (which has been expensive) and so other people could
! 483: help with the tagging.
! 484: This would free up more of my time to actually fix bugs (or do other
! 485: stuff) instead of administer.
! 487: The original plan was to build a new system (which got called
! 488: "swallowtail" after the Irish jig because swallows are insectivorous),
! 489: beat things into a better state than they ever could be with gnats,
! 490: and then take stock and decide whether to work on swallowtail or plan
! 491: a migration to something else.
! 493: In the course of trying to figure out which gnats bogosity was the
! 494: most critical to deal with, multiple versions of swallowtail got
! 495: planned out (more or less) and the possible subsequent migration part
! 496: got mostly forgotten, but that was always intended to be one of the
! 497: options on the table.
! 500: ## The (new) plan
! 502: Nothing has actually changed much since the old plan was formulated.
! 504: It is possible that accurate schema conversion is not going to be as
! 505: painful as we thought at the time; but this is unproven.
! 506: Importing the gnats data into postgres has been done but is pretty
! 507: easy; nobody has actually transformed the data into a schema some
! 508: existing bugtracker can use.
! 510: Nor has anybody seriously looked into adding hierarchical taxonomy
! 511: support to any existing bugtracker.
! 513: When/if that gets done, and if the results are positive, it might make
! 514: sense to forget about swallowtail and move directly to that system.
! 516: However, rushing to adopt something new without considering any of
! 517: this, which has been the apparent goal of recent argumentation, is
! 518: foolish.
CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb