While we now have mandoc for handling man pages, we currently still need groff in the tree to handle miscellaneous docs that are not man pages.

This is itself an inadequate solution as the groff we have does not support PDF output (which in this day and age is highly desirable) ... and while newer groff does support PDF output it does so via a Perl script. Also, importing a newer groff is problematic for assorted other reasons.

We need a way to typeset miscellaneous articles that we can import into base and that ideally is BSD licensed. (And that can produce PDFs.) Currently it looks like there are three decent ways forward:

These are all difficult and a lot of work, and in the case of new syntax are bound to cause a lot of shouting and stamping. Also, many of the miscellaneous documents use various roff preprocessors and it isn't clear how much of this mandoc can handle.

None of these options is particularly appealing.

There are also some less decent ways forward:

These options are even less appealing.

Maybe someone can think of a better idea. There are lots of choices if we give up on typeset output, but that doesn't seem like a good plan either.

Halibut might be worth including in the mix of options to consider. The source tarball is 919K, it's written in portable ANSI C (apart from requiring at least a 32-bit platform), it has no dependencies, and the license is MIT. It can directly generate plain ASCII text, HTML, PDF, PostScript, Unix man pages (i.e. nroff input to work with the -mandoc macro package), and Unix info.
Comment by J. Lewis Wednesday evening, February 4th, 2015

Thoughts from a passers-by:

Let the output formats (we need most!) dictate the requirements of the tool.
Let us end up with building a printing press. Let us forget a writer writes with a Fountain pen to concentrate on content. To getaway with distractions.
Instead let us provide him a letter case. To let him think about each single letter independently. Instead of thinking of the content.
If he has kept up till now, let us further distract him: Let him set the printing press to output almost any book format available.
Of course, none supports that letter combination he minutely composed earlier. Let him think about which format to choose or which letters to loose!

A writer needs a tool which concentrates his thought. A printing press expands thought.
What will the writer do if one gives the writer a printing press?

Why should the available book formats define the pen of the writer? 'Content rules format.'

I see a collection of printing presses. They can do a lot.
I read: They don't do the job 'software documentation'.
But what does this all have to do with solving the task 'software documentation'?
Nothing. One needs to specify what 'software documentation' is before choosing the tool.
To me it is:

I. Hierarchy
1. Headings
1. Lists
I. Descriptive text I. Non-descriptive text (quotes) 1. Inline 1. Multiline I. References
1. Internal 1. External 1. Author, Title, Date, Location 1. URI I. Images

Blocks of texts are organised in an hierarchy (a1).
Block of texts consist of descriptive (b) and non-descriptive elements (quotes) (c).
A descriptive element (b) explains stuff. A non-descriptive (c) is either input or output of the software. It may be inline (c1) or multiline (c2).
A descriptive element may contain lists (a2). It may contain references to (d1) the hierarchy itself or (d2) to external sources.
I leave it to some higher power whether images (e) are required.
The plain text should be (1) indented and (2) broken incorporating the hierarchy (a1) (Why is this not the editor's job? Consistency.).

That's it.

'But I need to highlight some description text.' No, you don't.
If it's important it should be reflected by the hierarchy anyway. Still, not satisfied? Then, look at your keyboard:
o r s p a c e.
O R B O T H.

You haven't seen anything. Continue your walk.

Comment by Benjamin late Tuesday evening, March 31st, 2015