Return to gcc.mdwn CVS log

Up to [NetBSD Developer Wiki] / wikisrc / pkgsrc

Add discussion of gcc version selection

Using gcc in pkgsrc

On many systems pkgsrc supports, gcc is the standard compiler.  In
general, different versions of each OS have different gcc versions,
and some packages require newer GCC versions, in order to support
newer language standards (e.g. c++11, written in the style of
USE_LANGUAGES), or because older versions don't work (infrequently).

This page discusses issues related to version selection, and intends
to be a design document for how pkgsrc should address this problem, to
be converted into historical design rationale once implemented.  It
freely takes content from extensive mailinglist discussions, and
attempts to follow the rough consensus that has emerged.

## Base system gcc vs pkgsrc gcc

Systems using gcc (e.g. NetBSD) have a compiler as /usr/bin/gcc, and
this is usable by pkgsrc without any bootstrapping activity.  One can
build gcc versions (typically newer versions) from pkgsrc, resulting
in a compiler within ${PREFIX}, e.g. /usr/pkg/gcc6/bin/gcc.  This
compiler can then be used to compile other packages.

Issues with using base system gcc are typically that it is too old,
such as gcc 4.5 with NetBSD 6, which cannot compile c++11.

Issues when using pkgsrc gcc are that

  - it must be bootstrapped, requiring compiling a number of packages
    with the system compiler
  - C++ packages that are linked together should be built with the
    same compiler, because the standard library ABI is not necessarily
    the same for each compiler version
  - While C packages can be built with mixed versions, the binary
    should be linked with the higher version because the support
    library is backwards compatible but not forward compatible.

## Specific constraints and requirements

This section attempts to gather all the requirements.

  - By default, pkgsrc should be able to build working packages, even
    for packages that need a newer compiler than that provided in the
    base system.

  - The set of packages that are needed when building a bootstrap
    compiler should be minimized.

  - All packages that use C++ should be built with the same compiler version.

  - All packages that use C should have final linking with the highest
    version used in any included library.

  - pkgsrc should avoid building gcc unless it is more or less
    necessary to build packges.  (As an example, if the base system
    gcc can build c99 but not c++11, building a c99-only program
    should not trigger building a gcc version adequate for c++11.)

  - The compiler selection logic should work on NetBSD 6, and in-use
    (including LTS) GNU/Linux systems.  It is desirable for this logic
    to work on NetBSD 5.

  - The compiler selection logic should be understandable and not brittle.

## Design

The above requirements could in theory be satisfied in many ways, but
most of them are too complicated.

  - Packages declare what languages they need, with c++, c++11, and
    c++14 being expressed differently.

  - The package-settable variable GCC_REQD will be used only when a
    compiler that generally can compile the declared language version
    is insufficient.  These cases are expected to be relatively rare.

  - A user-settable variable PKGSRC_GCC_VERSION will declare the
    version of gcc to be used for C programs, with an OS- and
    version--specific default.

  - A user-settable variable PKGSRC_GXX_VERSION will declare the version of gcc to
    be used for all C++ programs, again with an OS- and
    version-specific default.  It must be at least PKGSRC_GCC_VERSION.

  - Each of c99, c++, c++11, and c++14 will be associated with a
    minimum gcc version, such that almost all programs declaring that
    language can be built with that version.  (This avoids issues of
    strict compliance with c++11, which requires a far higher version
    of gcc than the version required to compile almost all actual
    programs in c++11.)

  - The minimum version inferred from the language tag will be
    combined with any GCC_REQD declarations to find a minimum version
    for a specific package.  If that is greater than
    PKGSRC_GCC_VERSION (programs using only C) or PKGSRC_GXX_VERSION,
    package building will fail.  We call the resulting
    PKGSRC_GCC_VERSION or PKGSRC_GXX_VERSION the chosen version.

  - When building a program using C or C++, the chosen version is not
    provided by the base system, and the chosen version is not
    installed via pkgsrc, then it (and its dependencies) will be built
    from pkgsrc in a special bootstrap mode.  When building in
    bootstrap mode, the version selection logic is ignored and the
    base system compiler is used.  Consistency and reproducible builds
    require that a package built with the normal prefix must be the
    same whether built because of compiler bootstrapping or normal
    use.

    There are thus two choices for dealing with bootstrapping.  One is
    to use a distinct prefix, which will ensure that all packages that
    are part of the compiler bootstrap will not be linked into normal
    pkgsrc programs.  This implies that any dependencies of gcc may
    exist twice, once in bootstrap mode and once if built normally.  A
    gcc version itself will be built twice, if it is desired for
    regular use.  This double building and the complexity of a second
    prefix are the negatives of this approach.

    The other choice is to mark gcc and all depending packages as used
    for compiler bootstrapping, and to always build those with the
    base compiler.  We use the package-settable variable
    PKGSRC_GCC_BOOTSTRAP=yes to denote this.  The negative with this
    approach is possible inconsistency with gcc's dependencies being
    built with the base compiler and used later.

  - We expect that any program containing C++ will undergo final
    linking with a C++ compiler.  This is not a change from the
    current situation.

## Remaining issues

### gcc dependencies

Because gcc can have dependencies, there could be packages built with
the system compiler that are then later used with the chosen version.
For now, we defer worrying about these problems (judging that they
will be less serious than the current situation where all c++11
programs fail to build on NetBSD 6).

\todo: Analyze what build-time and install-time dependencies actually
exist.

\todo: Discuss adjusting options to minimize dependencies, including
gcc-inplace-math and nls.

### Default versions for various systems

Note that if any particular system (or bulk build), a newer gcc has to
be built, it does not hurt incrementally to have built it earlier.

When the base system is old (e.g., gcc 4.5 in NetBSD 6, or 4.1, in
NetBSD 5), then it is clear that a newer version must be built.  For
these, PKGSRC_GXX_VERSION should default to a newish gcc, avoiding
being so new as to cause building issues.  Currently, gcc6 is probably
a good choice.  PKGSRC_GCC_VERSION should probably default to the
system version if it can build C99, or match PKGSRC_GXX_VERSION, if
the system version is too old.  Perhaps gcc 4.5 would be used, but 4.1
not used.  \todo Discuss.

When the base system is almost new enough, the decision about the
default is more complicated.  A key example is gcc 4.8, found in
NetBSD 7.  Firefox requires gcc 4.9 (\todo because the c++11 support
in 4.8 is not quite good enough), and all programs using c++14 also
need a newer version.  One options is to choose 4.8, resulting in
firefox failing, as well as all c++14 programs.  Another is to choose
4.9, but this makes little sense because c++14 programs will still
fail, and the general rule of moving to the most recent
generally-acceptable version applies, which currently leads to gcc6.
This is in effect a declaration that "almost new enough" does not
count as new enough.  Thus the plan for NetBSD 7 is to set
PKGSRC_GCC_VERSION to 4.8 and PKGSRC_GXX_VERSION to 6.

When the base system is new, e.g. gcc 5 or gcc 6 it should simply be
used.  By "new enough", we mean that almost no programs in pkgsrc fail
to build with it, which implies that it supports (almost all) C++14
programs.   Our current definiton of new enough is gcc 5.

### Fortran

Fortran support is currently somewhat troubled..  It seems obvious to
extend to PGKSRC_GFORTRAN_VERSION, and have that match
PKGSRC_GCC_VERSION or PKGSRC_GXX_VERSION, but the Fortran situation is
not worsened by the above design.  \todo Discuss.

## Path forward

 - Modify all gcc packages to have minimal dependencies, and to add
   PKGSRC_GCC_BOOTSTRAP.

 - Modify the compiler selection logic to do nothing if
   PKGSRC_GCC_BOOTSTRAP is set.

 - Modify the compiler selection logic for LANGUAGES= to fail if
   PKGSRC_GCC_VERSION/PKGSRC_GXX_VERSION is not new enough.

 - Modify the compiler selection logic for GCC_REQD to fail if the
   version of GCC/GXX is not new enough.

 - Decide on defaults.  The straw proposal is that PKGSRC_GCC_VERSION
   is the base system version if >= 4.5 (or 4.4?), and otherwise 6,
   and that PKGSRC_GXX_VERSION is the base system version if >= 5, and
   otherwise 6.