Return to gcc.mdwn CVS log

Up to [NetBSD Developer Wiki] / wikisrc / pkgsrc

Adjust straw proposal

On many systems pkgsrc supports, gcc is the standard compiler.  In
general, different versions of each OS have different gcc versions,
and some packages require newer gcc versions, in order to support
newer language standards (e.g. c++11, written in the style of
USE_LANGUAGES), or because older versions don't work (infrequently).

This page discusses issues related to version selection, and intends
to be a design document for how pkgsrc should address this problem, to
be converted into historical design rationale once implemented.  It
freely takes content from extensive mailinglist discussions, and
attempts to follow the rough consensus that has emerged.

## Base system gcc vs pkgsrc gcc

Systems using gcc (e.g. NetBSD) have a compiler as /usr/bin/gcc, and
this is usable by pkgsrc without any bootstrapping activity.  One can
build gcc versions (typically newer versions) from pkgsrc, resulting
in a compiler within ${PREFIX}, e.g. /usr/pkg/gcc6/bin/gcc.  This
compiler can then be used to compile other packages.

The Issue with using base system gcc is typically that it is too old,
such as gcc 4.5 with NetBSD 6, which cannot compile c++11.

Issues when using pkgsrc gcc are that

  - it must be bootstrapped, requiring compiling a number of packages
    with the system compiler
  - C++ packages that are linked together should be built with the
    same compiler, because the standard library ABI is not necessarily
    the same for each compiler version
  - While C packages can be built with mixed versions, the binary
    should be linked with the higher version because the support
    library is backwards compatible but not forward compatible.

## Specific constraints and requirements

This section attempts to gather all the requirements.

  - By default, pkgsrc should be able to build working packages, even
    for packages that need a newer compiler than that provided in the
    base system.

  - The set of packages that are needed when building a bootstrap
    compiler should be minimized.

  - All packages that use C should have final linking with the highest
    version used in any included library.

  - All packages that use C++ should be built with the same compiler
    version.  Because these in the general case may include C, the
    version used for C++ must be at least as new as the version used
    for any used C package.

  - pkgsrc should avoid building gcc unless it is more or less
    necessary to build packges.  (As an example, if the base system
    gcc can build c99 but not c++11, building a c99-only program
    should not trigger building a gcc version adequate for c++11.)

  - The compiler selection logic should work on NetBSD 6 and newer,
    and other systems currently supported by pkgsrc, including in-use
    LTS GNU/Linux systems.  It should work on systems that default to
    clang, when set to use GCC, at least as well as the current
    scheme.  It is desirable for this logic to work on NetBSD 5.

  - The compiler selection logic should be understandable and not brittle.

## Design

The above requirements could in theory be satisfied in many ways, but
most of them are too complicated.  We present a design that aims to be
sound while mimimizing complexity.

  - Packages declare what languages they need, with c++, c++11, and
    c++14 being expressed differently.  (This is exactly current
    practice and just noted for completeness.)

  - The package-settable variable GCC_REQD will be used only when a
    compiler that generally can compile the declared language version
    is insufficient.  These cases are expected to be relatively rare;
    an example is firefox that is in c++ (but not c+11) and needs gcc
    4.9.

  - A user-settable variable PKGSRC_GCC_VERSION will declare the
    version of gcc to be used for C programs, with an OS- and
    version--specific default.

  - A user-settable variable PKGSRC_GXX_VERSION will declare the version of gcc to
    be used for all C++ programs, again with an OS- and
    version-specific default.  It must be at least PKGSRC_GCC_VERSION.

  - Each of c99, c++, c++11, and c++14 will be associated with a
    minimum gcc version, such that almost all programs declaring that
    language can be built with that version.  (This avoids issues of
    strict compliance with c++11, which requires a far higher version
    of gcc than the version required to compile almost all actual
    programs in c++11.)

  - The minimum version inferred from the language tag will be
    combined with any GCC_REQD declarations to find a minimum version
    for a specific package.  If that is greater than
    PKGSRC_GCC_VERSION (programs using only C) or PKGSRC_GXX_VERSION,
    package building will fail.  We call the resulting
    PKGSRC_GCC_VERSION or PKGSRC_GXX_VERSION the chosen version.

  - When building a program using C or C++, if the chosen version is
    not provided by the base system, and the chosen version is not
    installed via pkgsrc, then it (and its dependencies) will be built
    from pkgsrc in a special bootstrap mode.  When building in
    bootstrap mode, the version selection logic is ignored and the
    base system compiler is used.  Consistency and reproducible builds
    require that a package built with the normal prefix must be the
    same whether built because of compiler bootstrapping or normal
    use.

    There are thus two choices for dealing with bootstrapping.  One is
    to use a distinct prefix, which will ensure that all packages that
    are part of the compiler bootstrap will not be linked into normal
    pkgsrc programs.  This implies that any dependencies of gcc may
    exist twice, once in bootstrap mode and once if built normally.  A
    gcc version itself will be built twice, if it is desired for
    regular use.  This double building and the complexity of a second
    prefix are the negatives of this approach.

    The other choice is to mark gcc and all depending packages as used
    for compiler bootstrapping, and to always build those with the
    base compiler.  We use the package-settable variable
    PKGSRC_GCC_BOOTSTRAP=yes to denote this.  The negative with this
    approach is possible inconsistency with gcc's dependencies being
    built with the base compiler and used later.

  - We hope that the chosen version can be built using the base system
    version, and hope to avoid multi-stage bootstrapping.

  - We expect that any program containing C++ will undergo final
    linking with a C++ compiler.  This is not a change from the
    current situation.

## Remaining issues

### gcc dependencies

Because gcc can have dependencies, there could be packages built with
the system compiler that are then later used with the chosen version.
For now, we defer worrying about these problems (judging that they
will be less serious than the current situation where all c++11
programs fail to build on NetBSD 6).

\todo: Analyze what build-time and install-time dependencies actually
exist.

\todo: Discuss adjusting options to minimize dependencies, including
gcc-inplace-math and nls.

### Differing GCC and GXX versions

Perhaps it is a mistake to allow the chosen GCC and GXX versions to
differ.  If we require them to be the same, then essentially all
systems with a base system compiler older than gcc 5 will have to
bootstrap the compiler.  For now, we allow them to differ and will
permit the defaults to differ.

### Default versions for various systems

Note that if for any particular system's set of installed packages (or
bulk build), a newer gcc has to be built, it does not hurt to have
built it earlier.

When the base system is old (e.g., gcc 4.5 in NetBSD 6, or 4.1, in
NetBSD 5), then it is clear that a newer version must be built.  For
these, PKGSRC_GXX_VERSION should default to a newish gcc, avoiding
being so new as to cause building issues.  Currently, gcc5 is probably
a good choice, with gcc6 compiling significantly but not vastly fewer
packages.  PKGSRC_GCC_VERSION should probably default to the system
version if it can build all C99 programs, or match PKGSRC_GXX_VERSION,
if the system version is too old.  Perhaps gcc 4.5 would be used, but
4.1 not used.  \todo Discuss.

When the base system is almost new enough, the decision about the
default is more complicated.  A key example is gcc 4.8, found in
NetBSD 7.  Firefox requires gcc 4.9, and all programs using c++14 also
need a newer version.  One options is to choose 4.8, resulting in
firefox failing, as well as all c++14 programs.  Another is to choose
4.9, but this makes little sense because c++14 programs will still
fail, and the general rule of moving to the most recent
generally-acceptable version applies, which currently leads to gcc6.
This is in effect a declaration that "almost new enough" does not
count as new enough.  Thus the plan for NetBSD 7 is to set
PKGSRC_GCC_VERSION to 4.8 and PKGSRC_GXX_VERSION to 5.

When the base system is new enough, e.g. gcc 5, 6 or 7 it should
simply be used.  By "new enough", we mean that almost no programs in
pkgsrc fail to build with it, which implies that it supports (almost
all) C++14 programs.  Our current definiton of new enough is gcc 5.

### Limited mixed versions

One approach would be to allow limited mixed versions, where
individual programs could force a specific version to be bootstrapped
and used, so that e.g. firefox could use 4.9 even though most programs
use 4.8, which is what happens now on NetBSD 7.  This would rely on
being able to link c++ with 4.9 including some things built with 4.8
(which is done presently).  However, this approach would become
unsound with a library rather than an end program.  We reject this as
too much complexity for avoiding building a newer compiler in limited
situations.

### Fortran

Fortran support is currently somewhat troubled..  It seems obvious to
extend to PGKSRC_GFORTRAN_VERSION, and have that match
PKGSRC_GCC_VERSION or PKGSRC_GXX_VERSION, but the Fortran situation is
not worsened by the above design.  \todo Discuss.

## Path forward

 - Modify all gcc packages to have minimal dependencies, and to add
   PKGSRC_GCC_BOOTSTRAP.

 - Modify the compiler selection logic to do nothing if
   PKGSRC_GCC_BOOTSTRAP is set.

 - Modify the compiler selection logic for LANGUAGES= to fail if
   PKGSRC_GCC_VERSION/PKGSRC_GXX_VERSION is not new enough.

 - Modify the compiler selection logic for GCC_REQD to fail if the
   version of GCC/GXX is not new enough.

 - Decide on defaults.  The straw proposal is that PKGSRC_GCC_VERSION
   is the base system version if >= 4.5 (or 4.4?), and otherwise 5,
   and that PKGSRC_GXX_VERSION is the base system version if >= 5, and
   otherwise 5.

### Later steps

 - Address fortran.