Annotation of wikisrc/pkgsrc/gcc.mdwn, revision 1.4

1.1       gdt         1: On many systems pkgsrc supports, gcc is the standard compiler.  In
                      2: general, different versions of each OS have different gcc versions,
1.2       gdt         3: and some packages require newer gcc versions, in order to support
1.1       gdt         4: newer language standards (e.g. c++11, written in the style of
                      5: USE_LANGUAGES), or because older versions don't work (infrequently).
                      7: This page discusses issues related to version selection, and intends
                      8: to be a design document for how pkgsrc should address this problem, to
                      9: be converted into historical design rationale once implemented.  It
                     10: freely takes content from extensive mailinglist discussions, and
                     11: attempts to follow the rough consensus that has emerged.
                     13: ## Base system gcc vs pkgsrc gcc
                     15: Systems using gcc (e.g. NetBSD) have a compiler as /usr/bin/gcc, and
                     16: this is usable by pkgsrc without any bootstrapping activity.  One can
                     17: build gcc versions (typically newer versions) from pkgsrc, resulting
                     18: in a compiler within ${PREFIX}, e.g. /usr/pkg/gcc6/bin/gcc.  This
                     19: compiler can then be used to compile other packages.
1.3       gdt        21: The Issue with using base system gcc is typically that it is too old,
1.1       gdt        22: such as gcc 4.5 with NetBSD 6, which cannot compile c++11.
                     24: Issues when using pkgsrc gcc are that
                     26:   - it must be bootstrapped, requiring compiling a number of packages
                     27:     with the system compiler
                     28:   - C++ packages that are linked together should be built with the
                     29:     same compiler, because the standard library ABI is not necessarily
                     30:     the same for each compiler version
                     31:   - While C packages can be built with mixed versions, the binary
                     32:     should be linked with the higher version because the support
                     33:     library is backwards compatible but not forward compatible.
                     35: ## Specific constraints and requirements
                     37: This section attempts to gather all the requirements.
                     39:   - By default, pkgsrc should be able to build working packages, even
                     40:     for packages that need a newer compiler than that provided in the
                     41:     base system.
                     43:   - The set of packages that are needed when building a bootstrap
                     44:     compiler should be minimized.
                     46:   - All packages that use C should have final linking with the highest
                     47:     version used in any included library.
1.2       gdt        49:   - All packages that use C++ should be built with the same compiler
1.3       gdt        50:     version.  Because these in the general case may include C, the
                     51:     version used for C++ must be at least as new as the version used
                     52:     for any used C package.
1.2       gdt        53: 
1.1       gdt        54:   - pkgsrc should avoid building gcc unless it is more or less
                     55:     necessary to build packges.  (As an example, if the base system
                     56:     gcc can build c99 but not c++11, building a c99-only program
                     57:     should not trigger building a gcc version adequate for c++11.)
1.2       gdt        59:   - The compiler selection logic should work on NetBSD 6 and newer,
                     60:     and other systems currently supported by pkgsrc, including in-use
                     61:     LTS GNU/Linux systems.  It should work on systems that default to
                     62:     clang, when set to use GCC, at least as well as the current
                     63:     scheme.  It is desirable for this logic to work on NetBSD 5.
1.1       gdt        64: 
                     65:   - The compiler selection logic should be understandable and not brittle.
                     67: ## Design
                     69: The above requirements could in theory be satisfied in many ways, but
1.2       gdt        70: most of them are too complicated.  We present a design that aims to be
                     71: sound while mimimizing complexity.
1.1       gdt        72: 
                     73:   - Packages declare what languages they need, with c++, c++11, and
1.2       gdt        74:     c++14 being expressed differently.  (This is exactly current
                     75:     practice and just noted for completeness.)
1.1       gdt        76: 
                     77:   - The package-settable variable GCC_REQD will be used only when a
                     78:     compiler that generally can compile the declared language version
1.2       gdt        79:     is insufficient.  These cases are expected to be relatively rare;
                     80:     an example is firefox that is in c++ (but not c+11) and needs gcc
                     81:     4.9.
1.1       gdt        82: 
                     83:   - A user-settable variable PKGSRC_GCC_VERSION will declare the
                     84:     version of gcc to be used for C programs, with an OS- and
                     85:     version--specific default.
                     87:   - A user-settable variable PKGSRC_GXX_VERSION will declare the version of gcc to
                     88:     be used for all C++ programs, again with an OS- and
                     89:     version-specific default.  It must be at least PKGSRC_GCC_VERSION.
                     91:   - Each of c99, c++, c++11, and c++14 will be associated with a
                     92:     minimum gcc version, such that almost all programs declaring that
                     93:     language can be built with that version.  (This avoids issues of
                     94:     strict compliance with c++11, which requires a far higher version
                     95:     of gcc than the version required to compile almost all actual
                     96:     programs in c++11.)
                     98:   - The minimum version inferred from the language tag will be
                     99:     combined with any GCC_REQD declarations to find a minimum version
                    100:     for a specific package.  If that is greater than
                    101:     PKGSRC_GCC_VERSION (programs using only C) or PKGSRC_GXX_VERSION,
                    102:     package building will fail.  We call the resulting
                    103:     PKGSRC_GCC_VERSION or PKGSRC_GXX_VERSION the chosen version.
1.3       gdt       105:   - When building a program using C or C++, if the chosen version is
                    106:     not provided by the base system, and the chosen version is not
1.1       gdt       107:     installed via pkgsrc, then it (and its dependencies) will be built
                    108:     from pkgsrc in a special bootstrap mode.  When building in
                    109:     bootstrap mode, the version selection logic is ignored and the
                    110:     base system compiler is used.  Consistency and reproducible builds
                    111:     require that a package built with the normal prefix must be the
                    112:     same whether built because of compiler bootstrapping or normal
                    113:     use.
                    115:     There are thus two choices for dealing with bootstrapping.  One is
                    116:     to use a distinct prefix, which will ensure that all packages that
                    117:     are part of the compiler bootstrap will not be linked into normal
                    118:     pkgsrc programs.  This implies that any dependencies of gcc may
                    119:     exist twice, once in bootstrap mode and once if built normally.  A
                    120:     gcc version itself will be built twice, if it is desired for
                    121:     regular use.  This double building and the complexity of a second
                    122:     prefix are the negatives of this approach.
                    124:     The other choice is to mark gcc and all depending packages as used
                    125:     for compiler bootstrapping, and to always build those with the
                    126:     base compiler.  We use the package-settable variable
                    127:     PKGSRC_GCC_BOOTSTRAP=yes to denote this.  The negative with this
                    128:     approach is possible inconsistency with gcc's dependencies being
                    129:     built with the base compiler and used later.
1.2       gdt       131:   - We hope that the chosen version can be built using the base system
                    132:     version, and hope to avoid multi-stage bootstrapping.
1.1       gdt       134:   - We expect that any program containing C++ will undergo final
                    135:     linking with a C++ compiler.  This is not a change from the
                    136:     current situation.
                    138: ## Remaining issues
                    140: ### gcc dependencies
                    142: Because gcc can have dependencies, there could be packages built with
                    143: the system compiler that are then later used with the chosen version.
                    144: For now, we defer worrying about these problems (judging that they
                    145: will be less serious than the current situation where all c++11
                    146: programs fail to build on NetBSD 6).
                    148: \todo: Analyze what build-time and install-time dependencies actually
                    149: exist.
                    151: \todo: Discuss adjusting options to minimize dependencies, including
                    152: gcc-inplace-math and nls.
1.3       gdt       154: ### Differing GCC and GXX versions
                    156: Perhaps it is a mistake to allow the chosen GCC and GXX versions to
                    157: differ.  If we require them to be the same, then essentially all
                    158: systems with a base system compiler older than gcc 5 will have to
                    159: bootstrap the compiler.  For now, we allow them to differ and will
                    160: permit the defaults to differ.
1.1       gdt       162: ### Default versions for various systems
1.3       gdt       164: Note that if for any particular system's set of installed packages (or
                    165: bulk build), a newer gcc has to be built, it does not hurt to have
                    166: built it earlier.
1.1       gdt       167: 
                    168: When the base system is old (e.g., gcc 4.5 in NetBSD 6, or 4.1, in
                    169: NetBSD 5), then it is clear that a newer version must be built.  For
                    170: these, PKGSRC_GXX_VERSION should default to a newish gcc, avoiding
1.4     ! gdt       171: being so new as to cause building issues.  Currently, gcc5 is probably
        !           172: a good choice, with gcc6 compiling significantly but not vastly fewer
        !           173: packages.  PKGSRC_GCC_VERSION should probably default to the system
        !           174: version if it can build all C99 programs, or match PKGSRC_GXX_VERSION,
        !           175: if the system version is too old.  Perhaps gcc 4.5 would be used, but
        !           176: 4.1 not used.  \todo Discuss.
1.1       gdt       177: 
                    178: When the base system is almost new enough, the decision about the
                    179: default is more complicated.  A key example is gcc 4.8, found in
1.2       gdt       180: NetBSD 7.  Firefox requires gcc 4.9, and all programs using c++14 also
1.1       gdt       181: need a newer version.  One options is to choose 4.8, resulting in
                    182: firefox failing, as well as all c++14 programs.  Another is to choose
                    183: 4.9, but this makes little sense because c++14 programs will still
                    184: fail, and the general rule of moving to the most recent
                    185: generally-acceptable version applies, which currently leads to gcc6.
                    186: This is in effect a declaration that "almost new enough" does not
                    187: count as new enough.  Thus the plan for NetBSD 7 is to set
1.4     ! gdt       188: PKGSRC_GCC_VERSION to 4.8 and PKGSRC_GXX_VERSION to 5.
1.1       gdt       189: 
1.4     ! gdt       190: When the base system is new enough, e.g. gcc 5, 6 or 7 it should
        !           191: simply be used.  By "new enough", we mean that almost no programs in
        !           192: pkgsrc fail to build with it, which implies that it supports (almost
        !           193: all) C++14 programs.  Our current definiton of new enough is gcc 5.
1.1       gdt       194: 
1.2       gdt       195: ### Limited mixed versions
                    197: One approach would be to allow limited mixed versions, where
                    198: individual programs could force a specific version to be bootstrapped
                    199: and used, so that e.g. firefox could use 4.9 even though most programs
                    200: use 4.8, which is what happens now on NetBSD 7.  This would rely on
                    201: being able to link c++ with 4.9 including some things built with 4.8
                    202: (which is done presently).  However, this approach would become
                    203: unsound with a library rather than an end program.  We reject this as
                    204: too much complexity for avoiding building a newer compiler in limited
                    205: situations.
1.1       gdt       207: ### Fortran
                    209: Fortran support is currently somewhat troubled..  It seems obvious to
                    210: extend to PGKSRC_GFORTRAN_VERSION, and have that match
                    211: PKGSRC_GCC_VERSION or PKGSRC_GXX_VERSION, but the Fortran situation is
                    212: not worsened by the above design.  \todo Discuss.
                    214: ## Path forward
                    216:  - Modify all gcc packages to have minimal dependencies, and to add
                    217:    PKGSRC_GCC_BOOTSTRAP.
                    219:  - Modify the compiler selection logic to do nothing if
                    220:    PKGSRC_GCC_BOOTSTRAP is set.
                    222:  - Modify the compiler selection logic for LANGUAGES= to fail if
                    223:    PKGSRC_GCC_VERSION/PKGSRC_GXX_VERSION is not new enough.
                    225:  - Modify the compiler selection logic for GCC_REQD to fail if the
                    226:    version of GCC/GXX is not new enough.
                    228:  - Decide on defaults.  The straw proposal is that PKGSRC_GCC_VERSION
                    229:    is the base system version if >= 4.5 (or 4.4?), and otherwise 6,
                    230:    and that PKGSRC_GXX_VERSION is the base system version if >= 5, and
                    231:    otherwise 6.
1.3       gdt       232: 
                    233: ### Later steps
                    235:  - Address fortran.

CVSweb for NetBSD wikisrc <> software: FreeBSD-CVSweb