wikisrc/pkgsrc/gcc.mdwn - annotate

Return to gcc.mdwn CVS log

Up to [NetBSD Developer Wiki] / wikisrc / pkgsrc

Annotation of wikisrc/pkgsrc/gcc.mdwn, revision 1.10

1.1 gdt 1: On many systems pkgsrc supports, gcc is the standard compiler. In
2: general, different versions of each OS have different gcc versions,
1.2 gdt 3: and some packages require newer gcc versions, in order to support
1.1 gdt 4: newer language standards (e.g. c++11, written in the style of
5: USE_LANGUAGES), or because older versions don't work (infrequently).
6:
7: This page discusses issues related to version selection, and intends
8: to be a design document for how pkgsrc should address this problem, to
9: be converted into historical design rationale once implemented. It
10: freely takes content from extensive mailinglist discussions, and
11: attempts to follow the rough consensus that has emerged.
12:
13: ## Base system gcc vs pkgsrc gcc
14:
15: Systems using gcc (e.g. NetBSD) have a compiler as /usr/bin/gcc, and
16: this is usable by pkgsrc without any bootstrapping activity. One can
17: build gcc versions (typically newer versions) from pkgsrc, resulting
18: in a compiler within ${PREFIX}, e.g. /usr/pkg/gcc6/bin/gcc. This
19: compiler can then be used to compile other packages.
20:
1.9 gdt 21: The issue with using base system gcc is typically that it is too old,
1.6 gdt 22: such as gcc 4.5 with NetBSD 6, which cannot compile c++11. Another
23: example is gcc 4.8 with NetBSD 7. While this can compile most c++11
24: programs, it cannot be used for firefox or glibmm (and therefore any
25: package that links against glibmm).
1.1 gdt 26:
27: Issues when using pkgsrc gcc are that
28:
1.9 gdt 29: - on some platforms, pkgsrc gcc does not build and work
1.1 gdt 30: - it must be bootstrapped, requiring compiling a number of packages
31: with the system compiler
32: - C++ packages that are linked together should be built with the
33: same compiler, because the standard library ABI is not necessarily
34: the same for each compiler version
35: - While C packages can be built with mixed versions, the binary
36: should be linked with the higher version because the support
37: library is backwards compatible but not forward compatible.
38:
39: ## Specific constraints and requirements
40:
41: This section attempts to gather all the requirements.
42:
43: - By default, pkgsrc should be able to build working packages, even
44: for packages that need a newer compiler than that provided in the
45: base system.
46:
47: - The set of packages that are needed when building a bootstrap
48: compiler should be minimized.
49:
50: - All packages that use C should have final linking with the highest
51: version used in any included library.
52:
1.2 gdt 53: - All packages that use C++ should be built with the same compiler
1.3 gdt 54: version. Because these in the general case may include C, the
55: version used for C++ must be at least as new as the version used
56: for any used C package.
1.2 gdt 57:
1.1 gdt 58: - pkgsrc should avoid building gcc unless it is more or less
59: necessary to build packges. (As an example, if the base system
60: gcc can build c99 but not c++11, building a c99-only program
61: should not trigger building a gcc version adequate for c++11.)
62:
1.2 gdt 63: - The compiler selection logic should work on NetBSD 6 and newer,
64: and other systems currently supported by pkgsrc, including in-use
65: LTS GNU/Linux systems. It should work on systems that default to
66: clang, when set to use GCC, at least as well as the current
67: scheme. It is desirable for this logic to work on NetBSD 5.
1.1 gdt 68:
1.9 gdt 69: - All systems should work at least as well as they do before
70: implementation of new compiler selection logic.
71:
1.1 gdt 72: - The compiler selection logic should be understandable and not brittle.
73:
74: ## Design
75:
76: The above requirements could in theory be satisfied in many ways, but
1.2 gdt 77: most of them are too complicated. We present a design that aims to be
78: sound while mimimizing complexity.
1.1 gdt 79:
80: - Packages declare what languages they need, with c++, c++11, and
1.2 gdt 81: c++14 being expressed differently. (This is exactly current
82: practice and just noted for completeness.)
1.1 gdt 83:
84: - The package-settable variable GCC_REQD will be used only when a
85: compiler that generally can compile the declared language version
1.2 gdt 86: is insufficient. These cases are expected to be relatively rare;
87: an example is firefox that is in c++ (but not c+11) and needs gcc
88: 4.9.
1.1 gdt 89:
90: - A user-settable variable PKGSRC_GCC_VERSION will declare the
1.9 gdt 91: version of gcc to be used for C programs, with an OS-,
92: version- and architeture- specific default.
1.1 gdt 93:
1.6 gdt 94: - A user-settable variable PKGSRC_GXX_VERSION will declare the
1.9 gdt 95: version of gcc to be used for all C++ programs, again with an OS-,
96: version- and architeture-specific default. It must be at least
1.6 gdt 97: PKGSRC_GCC_VERSION.
1.1 gdt 98:
1.9 gdt 99: - If PKGSRC_GCC_VERSION and PKGSRC_GXX_VERSION are not set, the
100: system will behave much as before. As a possible exception,
101: builds may still fail if the required version is greater than the
102: base system version. So far the only known reason to avoid
103: setting these variable is if pkgsrc gcc cannot be built.
104:
1.1 gdt 105: - Each of c99, c++, c++11, and c++14 will be associated with a
106: minimum gcc version, such that almost all programs declaring that
107: language can be built with that version. (This avoids issues of
108: strict compliance with c++11, which requires a far higher version
109: of gcc than the version required to compile almost all actual
110: programs in c++11.)
111:
112: - The minimum version inferred from the language tag will be
113: combined with any GCC_REQD declarations to find a minimum version
114: for a specific package. If that is greater than
115: PKGSRC_GCC_VERSION (programs using only C) or PKGSRC_GXX_VERSION,
116: package building will fail. We call the resulting
117: PKGSRC_GCC_VERSION or PKGSRC_GXX_VERSION the chosen version.
118:
1.3 gdt 119: - When building a program using C or C++, if the chosen version is
120: not provided by the base system, and the chosen version is not
1.1 gdt 121: installed via pkgsrc, then it (and its dependencies) will be built
122: from pkgsrc in a special bootstrap mode. When building in
123: bootstrap mode, the version selection logic is ignored and the
124: base system compiler is used. Consistency and reproducible builds
125: require that a package built with the normal prefix must be the
126: same whether built because of compiler bootstrapping or normal
127: use.
128:
129: There are thus two choices for dealing with bootstrapping. One is
130: to use a distinct prefix, which will ensure that all packages that
131: are part of the compiler bootstrap will not be linked into normal
132: pkgsrc programs. This implies that any dependencies of gcc may
133: exist twice, once in bootstrap mode and once if built normally. A
134: gcc version itself will be built twice, if it is desired for
135: regular use. This double building and the complexity of a second
136: prefix are the negatives of this approach.
137:
138: The other choice is to mark gcc and all depending packages as used
139: for compiler bootstrapping, and to always build those with the
140: base compiler. We use the package-settable variable
141: PKGSRC_GCC_BOOTSTRAP=yes to denote this. The negative with this
142: approach is possible inconsistency with gcc's dependencies being
143: built with the base compiler and used later.
144:
1.9 gdt 145: As an alternative, we store lists of bootstrap packages in a
146: variable, because it will vary with OS and version, and with
147: PREFER_PKGSRC settings.
148:
149: As a third alternative, we pass a GCC_BOOTSTRAPPING variable
150: recursively. This is easier but less consistent.
151:
1.2 gdt 152: - We hope that the chosen version can be built using the base system
153: version, and hope to avoid multi-stage bootstrapping.
154:
1.1 gdt 155: - We expect that any program containing C++ will undergo final
156: linking with a C++ compiler. This is not a change from the
157: current situation.
158:
159: ## Remaining issues
160:
1.6 gdt 161: ### gcc dependencies introduction
1.1 gdt 162:
163: Because gcc can have dependencies, there could be packages built with
164: the system compiler that are then later used with the chosen version.
165: For now, we defer worrying about these problems (judging that they
166: will be less serious than the current situation where all c++11
167: programs fail to build on NetBSD 6).
168:
1.9 gdt 169: \todo: Perhaps change gcc 4.8 and 4.9 to enable gcc-inplace-math by
170: default. Perhaps decide that if we want to build gcc, we want to
171: build 5 or 6, and 4.9 is no longer of interest as a bootstrap target.
1.6 gdt 172:
1.1 gdt 173: \todo: Analyze what build-time and install-time dependencies actually
1.6 gdt 174: exist. Include old GNU/Linux in this analysis.
175:
176: \todo: Consider if dropping nls would help. (On NetBSD, it seems that
177: base system libraries are used, so it would not help.)
178:
1.7 gdt 179: \todo: Consider failing if optins that we want one way are another,
180: when bootstrapping.
181:
1.6 gdt 182: ### managing gcc dependencies
183:
184: There are multiple paths forward.
185:
186: \todo Choose one. Straw proposal is "Don't worry" and recursive
187: variable for the initial implementation.
188:
189: #### Separate prefix
190:
191: Build compilers in a separate prefix, or a subprefix, so that the
192: compiler and the packages needed to build it will not be used by any
193: normal packages. This completely avoids the issue of building a
194: package one way in bootstrap and another not in bootstrap, at the cost
195: of two builds and writing the separate-prefix code.
196:
197: #### Don't worry
198:
199: Don't worry that packages used to bootstrap the needed compiler are
200: compiled with an older compiler. Don't worry that they might be
201: different depending on build order. If we have an actual problem,
202: deal with it. This requires choosing an approach to omit compiler
203: selection logic when building the compiler:
204:
205: ##### Mark bootstrap packages
206:
207: Mark packages used to build gcc as PKGSRC_GCC_BOOTSTRAP=yes.
208: Conditionalize this on OPSYS if necessary. Don't force the compiler
209: if this is set.
1.1 gdt 210:
1.8 gdt 211: Alternatively, manage a per-OS list of packages in a central mk file.
212:
1.6 gdt 213: ##### Pass a recursive variable
214:
215: As above, but set PKGSRC_GCC_BOOTSTRAP=yes in the evniroment of the
216: call to build the compiler, so that all dependencies inherit
217: permission to skip compiler selection logic. (Alternatively, use some
218: other mechanism such as passing a make variable explicitly.)
1.1 gdt 219:
1.3 gdt 220: ### Differing GCC and GXX versions
221:
222: Perhaps it is a mistake to allow the chosen GCC and GXX versions to
223: differ. If we require them to be the same, then essentially all
224: systems with a base system compiler older than gcc 5 will have to
225: bootstrap the compiler. For now, we allow them to differ and will
226: permit the defaults to differ.
227:
1.6 gdt 228: ### gcc versions and number of buildable packages
229:
230: A gcc version that is too old will not build a number of packages.
231: Anything older than 4.8 fails for c++11. 4.8 fails on some c++11
232: packages, such as firefox and glibmm.
233:
1.10 ! gdt 234: A version that is too new also fails to build packages. Jason Bacon
! 235: posted counts to tech-pkg indicate that 5 is close to 4.8 in the
! 236: number of packages built, and that moving to 6 causes hundreds of
! 237: additional failures. (Keep in mind that currently, building with 4.8
! 238: will build 4.9 for firefox, but in the future will not.)
! 239:
! 240: www/pkgsrc/packages/sharedapps/pkg-2017Q3/RHEL6-gcc48/All 16461
! 241: www/pkgsrc/packages/sharedapps/pkg-2017Q3/RHEL6-gcc6/All 15849
! 242:
! 243: www/pkgsrc/packages/sharedapps/pkg-2017Q3/RHEL7-gcc48/All 16414
! 244: www/pkgsrc/packages/sharedapps/pkg-2017Q3/RHEL7-gcc5/All 16338
1.6 gdt 245:
246: Therefore, the current answer to "What is the best version to use" is
247: 5.
248:
1.1 gdt 249: ### Default versions for various systems
250:
1.3 gdt 251: Note that if for any particular system's set of installed packages (or
252: bulk build), a newer gcc has to be built, it does not hurt to have
253: built it earlier.
1.1 gdt 254:
255: When the base system is old (e.g., gcc 4.5 in NetBSD 6, or 4.1, in
256: NetBSD 5), then it is clear that a newer version must be built. For
257: these, PKGSRC_GXX_VERSION should default to a newish gcc, avoiding
1.4 gdt 258: being so new as to cause building issues. Currently, gcc5 is probably
259: a good choice, with gcc6 compiling significantly but not vastly fewer
260: packages. PKGSRC_GCC_VERSION should probably default to the system
261: version if it can build all C99 programs, or match PKGSRC_GXX_VERSION,
262: if the system version is too old. Perhaps gcc 4.5 would be used, but
263: 4.1 not used. \todo Discuss.
1.1 gdt 264:
265: When the base system is almost new enough, the decision about the
266: default is more complicated. A key example is gcc 4.8, found in
1.2 gdt 267: NetBSD 7. Firefox requires gcc 4.9, and all programs using c++14 also
1.1 gdt 268: need a newer version. One options is to choose 4.8, resulting in
269: firefox failing, as well as all c++14 programs. Another is to choose
270: 4.9, but this makes little sense because c++14 programs will still
271: fail, and the general rule of moving to the most recent
272: generally-acceptable version applies, which currently leads to gcc6.
273: This is in effect a declaration that "almost new enough" does not
274: count as new enough. Thus the plan for NetBSD 7 is to set
1.4 gdt 275: PKGSRC_GCC_VERSION to 4.8 and PKGSRC_GXX_VERSION to 5.
1.1 gdt 276:
1.4 gdt 277: When the base system is new enough, e.g. gcc 5, 6 or 7 it should
278: simply be used. By "new enough", we mean that almost no programs in
1.6 gdt 279: pkgsrc fail to build with it (because it is too old), which implies
280: that it supports (almost all) C++14 programs. Our current definiton
281: of new enough is gcc 5.
1.1 gdt 282:
1.2 gdt 283: ### Limited mixed versions
284:
285: One approach would be to allow limited mixed versions, where
286: individual programs could force a specific version to be bootstrapped
287: and used, so that e.g. firefox could use 4.9 even though most programs
288: use 4.8, which is what happens now on NetBSD 7. This would rely on
289: being able to link c++ with 4.9 including some things built with 4.8
290: (which is done presently). However, this approach would become
291: unsound with a library rather than an end program. We reject this as
292: too much complexity for avoiding building a newer compiler in limited
293: situations.
294:
1.1 gdt 295: ### Fortran
296:
297: Fortran support is currently somewhat troubled.. It seems obvious to
298: extend to PGKSRC_GFORTRAN_VERSION, and have that match
299: PKGSRC_GCC_VERSION or PKGSRC_GXX_VERSION, but the Fortran situation is
1.6 gdt 300: not worsened by the above design.
301:
302: When building a gcc version, we get gfortran. Perhaps, because of
303: fortran, we should require a single version, vs a C and a C++ version.
304:
305: \todo Discuss.
306:
1.9 gdt 307: ### C++ libraries used by C programs
1.6 gdt 308:
309: The choice of one version for C++ and one for C (e.g. 5, 4.8 on
310: netbsd-7) breaks down if a C program links against a library that is
311: written in C++ but provides a C API, because we still need the C++
312: version's stdlib.
313:
314: \todo Define a variable for such packages to have in their buildlink3,
315: which will not add c++ to USE_LANGUAGES but will force
316: PKGSRC_GXX_VERSION to be used. Or decide that this is a good reason
317: to really just have one compiler version.
1.1 gdt 318:
319: ## Path forward
320:
1.9 gdt 321: (This assumes per-package marking of bootstrap packages, but is
322: reasonably obviously extended to the other schemes.)
323:
1.1 gdt 324: - Modify all gcc packages to have minimal dependencies, and to add
325: PKGSRC_GCC_BOOTSTRAP.
326:
327: - Modify the compiler selection logic to do nothing if
328: PKGSRC_GCC_BOOTSTRAP is set.
329:
330: - Modify the compiler selection logic for LANGUAGES= to fail if
331: PKGSRC_GCC_VERSION/PKGSRC_GXX_VERSION is not new enough.
332:
1.9 gdt 333: - Modify the compiler selection logic for GCC_REQD to fail if
334: PKGSRC_GCC_VERSION/PKGSRC_GXX_VERSION is not new enough.
1.1 gdt 335:
336: - Decide on defaults. The straw proposal is that PKGSRC_GCC_VERSION
1.5 gdt 337: is the base system version if >= 4.5 (or 4.4?), and otherwise 5,
1.1 gdt 338: and that PKGSRC_GXX_VERSION is the base system version if >= 5, and
1.9 gdt 339: otherwise 5. Implement these in platform.mk as they are tested.
1.3 gdt 340:
341: ### Later steps
342:
1.9 gdt 343: - Address fortran. Probably add PKGSRC_GFORTRAN_VERSION, after
344: determining how Fortran, C and C++ interact with library ABI
345: compatibility.

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb