Annotation of wikisrc/symbol_versions.mdwn, revision 1.1

1.1     ! riastrad    1: [[!meta title="Symbol Versions in NetBSD Libraries"]]
        !             2: 
        !             3: NetBSD implements various standard C language interfaces such as the
        !             4: [[!template id=man name="time" section="3"]] function in POSIX in
        !             5: `libc`, which has a prototype like this:
        !             6: 
        !             7:     time_t time(time_t *);
        !             8: 
        !             9: However, between NetBSD 5 and NetBSD 6, the definition of the type
        !            10: `time_t` in NetBSD changed on many architectures from 32-bit to 64-bit
        !            11: to avoid the
        !            12: [year 2038 problem](https://en.wikipedia.org/wiki/Year_2038_problem).
        !            13: So programs compiled in NetBSD<=5 saw a declaration like
        !            14: 
        !            15:     int time(int *);
        !            16: 
        !            17: which on most architectures is 32-bit, while programs compiled in
        !            18: NetBSD>=6 see a declaration like
        !            19: 
        !            20:     int64_t time(int64_t *);
        !            21: 
        !            22: These declarations are not compatible -- consider a program with a
        !            23: fragment like:
        !            24: 
        !            25:     int before, after;
        !            26: 
        !            27:     time(&before);
        !            28:     ...
        !            29:     time(&after);
        !            30: 
        !            31: This would work in NetBSD<=5, but in NetBSD>=6, the calls to
        !            32: [[!template id=man name="time" section="3"]] might overwrite adjacent
        !            33: positions on the stack, or crash altogether because the argument is
        !            34: misaligned.
        !            35: 
        !            36: Programs written and compiled on older versions of NetBSD are supposed
        !            37: to continue to work -- with suitable emulators/compatNN packages and
        !            38: compatNN.kmod modules or COMPAT_NN kernel options -- on newer versions
        !            39: of NetBSD.
        !            40: 
        !            41: To make this work, NetBSD's `libc` provides _two_ symbols:
        !            42: 
        !            43: - `time`, which still implements the legacy prototype as before; and
        !            44: - `__time50` (yes, this is not a typo for `__time60`), which implements
        !            45:   the new 64-bit prototype.
        !            46: 
        !            47: The declaration in newer NetBSD
        !            48: [time.h](https://nxr.netbsd.org/xref/src/include/time.h) is actually:
        !            49: 
        !            50:     time_t time(time_t *) __RENAME(__time50);
        !            51: 
        !            52: where `__RENAME(__time50)` is a macro expanding to `__asm("__time50")`,
        !            53: which has the effect that the compiler will use the symbol `__time50`
        !            54: for calls to the C function this declares.
        !            55: Thus, old programs with calls to the symbol `time` using the 32-bit
        !            56: prototype will continue to work, and new programs will be compiled to
        !            57: call the symbol `__time50` using the 64-bit prototype.
        !            58: ([Details on how the symbols are implemented in `libc`.](https://nxr.netbsd.org/xref/src/lib/libc/README))
        !            59: 
        !            60: * [[!template id=man name="dlsym" section="3"]] and symbol interposition
        !            61: 
        !            62: **Programs that use
        !            63: [[!template id=man name="dlsym" section="3"]],
        !            64: such as C foreign function interfaces in dynamic languages like Python,
        !            65: need to know that if they want the legacy 32-bit time() function, they
        !            66: must use the symbol `time`, and if they want the modern 64-bit time()
        !            67: function, they must use the symbol `__time50`.
        !            68: Similarly, programs that use `LD_PRELOAD` (see
        !            69: [[!template id=man name="ld.elf_so" section="3"]])
        !            70: to interpose their own definitions of symbols, such as
        !            71: [torsocks](https://gitlab.torproject.org/legacy/trac/-/wikis/doc/torsocks),
        !            72: must know to define `__time50` if they want to replace the new
        !            73: semantics in new programs, or `time` if they want to replace the old
        !            74: semantics in old programs.**
        !            75: The same applies to many other standard C functions, such as
        !            76: [[!template id=man name="clock_gettime" section="3"]]
        !            77: (`__clock_gettime50`) and
        !            78: [[!template id=man name="socket" section="3"]]
        !            79: (`__socket30`), which have all had their prototypes or semantics
        !            80: revised at some point.
        !            81: 
        !            82: Symbol interposition is very difficult to get right, and it is hard to
        !            83: make programs that do it reliably.
        !            84: On NetBSD, it should be reserved for certain standard library functions
        !            85: like `malloc` and `free` (and `calloc` and everything else in that
        !            86: family), and some system call stubs; except for the `__...50`
        !            87: pseudo-versioned renames of public functions, you should not try to
        !            88: interpose your own definition of any symbol beginning with `_`, which
        !            89: is reserved to the implementation in C.
        !            90: 
        !            91: * Appendix: ELF symbol versions
        !            92: 
        !            93: The renaming scheme of `__time50` is informal -- any symbol can be
        !            94: renamed the same way, and NetBSD uses it for some other purposes, such
        !            95: as exposing a slightly different
        !            96: [[!template id=man name="rename" section="2"]]
        !            97: function via the symbol `__posix_rename` in programs that define
        !            98: `_POSIX_C_SOURCE` but not `_NETBSD_SOURCE`.
        !            99: 
        !           100: The GNU ELF toolchain (gcc, ld, &c.) supports a formal concept of
        !           101: ‘symbol versions’ with sections called `.gnu.version` (associating
        !           102: versions with symbols), `.gnu.version_d` (versions defined in an
        !           103: object), and `.gnu.version_n` (versions needed in an object).
        !           104: As of 2020, NetBSD does not use ELF symbol versions (although the
        !           105: linker and loader support them).
        !           106: 
        !           107: The semantics is:
        !           108: 
        !           109: - When creating a library, a version map may be specified like so:
        !           110: 
        !           111:     NetBSD_BASE {
        !           112:             global:
        !           113:                     __time50;
        !           114:                     free;
        !           115:                     malloc;
        !           116:                     time;
        !           117:             local:
        !           118:                     *;
        !           119:     };
        !           120: 
        !           121:     NetBSD_6 {
        !           122:             global:
        !           123:                     time;
        !           124:     };
        !           125: 
        !           126:   The library can specify what versioned symbol each definition in the
        !           127:   library is exposed with:
        !           128: 
        !           129:     __asm(".symver time_legacy,time@NetBSD_BASE");
        !           130:     int time_legacy(int *t) { ... }
        !           131: 
        !           132:     __asm(".symver time64,time@@NetBSD_6");         /* default version */
        !           133:     int64_t time64(int64_t *t) { ... }
        !           134: 
        !           135:     __asm(".symver __time50,__time50@NetBSD_BASE");
        !           136:     __typeof(time) __time50 __attribute__((__alias__("time64")));
        !           137: 
        !           138:   Versions marked with `@@` are _default_ versions; versions marked
        !           139:   with `@` are non-default.
        !           140: 
        !           141: - When running a program that was linked _without_ ELF symbol versions,
        !           142:   from before the library had ELF symbol versions (like `libc` today),
        !           143:   the first version in the map is used to resolve symbols:
        !           144: 
        !           145:   - Old programs calling the legacy `time` symbol will get
        !           146:     `time@NetBSD_BASE`, which is defined via `time_legacy` above.
        !           147: 
        !           148:   - Programs calling `__time50` will get `__time50@NetBSD_BASE`, which
        !           149:     is defined via `time64` above.
        !           150: 
        !           151: - When linking a program against a library with symbol versions, the
        !           152:   linker will record what the default version was; when later running
        !           153:   the program, the stored symbol version will be used.
        !           154:   If there is no default version, and the program did not request a
        !           155:   specific version with `.symver`, then the linker refuses to link, so
        !           156:   obsolete symbols can be ‘removed’ by giving them only non-default
        !           157:   versions -- thus old programs continue to work but new programs can't
        !           158:   be made that use the obsolete symbols.
        !           159: 
        !           160:   For example, if [[!template id=man name="time" section="3"]] is
        !           161:   declared in a header file as simply
        !           162: 
        !           163:     typedef int64_t time_t;
        !           164:     time_t time(time_t *);
        !           165: 
        !           166:   then new programs will be linked against `time@NetBSD_6`, which is
        !           167:   the default version for the symbol name `time`.
        !           168:   If NetBSD ever changed the prototype of
        !           169:   [[!template id=man name="time" section="3"]]
        !           170:   again, and defined a `time@NetBSD_11` as the new default version,
        !           171:   existing programs compiled with `time@NetBSD_6` would continue to get
        !           172:   the semantics they were built against.
        !           173: 
        !           174: - When a program uses
        !           175:   [[!template id=man name="dlsym" section="3"]],
        !           176:   it always gets the default version, if any.
        !           177:   Programs can request specific versions with
        !           178:   [[!template id=man name="dlvsym" section="3"]].
        !           179: 
        !           180: ** ELF symbol versions versus `__...50` pseudo-versions
        !           181: 
        !           182: ELF symbol versions and NetBSD's `__time50` pseudo-version renaming
        !           183: scheme both try to address the same problem: making sure old programs
        !           184: that were built under the assumption of the old semantics continue to
        !           185: run unmodified with new libraries.
        !           186: 
        !           187: Both of them run into problems with
        !           188: [[!template id=man name="dlsym" section="3"]]
        !           189: and symbol interposition:
        !           190: 
        !           191: - A program written _today_ that expects to find the function time() in
        !           192:   `libc`, such as a C foreign function interface for a dynamic language
        !           193:   like Python, needs to know to call `dlsym("__time50")`; otherwise it
        !           194:   will get an obsolete definition that does not match the semantics of
        !           195:   the current definition of `time_t`, possibly leading to data
        !           196:   corruption, crashes, or worse.
        !           197: 
        !           198: - If `libc` used used ELF symbol versions, then `dlsym("time")` would
        !           199:   return the modern symbol.
        !           200: 
        !           201:   But any _old_ programs that used `dlsym("time")` assuming it
        !           202:   returned the legacy definition (which was the ‘modern’ definition at
        !           203:   the time the programs were written and built) will break if it
        !           204:   instead returns the 64-bit definition.
        !           205: 
        !           206:   And if we ever modified
        !           207:   [[!template id=man name="time" section="3"]]
        !           208:   again (hypothetically, to extend it to 128-bit galactic-scale times),
        !           209:   programs written assuming that `dlsym("time")` returns the 64-bit
        !           210:   definition will break if it begins to return the 128-bit definition.
        !           211:   Programs could future-proof themselves by using `dlsym("time",
        !           212:   "NetBSD_6")` explicitly, but this is no better than writing
        !           213:   `dlsym("__time50")` explicitly.
        !           214: 
        !           215: **Thus, switching from the pseudo-versions we use to ELF symbol
        !           216: versions doesn't improve the
        !           217: [[!template id=man name="dlsym" section="3"]]
        !           218: situation -- in fact, it makes the situation _worse_, by breaking old
        !           219: programs and providing no way for new programs to bind to the name of
        !           220: the current version.**
        !           221: 
        !           222: Perhaps we could create a compiler builtin `__builtin_asm_name` which
        !           223: would expand to the `__asm("...")` name by which a C identifier has
        !           224: been declared -- then programs could instead do:
        !           225: 
        !           226:     __typeof(time) *timep = dlsym(dso, __builtin_asm_name(time));
        !           227: 
        !           228: This way the text of the program is the same no matter how
        !           229: [[!template id=man name="time" section="3"]]
        !           230: is declared in the header file, but it will continue to work across
        !           231: changes to the signature of the
        !           232: [[!template id=man name="time" section="3"]]
        !           233: function in newer releases of NetBSD.
        !           234: 
        !           235: * References
        !           236: 
        !           237: - Jörg Sonnenberger, ‘How to break long-term compatibility in NetBSD’,
        !           238:   AsiaBSDcon 2016.
        !           239:   https://www.NetBSD.org/gallery/presentations/joerg/asiabsdcon2016/asiabsdcon2016.pdf
        !           240: 
        !           241: - Ulrich Drepper, ‘How To Write Shared Libraries’, 2011-12-10.
        !           242:   https://akkadia.org/drepper/dsohowto.pdf
        !           243: 
        !           244: - Ulrich Drepper, ‘ELF Symbol Versioning’.
        !           245:   https://akkadia.org/drepper/symbol-versioning

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb