Annotation of wikisrc/kyua/import.mdwn, revision 1.1
1.1 ! jmmv 1: [[!meta title="Kyua: The way into NetBSD"]]
! 2: [[!toc levels=2]]
! 4: **Project owner: [Julio Merino](mailto:jmmv@NetBSD.org).**
! 5: **Status: draft as of 2012-08-31; not even sent for review yet.**
! 7: The import of Kyua into NetBSD to replace the deprecated ATF tools is
! 8: planned to happen in NetBSD 7.0. The ATF libraries will remain in place,
! 9: and as such no changes will happen to the tests that already live in
! 10: `src/tests`.
! 12: The transition from ATF to Kyua includes steps and tools to offer backwards
! 13: compatibility with users that may rely on the ATF tools shipped since
! 14: NetBSD 5.0. These backwards compatibility tools cover the most common use
! 15: cases but might not be perfect initially. Despite these provisions, this
! 16: plan should be executed well in advance the creation of the 7.0 branch to
! 17: ensure there is enough time in NetBSD-current to flesh out any major
! 18: problems.
! 20: Before moving on, please read the
! 21: [Kyua: An introduction for NetBSD users](kyua) page. You should be
! 22: familiar with the structure of Kyua and its major components to be able to
! 23: review this plan.
! 25: # Proposed changes
! 27: The final user visible changes that this project will bring to NetBSD are
! 28: as follows. Because these change the availability of tools that have
! 29: already been shipped since 5.0, the modifications are staged to happen
! 30: across two major releases:
! 32: ## For NetBSD 7.0
! 34: * Addition of the new `kyua` command-line tool. This provides the runtime
! 35: engine for the test programs (`atf-run`'s functionality) and the ability
! 36: to generate reports of the results in plain text form and HTML form
! 37: (`atf-report`'s functionality). This tool is able to execute all the
! 38: existing test programs without modifications.
! 40: * Replacement of the `Atffile`s files in `/usr/tests` with `Kyuafile`s.
! 42: * Ability to generate HTML reports right form the base system, with such
! 43: reports hooked into the various continuous build systems.
! 45: * Replacement of the `atf-run` and `atf-report` tools with shell scripts
! 46: that provide backwards compatibility implementations based on `kyua`.
! 47: These can deal with old-style `Atffile`s so that users with custom test
! 48: suites can continue to run them.
! 50: * Introduction of a new `atf2kyua` script to convert old `Atffile`s to
! 51: `Kyuafile`s. This is used internally by the compatibility `atf-run`
! 52: script, but can also be invoked by the end user by hand to deal with his
! 53: own test suites. Could be placed into `libexec` if we do not want to
! 54: make this public.
! 56: ## For NetBSD 8.0
! 58: * Removal of the backwards compatibility `atf-run` and `atf-report`
! 59: scripts, as well as the supporting `atf2kyua` tool.
! 61: # Why now?
! 63: Kyua has existed for almost 3 years already, so you may be wondering why
! 64: this import is being proposed now. The major reasons are:
! 66: * Since 0.5, Kyua has now feature-parity with ATF. Replacing the ATF tools
! 67: with Kyua should not introduce functionality regressions.
! 69: * NetBSD-6 has been branched relatively recently, so we can expect NetBSD-7
! 70: to be far away enough in the future to provide plenty of time to
! 71: stabilize Kyua in NetBSD-current.
! 73: * The existence of the deprecated `atf-run` and `atf-report` tools hinders
! 74: the development of new features that require changes to the inteface of
! 75: ATF test programs. As an example: a highly requested feature is the
! 76: ability to change the timeout settings of the test programs (for the
! 77: benefit of old, slow platforms); doing this in the ATF codebase is
! 78: tricky.
! 80: * Some developers continue to add old-style tests to the tree. While this
! 81: is suboptimal and against the policy dictated by core@, Kyua brings in a
! 82: mechanism to allow running these tests unmodified (i.e. without adding
! 83: any ATF calls in their code). This, in turn, will let such developers
! 84: add their tests more easily, and thus increase the tests coverage.
! 86: # The plan
! 88: This section details what the transition plan looks like and an estimate
! 89: timeline. As all things that depend on free time (opensource software
! 90: hacking, in this case), take the estimates with a grain of salt.
! 92: ## Discuss this plan in tech-userlevel@
! 94: If you are here it's possibly because a review request for this plan has
! 95: already been published and thus the plan has already begun. If
! 96: not... well, hold off for a little bit ;-)
! 98: This plan will be sent to the tech-userlevel@ mailing list asking for
! 99: comments. **Two weeks shall be allowed for initial discussion.** Depending
! 100: on the outcome, the plan and/or the software will need to be adjusted
! 101: (which in turn may require significant amounts of time not yet accounted
! 102: for). I'll be optimistic for now.
! 104: ## Get core@ approval
! 106: By policy, the import of any new software component into src requires core@
! 107: approval. Even though this will have been discussed at length in
! 108: tech-userlevel@ (as per the previous step), the final decider on this issue
! 109: will be core@. The corollary of this is that, if no consensus can be
! 110: reached in tech-userlevel@ regarding this plan, core@ will be asked to come
! 111: up with a decision.
! 113: If core@ approves the plan, the next steps shall start immediately. If
! 114: core@ disagrees, core@ will be asked to provide advice on the corrections
! 115: that should be made before the plan can be approved.
! 117: ## Import Kyua into src
! 119: As the [introductory page to Kyua](kyua) describes, Kyua has been available
! 120: in pkgsrc for a while and can be readily installed and used to run the
! 121: tests from `/usr/tests`.
! 123: However, because ATF lives in src, and because NetBSD aims to provide the
! 124: best environment for testing "out of the box", Kyua should be imported into
! 125: src just like ATF was. The major reasons for this, as have been explained
! 126: in the past, are: first, to allow any new deployment of NetBSD to be
! 127: validated right after installation and continuously afterwards; and,
! 128: second, to permit the execution of tests during development without having
! 129: to install any additional software.
! 131: The specific steps to perform this import are:
! 133: 1. Import Lutok into `external/bsd/lutok/`. This is a shared library that
! 134: wraps the native Lua C interface in C++. Lutok was originally part of
! 135: Kyua, and was split into its own package per the request of some users
! 136: that found this component useful on its own.
! 138: 1. Import Kyua into `external/bsd/kyua-cli/`. This yields a new kyua
! 139: binary in `/usr/bin`, a lot of test programs in `/usr/tests/kyua-cli`
! 140: (around 100) and some auxiliary files in `/usr/share/kyua`.
! 142: 1. Protect all products of Lutok and Kyua with the existing `MKATF` knob.
! 143: If desired, these could instead be protected by a new and separate
! 144: `MKKYUA` knob, but because the transition is only temporary, I do not
! 145: think it's worth the effort.
! 147: 1. Update `bsd.test.mk` to generate `Kyuafile`s *in addition to*
! 148: `Atffile`s.
! 150: There is no real need to do this import in a branch given that this import
! 151: only adds new functionality without touching existing stuff.
! 153: All the preparatory work for the import can be done offline (in about two
! 154: weeks at most, given that I have mot of this ready). Aside from the code
! 155: changes, this will involve the validation of NetBSD/amd64, NetBSD/i386 and
! 156: NetBSD/macppc builds (which are the ports I have access to). If you
! 157: consider that some other tricky architecture should be build-tested
! 158: (sparc64?), let me know and I'll include it in the list.
! 160: The submission step to CVS, once all the code changes are ready locally,
! 161: and any post-commit validation checks will take a few hours. Any build
! 162: breakage should be addressed in a timely manner (and can possibly be
! 163: worked-around by setting `MKATF=no`).
! 165: ## Update documentation
! 167: The [Kyua: An introduction for NetBSD users](kyua) wiki page will be
! 168: updated to explain how Kyua is bundled in NetBSD and how to use the bundled
! 169: version.
! 171: The [Creating atf-based tests for NetBSD src](atf) wiki page will be
! 172: updated to account for the differences in test programs execution with Kyua
! 173: instead of ATF.
! 175: The afterboot(8) and tests(7) manpages will be adjusted to mention `kyua`
! 176: instead of the ATF tools.
! 178: ## Adjust continuous testing systems to use Kyua
! 180: With `kyua` being part of the release sets, it is possible to adjust the
! 181: continuous test systems to make use of this tool in the test environments
! 182: without having to take any additional step.
! 184: I'll work with gson@ and pgoyette@ to adapt their continuous testing
! 185: machines to use the new built-in `kyua` binary instead of `atf-run` and
! 186: `atf-report`. I'm planning to do the necessary work to change `anita`
! 187: myself, and I expect to help them deploy the changes to their own systems.
! 188: Because Kyua and ATF will cohexist in the base system at this point,
! 189: migrating the continuous testing systems to Kyua can happen at its own
! 190: peace.
! 192: It might happen that Kyua misses some little detail needed by these
! 193: systems. In that case, this may require a new release of Kyua and a
! 194: reimport into the tree. Incremental imports with new features are much
! 195: easier than the original import described in here. Also, it will be
! 196: possible to cherry-pick any external changes into the tree without a
! 197: reimport (as has often been done in ATF).
! 199: ## User validation period
! 201: At this stage, **at least one month shall be given to the community** to
! 202: test the new tools and the new test results dashboards. Collect feedback
! 203: and address requests as appropriate.
! 205: ## Replace atf-run and atf-report with kyua-atf-compat
! 207: Import the Kyua-based `atf-run` and `atf-report` compatibility tools and
! 208: stop building the deprecated versions of these. The compatibility versions
! 209: are shipped in a separate `kyua-atf-compat` package, and thus this will be
! 210: imported into `external/bsd/kyua-atf-compat/`.
! 212: Once this is done, change `bsd.test.mk` to not generate `Atffile`s any
! 213: more, as the compatibility tools do not need them.
! 215: ## Drop atf-run and atf-report's code
! 217: Delete the native `atf-run` and `atf-report` utilities from the source tree
! 218: (and from the upstream repository). This also gets rid of a lot of
! 219: supporting helper code, which makes the various ATF libraries leaner (and
! 220: therefore benefits all test programs!).
! 222: ## Get rid of some ATF wrappers as a proof of concept
! 224: There are some test programs (e.g. the ones in the ipf test suite) that are
! 225: not ATF-based. To plug them into the NetBSD test suite, we have
! 226: implemented ATF-based wrappers that invoke the original plain tests. This
! 227: is certainly suboptimal, as this level of indirection hinders readability
! 228: and makes development harder. (In particular, it prevents some key
! 229: developers from contributing tests in the first place.)
! 231: Because Kyua has the ability to run these "plain" (non-ATF) test programs
! 232: directly, and because a bunch of developers are really looking forward to
! 233: this feature, I will convert a few tests to not include ATF wrappers as a
! 234: proof of concept of this feature. The ipf tests are probably a good choice
! 235: for this.
! 237: ## Remove atf-run and atf-report compatibility tools
! 239: This is still really far away in the future (needs to happen after NetBSD
! 240: 7.0 is branched), but I'm listing it for completeness of the plan. The
! 241: idea is to get rid of any ATF compatibility scripts by NetBSD 8.0. This
! 242: gives end users a full release cycle (that of 7) to adapt to Kyua while at
! 243: the same time being able to use the old-style tools.
! 245: # Possible concerns
! 247: This section attempts to collect the list of possible concerns and/or
! 248: objections that may come up during the review, together with an attempt of
! 249: rebuttal from my side.
! 251: ## Increased compilation time
! 253: The addition of any new component to src increases the build time of the
! 254: whole system. In the case of Kyua, this increase might be noticeable
! 255: *because of the large amount of test programs provided (roughly 100)*, all
! 256: of which are in C++.
! 258: Note that ATF also had a relatively large codebase and a bunch of tests
! 259: (although not as many). The "trick" was that the majority of these tests
! 260: were written in shell, and as such they did not increase the build time by
! 261: much. However, they significantly increased the run time of the test
! 262: suite, and they were less detailed (mostly integration tests, few unit
! 263: tests) than the Kyua tests.
! 265: In order to mitigate this issue, the build of all pieces of Kyua will be
! 266: protected by `MKATF` so that people allergic to C++ can avoid the whole
! 267: thing. Even more, there are some other additional provisions described
! 268: below.
! 270: ## Introduction of more C++ code in base
! 272: First of all, why did I use C++? To make the implementation simpler and
! 273: safer (the RAII programming pattern is really useful in avoiding memory and
! 274: resource leaks with minimal effort). And C++ is part of the base system
! 275: and a supported language, so there was no reason not to do so. But that's
! 276: not the point of this item: if you don't like C++, this is not going to
! 277: convince you otherwise.
! 279: It's true that, if we count the number of lines, Kyua brings in more C++
! 280: code than what will eventually be dropped by the removal of the ATF tools.
! 281: However, because ATF was also C++, the import of Kyua itself does not make
! 282: the situation significantly worse.
! 284: Additionally, the two compilers we can really use (GCC and LLVM) already
! 285: use, or will soon use, C++ in their code base. It is unlikely that we will
! 286: be able to remove all C++ support from base anytime soon due to this, while
! 287: at the same time keeping support for all the ports that NetBSD has.
! 289: Long term, if the use of C++ proves to be a problem, there are a few things
! 290: that can be done to slowly get rid of it that have been floating my mind
! 291: recently. The first is the rewrite of the performance-critical parts of
! 292: Kyua in plain C. This would involve splitting the runtime engine of a
! 293: single test case in its own binary (which, for other reasons may be a good
! 294: idea on its own). The second is the rewrite of most user-interface code in
! 295: Lua, which in itself would bring some extensibility advantages to the
! 296: program. Both options will be investigated when the time permits. In the
! 297: meantime, the replacement of ATF with Kyua does not make things worse; it
! 298: just changes one chunk of code with another.
! 300: ## No need for Lutok as a public library
! 302: Kyua depends on Lutok, which is a C++ interface to Lua. The code of this
! 303: library was originally part of Kyua, but I split it into its own project
! 304: because some users asked for it.
! 306: If there is no desire to ship Lutok as a shared library in `/usr/lib`, we
! 307: can build Lutok as a static private library and link Kyua against it.
! 308: There is no need to install this as a shared library.
CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb