Annotation of wikisrc/kyua/import.mdwn, revision 1.9

1.1       jmmv        1: [[!meta title="Kyua: The way into NetBSD"]]
                      2: [[!toc levels=2]]
                      4: **Project owner: [Julio Merino](**  
1.9     ! jmmv        5: **Status: Approved by core as of 2013-02-09.  Import of Lutok and a new
        !             6: release of ATF to begin soon.**
1.1       jmmv        7: 
                      8: The import of Kyua into NetBSD to replace the deprecated ATF tools is
                      9: planned to happen in NetBSD 7.0.  The ATF libraries will remain in place,
                     10: and as such no changes will happen to the tests that already live in
                     11: `src/tests`.
                     13: The transition from ATF to Kyua includes steps and tools to offer backwards
                     14: compatibility with users that may rely on the ATF tools shipped since
                     15: NetBSD 5.0.  These backwards compatibility tools cover the most common use
                     16: cases but might not be perfect initially.  Despite these provisions, this
                     17: plan should be executed well in advance the creation of the 7.0 branch to
                     18: ensure there is enough time in NetBSD-current to flesh out any major
                     19: problems.
1.3       jmmv       21: **Before moving on, please read the
1.2       jmmv       22: [[Kyua: An introduction for NetBSD users|/kyua]] page.  You should be
1.1       jmmv       23: familiar with the structure of Kyua and its major components to be able to
1.3       jmmv       24: review this plan.**
1.1       jmmv       25: 
                     26: # Proposed changes
                     28: The final user visible changes that this project will bring to NetBSD are
                     29: as follows.  Because these change the availability of tools that have
                     30: already been shipped since 5.0, the modifications are staged to happen
                     31: across two major releases:
                     33: ## For NetBSD 7.0
                     35: * Addition of the new `kyua` command-line tool.  This provides the runtime
                     36:   engine for the test programs (`atf-run`'s functionality) and the ability
                     37:   to generate reports of the results in plain text form and HTML form
                     38:   (`atf-report`'s functionality).  This tool is able to execute all the
                     39:   existing test programs without modifications.
                     41: * Replacement of the `Atffile`s files in `/usr/tests` with `Kyuafile`s.
                     43: * Ability to generate HTML reports right form the base system, with such
                     44:   reports hooked into the various continuous build systems.
                     46: * Replacement of the `atf-run` and `atf-report` tools with shell scripts
                     47:   that provide backwards compatibility implementations based on `kyua`.
                     48:   These can deal with old-style `Atffile`s so that users with custom test
                     49:   suites can continue to run them.
                     51: * Introduction of a new `atf2kyua` script to convert old `Atffile`s to
                     52:   `Kyuafile`s.  This is used internally by the compatibility `atf-run`
                     53:   script, but can also be invoked by the end user by hand to deal with his
                     54:   own test suites.  Could be placed into `libexec` if we do not want to
                     55:   make this public.
                     57: ## For NetBSD 8.0
                     59: * Removal of the backwards compatibility `atf-run` and `atf-report`
                     60:   scripts, as well as the supporting `atf2kyua` tool.
1.3       jmmv       62: # Why?
                     64: As mentioned in [[another page|/kyua]], Kyua should be seen as ATF 2.x even
                     65: though it carries a different name.  It is the evolution of the previous
                     66: ATF tools (*the tools only*), but written in a more modular and flexible
                     67: way, and with a more reliable codebase.  Therefore, you should consider
                     68: this project as the update of ATF to a newer version.
                     70: The `atf-run` and `atf-report` tools have effectively been in maintenance
                     71: mode for over a year already.  None of the desired features (see the list
                     72: of open PRs) have been implemented on top of them, mostly because doing so
                     73: is building upon a broken implementation.  Additionally, several developers
                     74: have had to implement their own test results dashboards due to
                     75: defficiencies in `atf-report`, effectively reinventing the wheel.
                     77: This update will permit the real removal of the obsolete tools, thus
                     78: allowing us to build additional features on top of Kyua without having to
                     79: worry about being compatible with `atf-run` (and thus adjusting this tool
                     80: to behave in the same manner).
                     82: Some possible answers to "Why not?" can be found later in this page.
                     84: ## And why now?
1.1       jmmv       85: 
                     86: Kyua has existed for almost 3 years already, so you may be wondering why
                     87: this import is being proposed now.  The major reasons are:
                     89: * Since 0.5, Kyua has now feature-parity with ATF.  Replacing the ATF tools
                     90:   with Kyua should not introduce functionality regressions.
                     92: * NetBSD-6 has been branched relatively recently, so we can expect NetBSD-7
                     93:   to be far away enough in the future to provide plenty of time to
                     94:   stabilize Kyua in NetBSD-current.
                     96: * The existence of the deprecated `atf-run` and `atf-report` tools hinders
                     97:   the development of new features that require changes to the inteface of
                     98:   ATF test programs.  As an example: a highly requested feature is the
                     99:   ability to change the timeout settings of the test programs (for the
                    100:   benefit of old, slow platforms); doing this in the ATF codebase is
                    101:   tricky.
                    103: * Some developers continue to add old-style tests to the tree.  While this
                    104:   is suboptimal and against the policy dictated by core@, Kyua brings in a
                    105:   mechanism to allow running these tests unmodified (i.e. without adding
                    106:   any ATF calls in their code).  This, in turn, will let such developers
                    107:   add their tests more easily, and thus increase the tests coverage.
                    109: # The plan
                    111: This section details what the transition plan looks like and an estimate
                    112: timeline.  As all things that depend on free time (opensource software
                    113: hacking, in this case), take the estimates with a grain of salt.
                    115: ## Discuss this plan in tech-userlevel@
                    117: If you are here it's possibly because a review request for this plan has
1.4       jmmv      118: already been published and thus the plan has already begun.
1.1       jmmv      119: 
                    120: This plan will be sent to the tech-userlevel@ mailing list asking for
                    121: comments.  **Two weeks shall be allowed for initial discussion.** Depending
                    122: on the outcome, the plan and/or the software will need to be adjusted
                    123: (which in turn may require significant amounts of time not yet accounted
                    124: for).  I'll be optimistic for now.
                    126: ## Get core@ approval
                    128: By policy, the import of any new software component into src requires core@
                    129: approval.  Even though this will have been discussed at length in
                    130: tech-userlevel@ (as per the previous step), the final decider on this issue
                    131: will be core@.  The corollary of this is that, if no consensus can be
                    132: reached in tech-userlevel@ regarding this plan, core@ will be asked to come
                    133: up with a decision.
                    135: If core@ approves the plan, the next steps shall start immediately.  If
                    136: core@ disagrees, core@ will be asked to provide advice on the corrections
                    137: that should be made before the plan can be approved.
1.4       jmmv      139: It is hard to tell how long this step will last, but possibly account for 2
                    140: to 4 weeks.
1.5       jmmv      142: ## Address feedback as a new release
                    144: Publish a new Kyua release that collects all the feedback from the reviews
                    145: above.
                    147: The list of issues to be addressed can be found by querying the
                    148: [bug tracker]( for the
                    149: [Milestone=Release0.6](
                    150: keyword.  In particular, the following are the issues that have arisen from
                    151: the review:
1.9     ! jmmv      153: * [Issue 36]( Fix the
        !           154:   `help` command to not fail if the configuration file is bogus.
        !           155: 
        !           156: * [Issue 37]( Simplify
        !           157:   the syntax definition of configuration and `Kyuafile` files by removing
        !           158:   the format name.
        !           159: 
1.5       jmmv      160: * [Issue 40]( Provide
1.6       jmmv      161:   manpages instead of an info document.  **DONE**
1.5       jmmv      162: 
1.8       jmmv      163: * [Issue 47]( Implement
                    164:   independent testers, which reduces the amount of C++ code and avoids the
                    165:   need of modifying ``.  **DONE**
1.9     ! jmmv      167: * [Issue 57]( Generalized
        !           168:   support for all metadata properties to plain test programs.  This is to
        !           169:   make plain test programs more versatile by bringing them closer to feature
        !           170:   parity with ATF test programs.
        !           171: 
1.1       jmmv      172: ## Import Kyua into src
1.2       jmmv      174: As the [[introductory page to Kyua|/kyua]] describes, Kyua has been
                    175: available in pkgsrc for a while and can be readily installed and used to
                    176: run the tests from `/usr/tests`.
1.1       jmmv      177: 
                    178: However, because ATF lives in src, and because NetBSD aims to provide the
                    179: best environment for testing "out of the box", Kyua should be imported into
                    180: src just like ATF was.  The major reasons for this, as have been explained
                    181: in the past, are: first, to allow any new deployment of NetBSD to be
                    182: validated right after installation and continuously afterwards; and,
                    183: second, to permit the execution of tests during development without having
                    184: to install any additional software.
                    186: The specific steps to perform this import are:
                    188: 1. Import Lutok into `external/bsd/lutok/`.  This is a shared library that
                    189:    wraps the native Lua C interface in C++.  Lutok was originally part of
                    190:    Kyua, and was split into its own package per the request of some users
                    191:    that found this component useful on its own.
1.8       jmmv      193: 1. Import the Kyua testers into `external/bsd/kyua-testers/`.  This yields two
                    194:    new binaries in `/usr/libexec` (`kyua-atf-tester` and
                    195:    `kyua-plain-tester`) and a bunch of tests in `/usr/tests/kyua-testers`.
                    197: 1. Import the Kyua frontend into `external/bsd/kyua-cli/`.  This yields a
                    198:    new kyua binary in `/usr/bin`, a lot of test programs in
                    199:    `/usr/tests/kyua-cli` (around 100) and some auxiliary files in
                    200:    `/usr/share/kyua`.
1.1       jmmv      201: 
1.6       jmmv      202: 1. Protect all products of Lutok and Kyua with a new `MKKYUA` knob.  **Set
                    203:    `MKKYUA=no` by default.**  Once the ATF tools are removed, the existence
                    204:    of both the `MKATF` and `MKKYUA` knows will probably be confusing.  When
                    205:    that happens, we can revisit this decision by possibly replacing both
                    206:    with an `MKTESTS`.
1.1       jmmv      207: 
                    208: 1. Update `` to generate `Kyuafile`s *in addition to*
                    209:    `Atffile`s.
                    211: There is no real need to do this import in a branch given that this import
1.6       jmmv      212: only adds new functionality without touching existing stuff, and the new
                    213: code is disabled by default.
1.1       jmmv      214: 
                    215: All the preparatory work for the import can be done offline (in about two
                    216: weeks at most, given that I have mot of this ready).  Aside from the code
                    217: changes, this will involve the validation of NetBSD/amd64, NetBSD/i386 and
                    218: NetBSD/macppc builds (which are the ports I have access to).  If you
                    219: consider that some other tricky architecture should be build-tested
                    220: (sparc64?), let me know and I'll include it in the list.
1.8       jmmv      222: The submission to CVS will be prepared locally and performed on a package
                    223: basis (i.e. `lutok`, `kyua-testers` and `kyua-cli`, in this order).  These
                    224: are to be imported separately to simplify the review of the changes and to
                    225: allow me to better test every individual change locally.  There may be an
                    226: arbitrary amount of time between the submission of each package: this
                    227: should not be a problem because these modules are still disabled due to
                    228: `MKKYUA` being set to `no` by default.
1.1       jmmv      229: 
                    230: ## Adjust continuous testing systems to use Kyua
                    232: With `kyua` being part of the release sets, it is possible to adjust the
                    233: continuous test systems to make use of this tool in the test environments
                    234: without having to take any additional step.
                    236: I'll work with gson@ and pgoyette@ to adapt their continuous testing
                    237: machines to use the new built-in `kyua` binary instead of `atf-run` and
                    238: `atf-report`.  I'm planning to do the necessary work to change `anita`
                    239: myself, and I expect to help them deploy the changes to their own systems.
                    240: Because Kyua and ATF will cohexist in the base system at this point,
                    241: migrating the continuous testing systems to Kyua can happen at its own
                    242: peace.
                    244: It might happen that Kyua misses some little detail needed by these
                    245: systems.  In that case, this may require a new release of Kyua and a
                    246: reimport into the tree.  Incremental imports with new features are much
                    247: easier than the original import described in here.  Also, it will be
                    248: possible to cherry-pick any external changes into the tree without a
                    249: reimport (as has often been done in ATF).
1.4       jmmv      251: This step can take a few weeks of time, mostly due to the back and forth
                    252: between different people in different timezones.
1.6       jmmv      254: ## Flip MKKYUA to yet
                    256: The previous steps imported Kyua but didn't enable its build by default so
                    257: that proper testing can be performed by the only people that care.  Once
                    258: basic testing (particularly build testing on a variety of platforms) is
                    259: performed, flip `MKKYUA=yes`.
                    261: ## Update documentation
                    263: The [[Kyua: An introduction for NetBSD users|/kyua]] wiki page will be
                    264: updated to explain how Kyua is bundled in NetBSD and how to use the bundled
                    265: version.
                    267: The [[Creating atf-based tests for NetBSD src|/tutorials/atf]] wiki page
                    268: will be updated to account for the differences in test programs execution
                    269: with Kyua instead of ATF.
                    271: The afterboot(8) and tests(7) manpages will be adjusted to mention `kyua`
                    272: instead of the ATF tools.
1.1       jmmv      274: ## User validation period
                    276: At this stage, **at least one month shall be given to the community** to
                    277: test the new tools and the new test results dashboards.  Collect feedback
1.5       jmmv      278: and address requests as appropriate (possibly by releasing and importing a
                    279: new version of Kyua).
                    281: One important thing to validate is that the results reported by `atf-run`
                    282: and `kyua test` match with each other.  I have already been validating this
                    283: with every public release of Kyua, but it has been a manual process.  To
                    284: make this more reliable, I will set up a continuous testing machine of my
                    285: own in which I will execute `atf-run` and `kyua test` in sequence (possibly
                    286: within anita) and will add an automatic comparison of the exit status of
                    287: each other.
1.1       jmmv      288: 
1.4       jmmv      289: *Because the old `atf-run` and `atf-report` tools have not yet been dropped
                    290: at this point, we can spend as much time as necessary on this phase to get
                    291: things right.*
1.1       jmmv      293: ## Replace atf-run and atf-report with kyua-atf-compat
                    295: Import the Kyua-based `atf-run` and `atf-report` compatibility tools and
                    296: stop building the deprecated versions of these.  The compatibility versions
                    297: are shipped in a separate `kyua-atf-compat` package, and thus this will be
                    298: imported into `external/bsd/kyua-atf-compat/`.
                    300: Once this is done, change `` to not generate `Atffile`s any
                    301: more, as the compatibility tools do not need them.
                    303: ## Drop atf-run and atf-report's code
                    305: Delete the native `atf-run` and `atf-report` utilities from the source tree
                    306: (and from the upstream repository).  This also gets rid of a lot of
                    307: supporting helper code, which makes the various ATF libraries leaner (and
                    308: therefore benefits all test programs!).
                    310: ## Get rid of some ATF wrappers as a proof of concept
                    312: There are some test programs (e.g. the ones in the ipf test suite) that are
                    313: not ATF-based.  To plug them into the NetBSD test suite, we have
                    314: implemented ATF-based wrappers that invoke the original plain tests.  This
                    315: is certainly suboptimal, as this level of indirection hinders readability
                    316: and makes development harder.  (In particular, it prevents some key
                    317: developers from contributing tests in the first place.)
                    319: Because Kyua has the ability to run these "plain" (non-ATF) test programs
                    320: directly, and because a bunch of developers are really looking forward to
                    321: this feature, I will convert a few tests to not include ATF wrappers as a
                    322: proof of concept of this feature.  The ipf tests are probably a good choice
                    323: for this.
                    325: ## Remove atf-run and atf-report compatibility tools
                    327: This is still really far away in the future (needs to happen after NetBSD
                    328: 7.0 is branched), but I'm listing it for completeness of the plan.  The
                    329: idea is to get rid of any ATF compatibility scripts by NetBSD 8.0.  This
                    330: gives end users a full release cycle (that of 7) to adapt to Kyua while at
                    331: the same time being able to use the old-style tools.
                    333: # Possible concerns
                    335: This section attempts to collect the list of possible concerns and/or
                    336: objections that may come up during the review, together with an attempt of
                    337: rebuttal from my side.
                    339: ## Increased compilation time
                    341: The addition of any new component to src increases the build time of the
                    342: whole system.  In the case of Kyua, this increase might be noticeable
                    343: *because of the large amount of test programs provided (roughly 100)*, all
                    344: of which are in C++.
                    346: Note that ATF also had a relatively large codebase and a bunch of tests
                    347: (although not as many).  The "trick" was that the majority of these tests
                    348: were written in shell, and as such they did not increase the build time by
                    349: much.  However, they significantly increased the run time of the test
                    350: suite, and they were less detailed (mostly integration tests, few unit
                    351: tests) than the Kyua tests.
                    353: In order to mitigate this issue, the build of all pieces of Kyua will be
1.5       jmmv      354: protected by `MKKYUA` so that people allergic to C++ can avoid the whole
1.1       jmmv      355: thing.  Even more, there are some other additional provisions described
                    356: below.
                    358: ## Introduction of more C++ code in base
                    360: First of all, why did I use C++?  To make the implementation simpler and
                    361: safer (the RAII programming pattern is really useful in avoiding memory and
                    362: resource leaks with minimal effort).  And C++ is part of the base system
                    363: and a supported language, so there was no reason not to do so.  But that's
1.4       jmmv      364: not the point of this item: if you dislike like C++, this is not going to
                    365: make you think I did right.
1.1       jmmv      366: 
                    367: It's true that, if we count the number of lines, Kyua brings in more C++
                    368: code than what will eventually be dropped by the removal of the ATF tools.
                    369: However, because ATF was also C++, the import of Kyua itself does not make
                    370: the situation significantly worse.
                    372: Additionally, the two compilers we can really use (GCC and LLVM) already
                    373: use, or will soon use, C++ in their code base.  It is unlikely that we will
                    374: be able to remove all C++ support from base anytime soon due to this, while
                    375: at the same time keeping support for all the ports that NetBSD has.
1.8       jmmv      377: Thanks to the
                    378: [testers project](, and
                    379: starting with Kyua 0.6, a lot of the tricky OS-specific code in Kyua has
                    380: been rewritten in plain C.  This paves the way to rewriting parts of the
                    381: now-simpler frontend in C or Lua, if the use of C++ proves to be a serious
                    382: problem in the future.
1.7       jmmv      383: 
                    384: In the short term, the replacement of ATF with Kyua does not make things
                    385: worse: this project just changes one chunk of code with another.
1.1       jmmv      386: 
                    387: ## No need for Lutok as a public library
                    389: Kyua depends on Lutok, which is a C++ interface to Lua.  The code of this
                    390: library was originally part of Kyua, but I split it into its own project
                    391: because some users asked for it.
                    393: If there is no desire to ship Lutok as a shared library in `/usr/lib`, we
                    394: can build Lutok as a static private library and link Kyua against it.
                    395: There is no need to install this as a shared library.

CVSweb for NetBSD wikisrc <> software: FreeBSD-CVSweb