File:  [NetBSD Developer Wiki] / wikisrc / kyua / import.mdwn
Revision 1.14: download - view: text, annotated - select for diffs
Mon Sep 22 15:21:59 2014 UTC (6 years ago) by jmmv
Branches: MAIN
CVS tags: HEAD
Point at Kyua in GitHub, not Google Code.

    1: [[!meta title="Kyua: The way into NetBSD"]]
    2: 
    3: **Contents**
    4: 
    5: [[!toc levels=2]]
    6: 
    7: **Project owner: [Julio Merino](mailto:jmmv@NetBSD.org).**  
    8: **Status: Import of Kyua done as of 2013-02-26.  All new code still
    9: disabled by default.  Call for testing sent on 2013-02-28.**
   10: 
   11: The import of Kyua into NetBSD to replace the deprecated ATF tools is
   12: planned to happen in NetBSD 7.0.  The ATF libraries will remain in place,
   13: and as such no changes will happen to the tests that already live in
   14: `src/tests`.
   15: 
   16: The transition from ATF to Kyua includes steps and tools to offer backwards
   17: compatibility with users that may rely on the ATF tools shipped since
   18: NetBSD 5.0.  These backwards compatibility tools cover the most common use
   19: cases but might not be perfect initially.  Despite these provisions, this
   20: plan should be executed well in advance the creation of the 7.0 branch to
   21: ensure there is enough time in NetBSD-current to flesh out any major
   22: problems.
   23: 
   24: **Before moving on, please read the
   25: [[Kyua: An introduction for NetBSD users|/kyua]] page.  You should be
   26: familiar with the structure of Kyua and its major components to be able to
   27: review this plan.**
   28: 
   29: # Proposed changes
   30: 
   31: The final user visible changes that this project will bring to NetBSD are
   32: as follows.  Because these change the availability of tools that have
   33: already been shipped since 5.0, the modifications are staged to happen
   34: across two major releases:
   35: 
   36: ## For NetBSD 7.0
   37: 
   38: * Addition of the new `kyua` command-line tool.  This provides the runtime
   39:   engine for the test programs (`atf-run`'s functionality) and the ability
   40:   to generate reports of the results in plain text form and HTML form
   41:   (`atf-report`'s functionality).  This tool is able to execute all the
   42:   existing test programs without modifications.
   43: 
   44: * Replacement of the `Atffile`s files in `/usr/tests` with `Kyuafile`s.
   45: 
   46: * Ability to generate HTML reports right form the base system, with such
   47:   reports hooked into the various continuous build systems.
   48: 
   49: * Replacement of the `atf-run` and `atf-report` tools with shell scripts
   50:   that provide backwards compatibility implementations based on `kyua`.
   51:   These can deal with old-style `Atffile`s so that users with custom test
   52:   suites can continue to run them.
   53: 
   54: * Introduction of a new `atf2kyua` script to convert old `Atffile`s to
   55:   `Kyuafile`s.  This is used internally by the compatibility `atf-run`
   56:   script, but can also be invoked by the end user by hand to deal with his
   57:   own test suites.  Could be placed into `libexec` if we do not want to
   58:   make this public.
   59: 
   60: ## For NetBSD 8.0
   61: 
   62: * Removal of the backwards compatibility `atf-run` and `atf-report`
   63:   scripts, as well as the supporting `atf2kyua` tool.
   64: 
   65: # Why?
   66: 
   67: As mentioned in [[another page|/kyua]], Kyua should be seen as ATF 2.x even
   68: though it carries a different name.  It is the evolution of the previous
   69: ATF tools (*the tools only*), but written in a more modular and flexible
   70: way, and with a more reliable codebase.  Therefore, you should consider
   71: this project as the update of ATF to a newer version.
   72: 
   73: The `atf-run` and `atf-report` tools have effectively been in maintenance
   74: mode for over a year already.  None of the desired features (see the list
   75: of open PRs) have been implemented on top of them, mostly because doing so
   76: is building upon a broken implementation.  Additionally, several developers
   77: have had to implement their own test results dashboards due to
   78: defficiencies in `atf-report`, effectively reinventing the wheel.
   79: 
   80: This update will permit the real removal of the obsolete tools, thus
   81: allowing us to build additional features on top of Kyua without having to
   82: worry about being compatible with `atf-run` (and thus adjusting this tool
   83: to behave in the same manner).
   84: 
   85: Some possible answers to "Why not?" can be found later in this page.
   86: 
   87: ## And why now?
   88: 
   89: Kyua has existed for almost 3 years already, so you may be wondering why
   90: this import is being proposed now.  The major reasons are:
   91: 
   92: * Since 0.5, Kyua has now feature-parity with ATF.  Replacing the ATF tools
   93:   with Kyua should not introduce functionality regressions.
   94: 
   95: * NetBSD-6 has been branched relatively recently, so we can expect NetBSD-7
   96:   to be far away enough in the future to provide plenty of time to
   97:   stabilize Kyua in NetBSD-current.
   98: 
   99: * The existence of the deprecated `atf-run` and `atf-report` tools hinders
  100:   the development of new features that require changes to the inteface of
  101:   ATF test programs.  As an example: a highly requested feature is the
  102:   ability to change the timeout settings of the test programs (for the
  103:   benefit of old, slow platforms); doing this in the ATF codebase is
  104:   tricky.
  105: 
  106: * Some developers continue to add old-style tests to the tree.  While this
  107:   is suboptimal and against the policy dictated by core@, Kyua brings in a
  108:   mechanism to allow running these tests unmodified (i.e. without adding
  109:   any ATF calls in their code).  This, in turn, will let such developers
  110:   add their tests more easily, and thus increase the tests coverage.
  111: 
  112: # The plan
  113: 
  114: This section details what the transition plan looks like and an estimate
  115: timeline.  As all things that depend on free time (opensource software
  116: hacking, in this case), take the estimates with a grain of salt.
  117: 
  118: ## Discuss this plan in tech-userlevel@
  119: 
  120: If you are here it's possibly because a review request for this plan has
  121: already been published and thus the plan has already begun.
  122: 
  123: This plan will be sent to the tech-userlevel@ mailing list asking for
  124: comments.  **Two weeks shall be allowed for initial discussion.** Depending
  125: on the outcome, the plan and/or the software will need to be adjusted
  126: (which in turn may require significant amounts of time not yet accounted
  127: for).  I'll be optimistic for now.
  128: 
  129: ## Get core@ approval
  130: 
  131: By policy, the import of any new software component into src requires core@
  132: approval.  Even though this will have been discussed at length in
  133: tech-userlevel@ (as per the previous step), the final decider on this issue
  134: will be core@.  The corollary of this is that, if no consensus can be
  135: reached in tech-userlevel@ regarding this plan, core@ will be asked to come
  136: up with a decision.
  137: 
  138: If core@ approves the plan, the next steps shall start immediately.  If
  139: core@ disagrees, core@ will be asked to provide advice on the corrections
  140: that should be made before the plan can be approved.
  141: 
  142: It is hard to tell how long this step will last, but possibly account for 2
  143: to 4 weeks.
  144: 
  145: ## Address feedback as a new release
  146: 
  147: Publish a new Kyua release that collects all the feedback from the reviews
  148: above.
  149: 
  150: The list of issues to be addressed can be found by querying the
  151: [bug tracker](https://github.com/jmmv/kyua/issues).
  152: In particular, the following are the issues that have arisen from
  153: the review:
  154: 
  155: * [Issue 36](https://github.com/jmmv/kyua/issues/36): Fix the
  156:   `help` command to not fail if the configuration file is bogus.  **DONE**
  157: 
  158: * [Issue 37](https://github.com/jmmv/kyua/issues/37): Simplify
  159:   the syntax definition of configuration and `Kyuafile` files by removing
  160:   the format name.  **DONE**
  161: 
  162: * [Issue 40](https://github.com/jmmv/kyua/issues/40): Provide
  163:   manpages instead of an info document.  **DONE**
  164: 
  165: * [Issue 47](https://github.com/jmmv/kyua/issues/47): Implement
  166:   independent testers, which reduces the amount of C++ code and avoids the
  167:   need of modifying `bsd.dep.mk`.  **DONE**
  168: 
  169: * [Issue 57](https://github.com/jmmv/kyua/issues/57): Generalized
  170:   support for all metadata properties to plain test programs.  This is to
  171:   make plain test programs more versatile by bringing them closer to feature
  172:   parity with ATF test programs.  **DONE**
  173: 
  174: ## Import Kyua into src
  175: 
  176: As the [[introductory page to Kyua|/kyua]] describes, Kyua has been
  177: available in pkgsrc for a while and can be readily installed and used to
  178: run the tests from `/usr/tests`.
  179: 
  180: However, because ATF lives in src, and because NetBSD aims to provide the
  181: best environment for testing "out of the box", Kyua should be imported into
  182: src just like ATF was.  The major reasons for this, as have been explained
  183: in the past, are: first, to allow any new deployment of NetBSD to be
  184: validated right after installation and continuously afterwards; and,
  185: second, to permit the execution of tests during development without having
  186: to install any additional software.
  187: 
  188: The specific steps to perform this import are:
  189: 
  190: 1. Import Lutok into `external/bsd/lutok/`.  This is a shared library that
  191:    wraps the native Lua C interface in C++.  Lutok was originally part of
  192:    Kyua, and was split into its own package per the request of some users
  193:    that found this component useful on its own.
  194: 
  195: 1. Import the Kyua testers into `external/bsd/kyua-testers/`.  This yields two
  196:    new binaries in `/usr/libexec` (`kyua-atf-tester` and
  197:    `kyua-plain-tester`) and a bunch of tests in `/usr/tests/kyua-testers`.
  198: 
  199: 1. Import the Kyua frontend into `external/bsd/kyua-cli/`.  This yields a
  200:    new kyua binary in `/usr/bin`, a lot of test programs in
  201:    `/usr/tests/kyua-cli` (around 100) and some auxiliary files in
  202:    `/usr/share/kyua`.
  203: 
  204: 1. Protect all products of Lutok and Kyua with a new `MKKYUA` knob.  **Set
  205:    `MKKYUA=no` by default.**  Once the ATF tools are removed, the existence
  206:    of both the `MKATF` and `MKKYUA` knows will probably be confusing.  When
  207:    that happens, we can revisit this decision by possibly replacing both
  208:    with an `MKTESTS`.
  209: 
  210: 1. Update `bsd.test.mk` to generate `Kyuafile`s *in addition to*
  211:    `Atffile`s.
  212: 
  213: There is no real need to do this import in a branch given that this import
  214: only adds new functionality without touching existing stuff, and the new
  215: code is disabled by default.
  216: 
  217: All the preparatory work for the import can be done offline (in about two
  218: weeks at most, given that I have mot of this ready).  Aside from the code
  219: changes, this will involve the validation of NetBSD/amd64, NetBSD/i386 and
  220: NetBSD/macppc builds (which are the ports I have access to).  If you
  221: consider that some other tricky architecture should be build-tested
  222: (sparc64?), let me know and I'll include it in the list.
  223: 
  224: The submission to CVS will be prepared locally and performed on a package
  225: basis (i.e. `lutok`, `kyua-testers` and `kyua-cli`, in this order).  These
  226: are to be imported separately to simplify the review of the changes and to
  227: allow me to better test every individual change locally.  There may be an
  228: arbitrary amount of time between the submission of each package: this
  229: should not be a problem because these modules are still disabled due to
  230: `MKKYUA` being set to `no` by default.
  231: 
  232: ## Adjust continuous testing systems to use Kyua
  233: 
  234: With `kyua` being part of the release sets, it is possible to adjust the
  235: continuous test systems to make use of this tool in the test environments
  236: without having to take any additional step.
  237: 
  238: I'll work with gson@ and pgoyette@ to adapt their continuous testing
  239: machines to use the new built-in `kyua` binary instead of `atf-run` and
  240: `atf-report`.  I'm planning to do the necessary work to change `anita`
  241: myself, and I expect to help them deploy the changes to their own systems.
  242: Because Kyua and ATF will cohexist in the base system at this point,
  243: migrating the continuous testing systems to Kyua can happen at its own
  244: peace.
  245: 
  246: It might happen that Kyua misses some little detail needed by these
  247: systems.  In that case, this may require a new release of Kyua and a
  248: reimport into the tree.  Incremental imports with new features are much
  249: easier than the original import described in here.  Also, it will be
  250: possible to cherry-pick any external changes into the tree without a
  251: reimport (as has often been done in ATF).
  252: 
  253: This step can take a few weeks of time, mostly due to the back and forth
  254: between different people in different timezones.
  255: 
  256: ## Flip MKKYUA to yes
  257: 
  258: The previous steps imported Kyua but didn't enable its build by default so
  259: that proper testing can be performed by the only people that care.  Once
  260: basic testing (particularly build testing on a variety of platforms) is
  261: performed, flip `MKKYUA=yes`.
  262: 
  263: ## Update documentation
  264: 
  265: The [[Kyua: An introduction for NetBSD users|/kyua]] wiki page will be
  266: updated to explain how Kyua is bundled in NetBSD and how to use the bundled
  267: version.
  268: 
  269: The [[Creating atf-based tests for NetBSD src|/tutorials/atf]] wiki page
  270: will be updated to account for the differences in test programs execution
  271: with Kyua instead of ATF.
  272: 
  273: The afterboot(8) and tests(7) manpages will be adjusted to mention `kyua`
  274: instead of the ATF tools.
  275: 
  276: ## User validation period
  277: 
  278: At this stage, **at least one month shall be given to the community** to
  279: test the new tools and the new test results dashboards.  Collect feedback
  280: and address requests as appropriate (possibly by releasing and importing a
  281: new version of Kyua).
  282: 
  283: One important thing to validate is that the results reported by `atf-run`
  284: and `kyua test` match with each other.  I have already been validating this
  285: with every public release of Kyua, but it has been a manual process.  To
  286: make this more reliable, I will set up a continuous testing machine of my
  287: own in which I will execute `atf-run` and `kyua test` in sequence (possibly
  288: within anita) and will add an automatic comparison of the exit status of
  289: each other.
  290: 
  291: *Because the old `atf-run` and `atf-report` tools have not yet been dropped
  292: at this point, we can spend as much time as necessary on this phase to get
  293: things right.*
  294: 
  295: ## Replace atf-run and atf-report with kyua-atf-compat
  296: 
  297: Import the Kyua-based `atf-run` and `atf-report` compatibility tools and
  298: stop building the deprecated versions of these.  The compatibility versions
  299: are shipped in a separate `kyua-atf-compat` package, and thus this will be
  300: imported into `external/bsd/kyua-atf-compat/`.
  301: 
  302: Once this is done, change `bsd.test.mk` to not generate `Atffile`s any
  303: more, as the compatibility tools do not need them.
  304: 
  305: ## Drop atf-run and atf-report's code
  306: 
  307: Delete the native `atf-run` and `atf-report` utilities from the source tree
  308: (and from the upstream repository).  This also gets rid of a lot of
  309: supporting helper code, which makes the various ATF libraries leaner (and
  310: therefore benefits all test programs!).
  311: 
  312: ## Get rid of some ATF wrappers as a proof of concept
  313: 
  314: There are some test programs (e.g. the ones in the ipf test suite) that are
  315: not ATF-based.  To plug them into the NetBSD test suite, we have
  316: implemented ATF-based wrappers that invoke the original plain tests.  This
  317: is certainly suboptimal, as this level of indirection hinders readability
  318: and makes development harder.  (In particular, it prevents some key
  319: developers from contributing tests in the first place.)
  320: 
  321: Because Kyua has the ability to run these "plain" (non-ATF) test programs
  322: directly, and because a bunch of developers are really looking forward to
  323: this feature, I will convert a few tests to not include ATF wrappers as a
  324: proof of concept of this feature.  The ipf tests are probably a good choice
  325: for this.
  326: 
  327: ## Remove atf-run and atf-report compatibility tools
  328: 
  329: This is still really far away in the future (needs to happen after NetBSD
  330: 7.0 is branched), but I'm listing it for completeness of the plan.  The
  331: idea is to get rid of any ATF compatibility scripts by NetBSD 8.0.  This
  332: gives end users a full release cycle (that of 7) to adapt to Kyua while at
  333: the same time being able to use the old-style tools.
  334: 
  335: # Possible concerns
  336: 
  337: This section attempts to collect the list of possible concerns and/or
  338: objections that may come up during the review, together with an attempt of
  339: rebuttal from my side.
  340: 
  341: ## Increased compilation time
  342: 
  343: The addition of any new component to src increases the build time of the
  344: whole system.  In the case of Kyua, this increase might be noticeable
  345: *because of the large amount of test programs provided (roughly 100)*, all
  346: of which are in C++.
  347: 
  348: Note that ATF also had a relatively large codebase and a bunch of tests
  349: (although not as many).  The "trick" was that the majority of these tests
  350: were written in shell, and as such they did not increase the build time by
  351: much.  However, they significantly increased the run time of the test
  352: suite, and they were less detailed (mostly integration tests, few unit
  353: tests) than the Kyua tests.
  354: 
  355: In order to mitigate this issue, the build of all pieces of Kyua will be
  356: protected by `MKKYUA` so that people allergic to C++ can avoid the whole
  357: thing.  Even more, there are some other additional provisions described
  358: below.
  359: 
  360: ## Introduction of more C++ code in base
  361: 
  362: First of all, why did I use C++?  To make the implementation simpler and
  363: safer (the RAII programming pattern is really useful in avoiding memory and
  364: resource leaks with minimal effort).  And C++ is part of the base system
  365: and a supported language, so there was no reason not to do so.  But that's
  366: not the point of this item: if you dislike like C++, this is not going to
  367: make you think I did right.
  368: 
  369: It's true that, if we count the number of lines, Kyua brings in more C++
  370: code than what will eventually be dropped by the removal of the ATF tools.
  371: However, because ATF was also C++, the import of Kyua itself does not make
  372: the situation significantly worse.
  373: 
  374: Additionally, the two compilers we can really use (GCC and LLVM) already
  375: use, or will soon use, C++ in their code base.  It is unlikely that we will
  376: be able to remove all C++ support from base anytime soon due to this, while
  377: at the same time keeping support for all the ports that NetBSD has.
  378: 
  379: Thanks to the
  380: [testers project](https://github.com/jmmv/kyua/wiki/Testers-design), and
  381: starting with Kyua 0.6, a lot of the tricky OS-specific code in Kyua has
  382: been rewritten in plain C.  This paves the way to rewriting parts of the
  383: now-simpler frontend in C or Lua, if the use of C++ proves to be a serious
  384: problem in the future.
  385: 
  386: In the short term, the replacement of ATF with Kyua does not make things
  387: worse: this project just changes one chunk of code with another.
  388: 
  389: ## No need for Lutok as a public library
  390: 
  391: Kyua depends on Lutok, which is a C++ interface to Lua.  The code of this
  392: library was originally part of Kyua, but I split it into its own project
  393: because some users asked for it.
  394: 
  395: If there is no desire to ship Lutok as a shared library in `/usr/lib`, we
  396: can build Lutok as a static private library and link Kyua against it.
  397: There is no need to install this as a shared library.

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb