Diff for /wikisrc/kyua/import.mdwn between versions 1.3 and 1.10

version 1.3, 2012/09/01 19:58:18 version 1.10, 2013/02/10 01:12:33
Line 2 Line 2
 [[!toc levels=2]]  [[!toc levels=2]]
   
 **Project owner: [Julio Merino](mailto:jmmv@NetBSD.org).**    **Project owner: [Julio Merino](mailto:jmmv@NetBSD.org).**  
 **Status: draft as of 2012-08-31; not even sent for review yet.**  **Status: Approved by core as of 2013-02-09.  Import of Lutok and a new
   release of ATF to begin soon.**
   
 The import of Kyua into NetBSD to replace the deprecated ATF tools is  The import of Kyua into NetBSD to replace the deprecated ATF tools is
 planned to happen in NetBSD 7.0.  The ATF libraries will remain in place,  planned to happen in NetBSD 7.0.  The ATF libraries will remain in place,
Line 114  hacking, in this case), take the estimat Line 115  hacking, in this case), take the estimat
 ## Discuss this plan in tech-userlevel@  ## Discuss this plan in tech-userlevel@
   
 If you are here it's possibly because a review request for this plan has  If you are here it's possibly because a review request for this plan has
 already been published and thus the plan has already begun.  If  already been published and thus the plan has already begun.
 not... well, hold off for a little bit ;-)  
   
 This plan will be sent to the tech-userlevel@ mailing list asking for  This plan will be sent to the tech-userlevel@ mailing list asking for
 comments.  **Two weeks shall be allowed for initial discussion.** Depending  comments.  **Two weeks shall be allowed for initial discussion.** Depending
Line 136  If core@ approves the plan, the next ste Line 136  If core@ approves the plan, the next ste
 core@ disagrees, core@ will be asked to provide advice on the corrections  core@ disagrees, core@ will be asked to provide advice on the corrections
 that should be made before the plan can be approved.  that should be made before the plan can be approved.
   
   It is hard to tell how long this step will last, but possibly account for 2
   to 4 weeks.
   
   ## Address feedback as a new release
   
   Publish a new Kyua release that collects all the feedback from the reviews
   above.
   
   The list of issues to be addressed can be found by querying the
   [bug tracker](http://code.google.com/p/kyua/issues/list) for the
   [Milestone=Release0.6](http://code.google.com/p/kyua/issues/list?can=2&q=Milestone%3DRelease0.6&colspec=ID+Type+Status+Priority+Milestone+Owner+Summary&cells=tiles)
   keyword.  In particular, the following are the issues that have arisen from
   the review:
   
   * [Issue 36](http://code.google.com/p/kyua/issues/detail?id=36): Fix the
     `help` command to not fail if the configuration file is bogus.  **DONE**
   
   * [Issue 37](http://code.google.com/p/kyua/issues/detail?id=37): Simplify
     the syntax definition of configuration and `Kyuafile` files by removing
     the format name.  **DONE**
   
   * [Issue 40](http://code.google.com/p/kyua/issues/detail?id=40): Provide
     manpages instead of an info document.  **DONE**
   
   * [Issue 47](http://code.google.com/p/kyua/issues/detail?id=47): Implement
     independent testers, which reduces the amount of C++ code and avoids the
     need of modifying `bsd.dep.mk`.  **DONE**
   
   * [Issue 57](http://code.google.com/p/kyua/issues/detail?id=57): Generalized
     support for all metadata properties to plain test programs.  This is to
     make plain test programs more versatile by bringing them closer to feature
     parity with ATF test programs.  **DONE**
   
 ## Import Kyua into src  ## Import Kyua into src
   
 As the [[introductory page to Kyua|/kyua]] describes, Kyua has been  As the [[introductory page to Kyua|/kyua]] describes, Kyua has been
Line 157  The specific steps to perform this impor Line 190  The specific steps to perform this impor
    Kyua, and was split into its own package per the request of some users     Kyua, and was split into its own package per the request of some users
    that found this component useful on its own.     that found this component useful on its own.
   
 1. Import Kyua into `external/bsd/kyua-cli/`.  This yields a new kyua  1. Import the Kyua testers into `external/bsd/kyua-testers/`.  This yields two
    binary in `/usr/bin`, a lot of test programs in `/usr/tests/kyua-cli`     new binaries in `/usr/libexec` (`kyua-atf-tester` and
    (around 100) and some auxiliary files in `/usr/share/kyua`.     `kyua-plain-tester`) and a bunch of tests in `/usr/tests/kyua-testers`.
   
 1. Protect all products of Lutok and Kyua with the existing `MKATF` knob.  1. Import the Kyua frontend into `external/bsd/kyua-cli/`.  This yields a
    If desired, these could instead be protected by a new and separate     new kyua binary in `/usr/bin`, a lot of test programs in
    `MKKYUA` knob, but because the transition is only temporary, I do not     `/usr/tests/kyua-cli` (around 100) and some auxiliary files in
    think it's worth the effort.     `/usr/share/kyua`.
   
   1. Protect all products of Lutok and Kyua with a new `MKKYUA` knob.  **Set
      `MKKYUA=no` by default.**  Once the ATF tools are removed, the existence
      of both the `MKATF` and `MKKYUA` knows will probably be confusing.  When
      that happens, we can revisit this decision by possibly replacing both
      with an `MKTESTS`.
   
 1. Update `bsd.test.mk` to generate `Kyuafile`s *in addition to*  1. Update `bsd.test.mk` to generate `Kyuafile`s *in addition to*
    `Atffile`s.     `Atffile`s.
   
 There is no real need to do this import in a branch given that this import  There is no real need to do this import in a branch given that this import
 only adds new functionality without touching existing stuff.  only adds new functionality without touching existing stuff, and the new
   code is disabled by default.
   
 All the preparatory work for the import can be done offline (in about two  All the preparatory work for the import can be done offline (in about two
 weeks at most, given that I have mot of this ready).  Aside from the code  weeks at most, given that I have mot of this ready).  Aside from the code
Line 179  NetBSD/macppc builds (which are the port Line 219  NetBSD/macppc builds (which are the port
 consider that some other tricky architecture should be build-tested  consider that some other tricky architecture should be build-tested
 (sparc64?), let me know and I'll include it in the list.  (sparc64?), let me know and I'll include it in the list.
   
 The submission step to CVS, once all the code changes are ready locally,  The submission to CVS will be prepared locally and performed on a package
 and any post-commit validation checks will take a few hours.  Any build  basis (i.e. `lutok`, `kyua-testers` and `kyua-cli`, in this order).  These
 breakage should be addressed in a timely manner (and can possibly be  are to be imported separately to simplify the review of the changes and to
 worked-around by setting `MKATF=no`).  allow me to better test every individual change locally.  There may be an
   arbitrary amount of time between the submission of each package: this
 ## Update documentation  should not be a problem because these modules are still disabled due to
   `MKKYUA` being set to `no` by default.
 The [[Kyua: An introduction for NetBSD users|/kyua]] wiki page will be  
 updated to explain how Kyua is bundled in NetBSD and how to use the bundled  
 version.  
   
 The [[Creating atf-based tests for NetBSD src|/tutorials/atf]] wiki page  
 will be updated to account for the differences in test programs execution  
 with Kyua instead of ATF.  
   
 The afterboot(8) and tests(7) manpages will be adjusted to mention `kyua`  
 instead of the ATF tools.  
   
 ## Adjust continuous testing systems to use Kyua  ## Adjust continuous testing systems to use Kyua
   
Line 218  easier than the original import describe Line 248  easier than the original import describe
 possible to cherry-pick any external changes into the tree without a  possible to cherry-pick any external changes into the tree without a
 reimport (as has often been done in ATF).  reimport (as has often been done in ATF).
   
   This step can take a few weeks of time, mostly due to the back and forth
   between different people in different timezones.
   
   ## Flip MKKYUA to yet
   
   The previous steps imported Kyua but didn't enable its build by default so
   that proper testing can be performed by the only people that care.  Once
   basic testing (particularly build testing on a variety of platforms) is
   performed, flip `MKKYUA=yes`.
   
   ## Update documentation
   
   The [[Kyua: An introduction for NetBSD users|/kyua]] wiki page will be
   updated to explain how Kyua is bundled in NetBSD and how to use the bundled
   version.
   
   The [[Creating atf-based tests for NetBSD src|/tutorials/atf]] wiki page
   will be updated to account for the differences in test programs execution
   with Kyua instead of ATF.
   
   The afterboot(8) and tests(7) manpages will be adjusted to mention `kyua`
   instead of the ATF tools.
   
 ## User validation period  ## User validation period
   
 At this stage, **at least one month shall be given to the community** to  At this stage, **at least one month shall be given to the community** to
 test the new tools and the new test results dashboards.  Collect feedback  test the new tools and the new test results dashboards.  Collect feedback
 and address requests as appropriate.  and address requests as appropriate (possibly by releasing and importing a
   new version of Kyua).
   
   One important thing to validate is that the results reported by `atf-run`
   and `kyua test` match with each other.  I have already been validating this
   with every public release of Kyua, but it has been a manual process.  To
   make this more reliable, I will set up a continuous testing machine of my
   own in which I will execute `atf-run` and `kyua test` in sequence (possibly
   within anita) and will add an automatic comparison of the exit status of
   each other.
   
   *Because the old `atf-run` and `atf-report` tools have not yet been dropped
   at this point, we can spend as much time as necessary on this phase to get
   things right.*
   
 ## Replace atf-run and atf-report with kyua-atf-compat  ## Replace atf-run and atf-report with kyua-atf-compat
   
Line 285  suite, and they were less detailed (most Line 351  suite, and they were less detailed (most
 tests) than the Kyua tests.  tests) than the Kyua tests.
   
 In order to mitigate this issue, the build of all pieces of Kyua will be  In order to mitigate this issue, the build of all pieces of Kyua will be
 protected by `MKATF` so that people allergic to C++ can avoid the whole  protected by `MKKYUA` so that people allergic to C++ can avoid the whole
 thing.  Even more, there are some other additional provisions described  thing.  Even more, there are some other additional provisions described
 below.  below.
   
Line 295  First of all, why did I use C++?  To mak Line 361  First of all, why did I use C++?  To mak
 safer (the RAII programming pattern is really useful in avoiding memory and  safer (the RAII programming pattern is really useful in avoiding memory and
 resource leaks with minimal effort).  And C++ is part of the base system  resource leaks with minimal effort).  And C++ is part of the base system
 and a supported language, so there was no reason not to do so.  But that's  and a supported language, so there was no reason not to do so.  But that's
 not the point of this item: if you don't like C++, this is not going to  not the point of this item: if you dislike like C++, this is not going to
 convince you otherwise.  make you think I did right.
   
 It's true that, if we count the number of lines, Kyua brings in more C++  It's true that, if we count the number of lines, Kyua brings in more C++
 code than what will eventually be dropped by the removal of the ATF tools.  code than what will eventually be dropped by the removal of the ATF tools.
Line 308  use, or will soon use, C++ in their code Line 374  use, or will soon use, C++ in their code
 be able to remove all C++ support from base anytime soon due to this, while  be able to remove all C++ support from base anytime soon due to this, while
 at the same time keeping support for all the ports that NetBSD has.  at the same time keeping support for all the ports that NetBSD has.
   
 Long term, if the use of C++ proves to be a problem, there are a few things  Thanks to the
 that can be done to slowly get rid of it that have been floating my mind  [testers project](http://code.google.com/p/kyua/wiki/TestersDesign), and
 recently.  The first is the rewrite of the performance-critical parts of  starting with Kyua 0.6, a lot of the tricky OS-specific code in Kyua has
 Kyua in plain C.  This would involve splitting the runtime engine of a  been rewritten in plain C.  This paves the way to rewriting parts of the
 single test case in its own binary (which, for other reasons may be a good  now-simpler frontend in C or Lua, if the use of C++ proves to be a serious
 idea on its own).  The second is the rewrite of most user-interface code in  problem in the future.
 Lua, which in itself would bring some extensibility advantages to the  
 program.  Both options will be investigated when the time permits.  In the  In the short term, the replacement of ATF with Kyua does not make things
 meantime, the replacement of ATF with Kyua does not make things worse; it  worse: this project just changes one chunk of code with another.
 just changes one chunk of code with another.  
   
 ## No need for Lutok as a public library  ## No need for Lutok as a public library
   

Removed from v.1.3  
changed lines
  Added in v.1.10


CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb