Annotation of wikisrc/kyua/import.mdwn, revision 1.1

1.1     ! jmmv        1: [[!meta title="Kyua: The way into NetBSD"]]
        !             2: [[!toc levels=2]]
        !             3: 
        !             4: **Project owner: [Julio Merino](mailto:jmmv@NetBSD.org).**  
        !             5: **Status: draft as of 2012-08-31; not even sent for review yet.**
        !             6: 
        !             7: The import of Kyua into NetBSD to replace the deprecated ATF tools is
        !             8: planned to happen in NetBSD 7.0.  The ATF libraries will remain in place,
        !             9: and as such no changes will happen to the tests that already live in
        !            10: `src/tests`.
        !            11: 
        !            12: The transition from ATF to Kyua includes steps and tools to offer backwards
        !            13: compatibility with users that may rely on the ATF tools shipped since
        !            14: NetBSD 5.0.  These backwards compatibility tools cover the most common use
        !            15: cases but might not be perfect initially.  Despite these provisions, this
        !            16: plan should be executed well in advance the creation of the 7.0 branch to
        !            17: ensure there is enough time in NetBSD-current to flesh out any major
        !            18: problems.
        !            19: 
        !            20: Before moving on, please read the
        !            21: [Kyua: An introduction for NetBSD users](kyua) page.  You should be
        !            22: familiar with the structure of Kyua and its major components to be able to
        !            23: review this plan.
        !            24: 
        !            25: # Proposed changes
        !            26: 
        !            27: The final user visible changes that this project will bring to NetBSD are
        !            28: as follows.  Because these change the availability of tools that have
        !            29: already been shipped since 5.0, the modifications are staged to happen
        !            30: across two major releases:
        !            31: 
        !            32: ## For NetBSD 7.0
        !            33: 
        !            34: * Addition of the new `kyua` command-line tool.  This provides the runtime
        !            35:   engine for the test programs (`atf-run`'s functionality) and the ability
        !            36:   to generate reports of the results in plain text form and HTML form
        !            37:   (`atf-report`'s functionality).  This tool is able to execute all the
        !            38:   existing test programs without modifications.
        !            39: 
        !            40: * Replacement of the `Atffile`s files in `/usr/tests` with `Kyuafile`s.
        !            41: 
        !            42: * Ability to generate HTML reports right form the base system, with such
        !            43:   reports hooked into the various continuous build systems.
        !            44: 
        !            45: * Replacement of the `atf-run` and `atf-report` tools with shell scripts
        !            46:   that provide backwards compatibility implementations based on `kyua`.
        !            47:   These can deal with old-style `Atffile`s so that users with custom test
        !            48:   suites can continue to run them.
        !            49: 
        !            50: * Introduction of a new `atf2kyua` script to convert old `Atffile`s to
        !            51:   `Kyuafile`s.  This is used internally by the compatibility `atf-run`
        !            52:   script, but can also be invoked by the end user by hand to deal with his
        !            53:   own test suites.  Could be placed into `libexec` if we do not want to
        !            54:   make this public.
        !            55: 
        !            56: ## For NetBSD 8.0
        !            57: 
        !            58: * Removal of the backwards compatibility `atf-run` and `atf-report`
        !            59:   scripts, as well as the supporting `atf2kyua` tool.
        !            60: 
        !            61: # Why now?
        !            62: 
        !            63: Kyua has existed for almost 3 years already, so you may be wondering why
        !            64: this import is being proposed now.  The major reasons are:
        !            65: 
        !            66: * Since 0.5, Kyua has now feature-parity with ATF.  Replacing the ATF tools
        !            67:   with Kyua should not introduce functionality regressions.
        !            68: 
        !            69: * NetBSD-6 has been branched relatively recently, so we can expect NetBSD-7
        !            70:   to be far away enough in the future to provide plenty of time to
        !            71:   stabilize Kyua in NetBSD-current.
        !            72: 
        !            73: * The existence of the deprecated `atf-run` and `atf-report` tools hinders
        !            74:   the development of new features that require changes to the inteface of
        !            75:   ATF test programs.  As an example: a highly requested feature is the
        !            76:   ability to change the timeout settings of the test programs (for the
        !            77:   benefit of old, slow platforms); doing this in the ATF codebase is
        !            78:   tricky.
        !            79: 
        !            80: * Some developers continue to add old-style tests to the tree.  While this
        !            81:   is suboptimal and against the policy dictated by core@, Kyua brings in a
        !            82:   mechanism to allow running these tests unmodified (i.e. without adding
        !            83:   any ATF calls in their code).  This, in turn, will let such developers
        !            84:   add their tests more easily, and thus increase the tests coverage.
        !            85: 
        !            86: # The plan
        !            87: 
        !            88: This section details what the transition plan looks like and an estimate
        !            89: timeline.  As all things that depend on free time (opensource software
        !            90: hacking, in this case), take the estimates with a grain of salt.
        !            91: 
        !            92: ## Discuss this plan in tech-userlevel@
        !            93: 
        !            94: If you are here it's possibly because a review request for this plan has
        !            95: already been published and thus the plan has already begun.  If
        !            96: not... well, hold off for a little bit ;-)
        !            97: 
        !            98: This plan will be sent to the tech-userlevel@ mailing list asking for
        !            99: comments.  **Two weeks shall be allowed for initial discussion.** Depending
        !           100: on the outcome, the plan and/or the software will need to be adjusted
        !           101: (which in turn may require significant amounts of time not yet accounted
        !           102: for).  I'll be optimistic for now.
        !           103: 
        !           104: ## Get core@ approval
        !           105: 
        !           106: By policy, the import of any new software component into src requires core@
        !           107: approval.  Even though this will have been discussed at length in
        !           108: tech-userlevel@ (as per the previous step), the final decider on this issue
        !           109: will be core@.  The corollary of this is that, if no consensus can be
        !           110: reached in tech-userlevel@ regarding this plan, core@ will be asked to come
        !           111: up with a decision.
        !           112: 
        !           113: If core@ approves the plan, the next steps shall start immediately.  If
        !           114: core@ disagrees, core@ will be asked to provide advice on the corrections
        !           115: that should be made before the plan can be approved.
        !           116: 
        !           117: ## Import Kyua into src
        !           118: 
        !           119: As the [introductory page to Kyua](kyua) describes, Kyua has been available
        !           120: in pkgsrc for a while and can be readily installed and used to run the
        !           121: tests from `/usr/tests`.
        !           122: 
        !           123: However, because ATF lives in src, and because NetBSD aims to provide the
        !           124: best environment for testing "out of the box", Kyua should be imported into
        !           125: src just like ATF was.  The major reasons for this, as have been explained
        !           126: in the past, are: first, to allow any new deployment of NetBSD to be
        !           127: validated right after installation and continuously afterwards; and,
        !           128: second, to permit the execution of tests during development without having
        !           129: to install any additional software.
        !           130: 
        !           131: The specific steps to perform this import are:
        !           132: 
        !           133: 1. Import Lutok into `external/bsd/lutok/`.  This is a shared library that
        !           134:    wraps the native Lua C interface in C++.  Lutok was originally part of
        !           135:    Kyua, and was split into its own package per the request of some users
        !           136:    that found this component useful on its own.
        !           137: 
        !           138: 1. Import Kyua into `external/bsd/kyua-cli/`.  This yields a new kyua
        !           139:    binary in `/usr/bin`, a lot of test programs in `/usr/tests/kyua-cli`
        !           140:    (around 100) and some auxiliary files in `/usr/share/kyua`.
        !           141: 
        !           142: 1. Protect all products of Lutok and Kyua with the existing `MKATF` knob.
        !           143:    If desired, these could instead be protected by a new and separate
        !           144:    `MKKYUA` knob, but because the transition is only temporary, I do not
        !           145:    think it's worth the effort.
        !           146: 
        !           147: 1. Update `bsd.test.mk` to generate `Kyuafile`s *in addition to*
        !           148:    `Atffile`s.
        !           149: 
        !           150: There is no real need to do this import in a branch given that this import
        !           151: only adds new functionality without touching existing stuff.
        !           152: 
        !           153: All the preparatory work for the import can be done offline (in about two
        !           154: weeks at most, given that I have mot of this ready).  Aside from the code
        !           155: changes, this will involve the validation of NetBSD/amd64, NetBSD/i386 and
        !           156: NetBSD/macppc builds (which are the ports I have access to).  If you
        !           157: consider that some other tricky architecture should be build-tested
        !           158: (sparc64?), let me know and I'll include it in the list.
        !           159: 
        !           160: The submission step to CVS, once all the code changes are ready locally,
        !           161: and any post-commit validation checks will take a few hours.  Any build
        !           162: breakage should be addressed in a timely manner (and can possibly be
        !           163: worked-around by setting `MKATF=no`).
        !           164: 
        !           165: ## Update documentation
        !           166: 
        !           167: The [Kyua: An introduction for NetBSD users](kyua) wiki page will be
        !           168: updated to explain how Kyua is bundled in NetBSD and how to use the bundled
        !           169: version.
        !           170: 
        !           171: The [Creating atf-based tests for NetBSD src](atf) wiki page will be
        !           172: updated to account for the differences in test programs execution with Kyua
        !           173: instead of ATF.
        !           174: 
        !           175: The afterboot(8) and tests(7) manpages will be adjusted to mention `kyua`
        !           176: instead of the ATF tools.
        !           177: 
        !           178: ## Adjust continuous testing systems to use Kyua
        !           179: 
        !           180: With `kyua` being part of the release sets, it is possible to adjust the
        !           181: continuous test systems to make use of this tool in the test environments
        !           182: without having to take any additional step.
        !           183: 
        !           184: I'll work with gson@ and pgoyette@ to adapt their continuous testing
        !           185: machines to use the new built-in `kyua` binary instead of `atf-run` and
        !           186: `atf-report`.  I'm planning to do the necessary work to change `anita`
        !           187: myself, and I expect to help them deploy the changes to their own systems.
        !           188: Because Kyua and ATF will cohexist in the base system at this point,
        !           189: migrating the continuous testing systems to Kyua can happen at its own
        !           190: peace.
        !           191: 
        !           192: It might happen that Kyua misses some little detail needed by these
        !           193: systems.  In that case, this may require a new release of Kyua and a
        !           194: reimport into the tree.  Incremental imports with new features are much
        !           195: easier than the original import described in here.  Also, it will be
        !           196: possible to cherry-pick any external changes into the tree without a
        !           197: reimport (as has often been done in ATF).
        !           198: 
        !           199: ## User validation period
        !           200: 
        !           201: At this stage, **at least one month shall be given to the community** to
        !           202: test the new tools and the new test results dashboards.  Collect feedback
        !           203: and address requests as appropriate.
        !           204: 
        !           205: ## Replace atf-run and atf-report with kyua-atf-compat
        !           206: 
        !           207: Import the Kyua-based `atf-run` and `atf-report` compatibility tools and
        !           208: stop building the deprecated versions of these.  The compatibility versions
        !           209: are shipped in a separate `kyua-atf-compat` package, and thus this will be
        !           210: imported into `external/bsd/kyua-atf-compat/`.
        !           211: 
        !           212: Once this is done, change `bsd.test.mk` to not generate `Atffile`s any
        !           213: more, as the compatibility tools do not need them.
        !           214: 
        !           215: ## Drop atf-run and atf-report's code
        !           216: 
        !           217: Delete the native `atf-run` and `atf-report` utilities from the source tree
        !           218: (and from the upstream repository).  This also gets rid of a lot of
        !           219: supporting helper code, which makes the various ATF libraries leaner (and
        !           220: therefore benefits all test programs!).
        !           221: 
        !           222: ## Get rid of some ATF wrappers as a proof of concept
        !           223: 
        !           224: There are some test programs (e.g. the ones in the ipf test suite) that are
        !           225: not ATF-based.  To plug them into the NetBSD test suite, we have
        !           226: implemented ATF-based wrappers that invoke the original plain tests.  This
        !           227: is certainly suboptimal, as this level of indirection hinders readability
        !           228: and makes development harder.  (In particular, it prevents some key
        !           229: developers from contributing tests in the first place.)
        !           230: 
        !           231: Because Kyua has the ability to run these "plain" (non-ATF) test programs
        !           232: directly, and because a bunch of developers are really looking forward to
        !           233: this feature, I will convert a few tests to not include ATF wrappers as a
        !           234: proof of concept of this feature.  The ipf tests are probably a good choice
        !           235: for this.
        !           236: 
        !           237: ## Remove atf-run and atf-report compatibility tools
        !           238: 
        !           239: This is still really far away in the future (needs to happen after NetBSD
        !           240: 7.0 is branched), but I'm listing it for completeness of the plan.  The
        !           241: idea is to get rid of any ATF compatibility scripts by NetBSD 8.0.  This
        !           242: gives end users a full release cycle (that of 7) to adapt to Kyua while at
        !           243: the same time being able to use the old-style tools.
        !           244: 
        !           245: # Possible concerns
        !           246: 
        !           247: This section attempts to collect the list of possible concerns and/or
        !           248: objections that may come up during the review, together with an attempt of
        !           249: rebuttal from my side.
        !           250: 
        !           251: ## Increased compilation time
        !           252: 
        !           253: The addition of any new component to src increases the build time of the
        !           254: whole system.  In the case of Kyua, this increase might be noticeable
        !           255: *because of the large amount of test programs provided (roughly 100)*, all
        !           256: of which are in C++.
        !           257: 
        !           258: Note that ATF also had a relatively large codebase and a bunch of tests
        !           259: (although not as many).  The "trick" was that the majority of these tests
        !           260: were written in shell, and as such they did not increase the build time by
        !           261: much.  However, they significantly increased the run time of the test
        !           262: suite, and they were less detailed (mostly integration tests, few unit
        !           263: tests) than the Kyua tests.
        !           264: 
        !           265: In order to mitigate this issue, the build of all pieces of Kyua will be
        !           266: protected by `MKATF` so that people allergic to C++ can avoid the whole
        !           267: thing.  Even more, there are some other additional provisions described
        !           268: below.
        !           269: 
        !           270: ## Introduction of more C++ code in base
        !           271: 
        !           272: First of all, why did I use C++?  To make the implementation simpler and
        !           273: safer (the RAII programming pattern is really useful in avoiding memory and
        !           274: resource leaks with minimal effort).  And C++ is part of the base system
        !           275: and a supported language, so there was no reason not to do so.  But that's
        !           276: not the point of this item: if you don't like C++, this is not going to
        !           277: convince you otherwise.
        !           278: 
        !           279: It's true that, if we count the number of lines, Kyua brings in more C++
        !           280: code than what will eventually be dropped by the removal of the ATF tools.
        !           281: However, because ATF was also C++, the import of Kyua itself does not make
        !           282: the situation significantly worse.
        !           283: 
        !           284: Additionally, the two compilers we can really use (GCC and LLVM) already
        !           285: use, or will soon use, C++ in their code base.  It is unlikely that we will
        !           286: be able to remove all C++ support from base anytime soon due to this, while
        !           287: at the same time keeping support for all the ports that NetBSD has.
        !           288: 
        !           289: Long term, if the use of C++ proves to be a problem, there are a few things
        !           290: that can be done to slowly get rid of it that have been floating my mind
        !           291: recently.  The first is the rewrite of the performance-critical parts of
        !           292: Kyua in plain C.  This would involve splitting the runtime engine of a
        !           293: single test case in its own binary (which, for other reasons may be a good
        !           294: idea on its own).  The second is the rewrite of most user-interface code in
        !           295: Lua, which in itself would bring some extensibility advantages to the
        !           296: program.  Both options will be investigated when the time permits.  In the
        !           297: meantime, the replacement of ATF with Kyua does not make things worse; it
        !           298: just changes one chunk of code with another.
        !           299: 
        !           300: ## No need for Lutok as a public library
        !           301: 
        !           302: Kyua depends on Lutok, which is a C++ interface to Lua.  The code of this
        !           303: library was originally part of Kyua, but I split it into its own project
        !           304: because some users asked for it.
        !           305: 
        !           306: If there is no desire to ship Lutok as a shared library in `/usr/lib`, we
        !           307: can build Lutok as a static private library and link Kyua against it.
        !           308: There is no need to install this as a shared library.

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb