- Proposed changes
- The plan
- Discuss this plan in tech-userlevel@
- Get core@ approval
- Address feedback as a new release
- Import Kyua into src
- Adjust continuous testing systems to use Kyua
- Flip MKKYUA to yes
- Update documentation
- User validation period
- Replace atf-run and atf-report with kyua-atf-compat
- Drop atf-run and atf-report's code
- Get rid of some ATF wrappers as a proof of concept
- Remove atf-run and atf-report compatibility tools
- Possible concerns
Project owner: Julio Merino.
Status: Import of Kyua done as of 2013-02-26. All new code still disabled by default. Call for testing sent on 2013-02-28.
The import of Kyua into NetBSD to replace the deprecated ATF tools is
planned to happen in NetBSD 7.0. The ATF libraries will remain in place,
and as such no changes will happen to the tests that already live in
The transition from ATF to Kyua includes steps and tools to offer backwards compatibility with users that may rely on the ATF tools shipped since NetBSD 5.0. These backwards compatibility tools cover the most common use cases but might not be perfect initially. Despite these provisions, this plan should be executed well in advance the creation of the 7.0 branch to ensure there is enough time in NetBSD-current to flesh out any major problems.
Before moving on, please read the Kyua: An introduction for NetBSD users page. You should be familiar with the structure of Kyua and its major components to be able to review this plan.
The final user visible changes that this project will bring to NetBSD are as follows. Because these change the availability of tools that have already been shipped since 5.0, the modifications are staged to happen across two major releases:
Addition of the new
kyuacommand-line tool. This provides the runtime engine for the test programs (
atf-run's functionality) and the ability to generate reports of the results in plain text form and HTML form (
atf-report's functionality). This tool is able to execute all the existing test programs without modifications.
Replacement of the
Atffiles files in
Ability to generate HTML reports right form the base system, with such reports hooked into the various continuous build systems.
Replacement of the
atf-reporttools with shell scripts that provide backwards compatibility implementations based on
kyua. These can deal with old-style
Atffiles so that users with custom test suites can continue to run them.
Introduction of a new
atf2kyuascript to convert old
Kyuafiles. This is used internally by the compatibility
atf-runscript, but can also be invoked by the end user by hand to deal with his own test suites. Could be placed into
libexecif we do not want to make this public.
- Removal of the backwards compatibility
atf-reportscripts, as well as the supporting
As mentioned in another page, Kyua should be seen as ATF 2.x even though it carries a different name. It is the evolution of the previous ATF tools (the tools only), but written in a more modular and flexible way, and with a more reliable codebase. Therefore, you should consider this project as the update of ATF to a newer version.
atf-report tools have effectively been in maintenance
mode for over a year already. None of the desired features (see the list
of open PRs) have been implemented on top of them, mostly because doing so
is building upon a broken implementation. Additionally, several developers
have had to implement their own test results dashboards due to
atf-report, effectively reinventing the wheel.
This update will permit the real removal of the obsolete tools, thus
allowing us to build additional features on top of Kyua without having to
worry about being compatible with
atf-run (and thus adjusting this tool
to behave in the same manner).
Some possible answers to "Why not?" can be found later in this page.
Kyua has existed for almost 3 years already, so you may be wondering why this import is being proposed now. The major reasons are:
Since 0.5, Kyua has now feature-parity with ATF. Replacing the ATF tools with Kyua should not introduce functionality regressions.
NetBSD-6 has been branched relatively recently, so we can expect NetBSD-7 to be far away enough in the future to provide plenty of time to stabilize Kyua in NetBSD-current.
The existence of the deprecated
atf-reporttools hinders the development of new features that require changes to the inteface of ATF test programs. As an example: a highly requested feature is the ability to change the timeout settings of the test programs (for the benefit of old, slow platforms); doing this in the ATF codebase is tricky.
Some developers continue to add old-style tests to the tree. While this is suboptimal and against the policy dictated by core@, Kyua brings in a mechanism to allow running these tests unmodified (i.e. without adding any ATF calls in their code). This, in turn, will let such developers add their tests more easily, and thus increase the tests coverage.
This section details what the transition plan looks like and an estimate timeline. As all things that depend on free time (opensource software hacking, in this case), take the estimates with a grain of salt.
If you are here it's possibly because a review request for this plan has already been published and thus the plan has already begun.
This plan will be sent to the tech-userlevel@ mailing list asking for comments. Two weeks shall be allowed for initial discussion. Depending on the outcome, the plan and/or the software will need to be adjusted (which in turn may require significant amounts of time not yet accounted for). I'll be optimistic for now.
By policy, the import of any new software component into src requires core@ approval. Even though this will have been discussed at length in tech-userlevel@ (as per the previous step), the final decider on this issue will be core@. The corollary of this is that, if no consensus can be reached in tech-userlevel@ regarding this plan, core@ will be asked to come up with a decision.
If core@ approves the plan, the next steps shall start immediately. If core@ disagrees, core@ will be asked to provide advice on the corrections that should be made before the plan can be approved.
It is hard to tell how long this step will last, but possibly account for 2 to 4 weeks.
Publish a new Kyua release that collects all the feedback from the reviews above.
The list of issues to be addressed can be found by querying the bug tracker. In particular, the following are the issues that have arisen from the review:
Issue 36: Fix the
helpcommand to not fail if the configuration file is bogus. DONE
Issue 37: Simplify the syntax definition of configuration and
Kyuafilefiles by removing the format name. DONE
Issue 40: Provide manpages instead of an info document. DONE
Issue 47: Implement independent testers, which reduces the amount of C++ code and avoids the need of modifying
Issue 57: Generalized support for all metadata properties to plain test programs. This is to make plain test programs more versatile by bringing them closer to feature parity with ATF test programs. DONE
As the introductory page to Kyua describes, Kyua has been
available in pkgsrc for a while and can be readily installed and used to
run the tests from
However, because ATF lives in src, and because NetBSD aims to provide the best environment for testing "out of the box", Kyua should be imported into src just like ATF was. The major reasons for this, as have been explained in the past, are: first, to allow any new deployment of NetBSD to be validated right after installation and continuously afterwards; and, second, to permit the execution of tests during development without having to install any additional software.
The specific steps to perform this import are:
Import Lutok into
external/bsd/lutok/. This is a shared library that wraps the native Lua C interface in C++. Lutok was originally part of Kyua, and was split into its own package per the request of some users that found this component useful on its own.
Import the Kyua testers into
external/bsd/kyua-testers/. This yields two new binaries in
kyua-plain-tester) and a bunch of tests in
Import the Kyua frontend into
external/bsd/kyua-cli/. This yields a new kyua binary in
/usr/bin, a lot of test programs in
/usr/tests/kyua-cli(around 100) and some auxiliary files in
Protect all products of Lutok and Kyua with a new
MKKYUA=noby default. Once the ATF tools are removed, the existence of both the
MKKYUAknows will probably be confusing. When that happens, we can revisit this decision by possibly replacing both with an
Kyuafiles in addition to
There is no real need to do this import in a branch given that this import only adds new functionality without touching existing stuff, and the new code is disabled by default.
All the preparatory work for the import can be done offline (in about two weeks at most, given that I have mot of this ready). Aside from the code changes, this will involve the validation of NetBSD/amd64, NetBSD/i386 and NetBSD/macppc builds (which are the ports I have access to). If you consider that some other tricky architecture should be build-tested (sparc64?), let me know and I'll include it in the list.
The submission to CVS will be prepared locally and performed on a package
kyua-cli, in this order). These
are to be imported separately to simplify the review of the changes and to
allow me to better test every individual change locally. There may be an
arbitrary amount of time between the submission of each package: this
should not be a problem because these modules are still disabled due to
MKKYUA being set to
no by default.
kyua being part of the release sets, it is possible to adjust the
continuous test systems to make use of this tool in the test environments
without having to take any additional step.
I'll work with gson@ and pgoyette@ to adapt their continuous testing
machines to use the new built-in
kyua binary instead of
atf-report. I'm planning to do the necessary work to change
myself, and I expect to help them deploy the changes to their own systems.
Because Kyua and ATF will cohexist in the base system at this point,
migrating the continuous testing systems to Kyua can happen at its own
It might happen that Kyua misses some little detail needed by these systems. In that case, this may require a new release of Kyua and a reimport into the tree. Incremental imports with new features are much easier than the original import described in here. Also, it will be possible to cherry-pick any external changes into the tree without a reimport (as has often been done in ATF).
This step can take a few weeks of time, mostly due to the back and forth between different people in different timezones.
The previous steps imported Kyua but didn't enable its build by default so
that proper testing can be performed by the only people that care. Once
basic testing (particularly build testing on a variety of platforms) is
The Kyua: An introduction for NetBSD users wiki page will be updated to explain how Kyua is bundled in NetBSD and how to use the bundled version.
The Creating atf-based tests for NetBSD src wiki page will be updated to account for the differences in test programs execution with Kyua instead of ATF.
The afterboot(8) and tests(7) manpages will be adjusted to mention
instead of the ATF tools.
At this stage, at least one month shall be given to the community to test the new tools and the new test results dashboards. Collect feedback and address requests as appropriate (possibly by releasing and importing a new version of Kyua).
One important thing to validate is that the results reported by
kyua test match with each other. I have already been validating this
with every public release of Kyua, but it has been a manual process. To
make this more reliable, I will set up a continuous testing machine of my
own in which I will execute
kyua test in sequence (possibly
within anita) and will add an automatic comparison of the exit status of
Because the old
atf-report tools have not yet been dropped
at this point, we can spend as much time as necessary on this phase to get
Import the Kyua-based
atf-report compatibility tools and
stop building the deprecated versions of these. The compatibility versions
are shipped in a separate
kyua-atf-compat package, and thus this will be
Once this is done, change
bsd.test.mk to not generate
more, as the compatibility tools do not need them.
Delete the native
atf-report utilities from the source tree
(and from the upstream repository). This also gets rid of a lot of
supporting helper code, which makes the various ATF libraries leaner (and
therefore benefits all test programs!).
There are some test programs (e.g. the ones in the ipf test suite) that are not ATF-based. To plug them into the NetBSD test suite, we have implemented ATF-based wrappers that invoke the original plain tests. This is certainly suboptimal, as this level of indirection hinders readability and makes development harder. (In particular, it prevents some key developers from contributing tests in the first place.)
Because Kyua has the ability to run these "plain" (non-ATF) test programs directly, and because a bunch of developers are really looking forward to this feature, I will convert a few tests to not include ATF wrappers as a proof of concept of this feature. The ipf tests are probably a good choice for this.
This is still really far away in the future (needs to happen after NetBSD 7.0 is branched), but I'm listing it for completeness of the plan. The idea is to get rid of any ATF compatibility scripts by NetBSD 8.0. This gives end users a full release cycle (that of 7) to adapt to Kyua while at the same time being able to use the old-style tools.
This section attempts to collect the list of possible concerns and/or objections that may come up during the review, together with an attempt of rebuttal from my side.
The addition of any new component to src increases the build time of the whole system. In the case of Kyua, this increase might be noticeable because of the large amount of test programs provided (roughly 100), all of which are in C++.
Note that ATF also had a relatively large codebase and a bunch of tests (although not as many). The "trick" was that the majority of these tests were written in shell, and as such they did not increase the build time by much. However, they significantly increased the run time of the test suite, and they were less detailed (mostly integration tests, few unit tests) than the Kyua tests.
In order to mitigate this issue, the build of all pieces of Kyua will be
MKKYUA so that people allergic to C++ can avoid the whole
thing. Even more, there are some other additional provisions described
First of all, why did I use C++? To make the implementation simpler and safer (the RAII programming pattern is really useful in avoiding memory and resource leaks with minimal effort). And C++ is part of the base system and a supported language, so there was no reason not to do so. But that's not the point of this item: if you dislike like C++, this is not going to make you think I did right.
It's true that, if we count the number of lines, Kyua brings in more C++ code than what will eventually be dropped by the removal of the ATF tools. However, because ATF was also C++, the import of Kyua itself does not make the situation significantly worse.
Additionally, the two compilers we can really use (GCC and LLVM) already use, or will soon use, C++ in their code base. It is unlikely that we will be able to remove all C++ support from base anytime soon due to this, while at the same time keeping support for all the ports that NetBSD has.
Thanks to the testers project, and starting with Kyua 0.6, a lot of the tricky OS-specific code in Kyua has been rewritten in plain C. This paves the way to rewriting parts of the now-simpler frontend in C or Lua, if the use of C++ proves to be a serious problem in the future.
In the short term, the replacement of ATF with Kyua does not make things worse: this project just changes one chunk of code with another.
Kyua depends on Lutok, which is a C++ interface to Lua. The code of this library was originally part of Kyua, but I split it into its own project because some users asked for it.
If there is no desire to ship Lutok as a shared library in
can build Lutok as a static private library and link Kyua against it.
There is no need to install this as a shared library.