Contents

Project owner: Julio Merino.
Status: Import of Kyua done as of 2013-02-26. All new code still disabled by default. Call for testing sent on 2013-02-28.

The import of Kyua into NetBSD to replace the deprecated ATF tools is planned to happen in NetBSD 7.0. The ATF libraries will remain in place, and as such no changes will happen to the tests that already live in src/tests.

The transition from ATF to Kyua includes steps and tools to offer backwards compatibility with users that may rely on the ATF tools shipped since NetBSD 5.0. These backwards compatibility tools cover the most common use cases but might not be perfect initially. Despite these provisions, this plan should be executed well in advance the creation of the 7.0 branch to ensure there is enough time in NetBSD-current to flesh out any major problems.

Before moving on, please read the Kyua: An introduction for NetBSD users page. You should be familiar with the structure of Kyua and its major components to be able to review this plan.

Proposed changes

The final user visible changes that this project will bring to NetBSD are as follows. Because these change the availability of tools that have already been shipped since 5.0, the modifications are staged to happen across two major releases:

For NetBSD 7.0

• Addition of the new kyua command-line tool. This provides the runtime engine for the test programs (atf-run's functionality) and the ability to generate reports of the results in plain text form and HTML form (atf-report's functionality). This tool is able to execute all the existing test programs without modifications.

• Replacement of the Atffiles files in /usr/tests with Kyuafiles.

• Ability to generate HTML reports right form the base system, with such reports hooked into the various continuous build systems.

• Replacement of the atf-run and atf-report tools with shell scripts that provide backwards compatibility implementations based on kyua. These can deal with old-style Atffiles so that users with custom test suites can continue to run them.

• Introduction of a new atf2kyua script to convert old Atffiles to Kyuafiles. This is used internally by the compatibility atf-run script, but can also be invoked by the end user by hand to deal with his own test suites. Could be placed into libexec if we do not want to make this public.

For NetBSD 8.0

• Removal of the backwards compatibility atf-run and atf-report scripts, as well as the supporting atf2kyua tool.

Why?

As mentioned in another page, Kyua should be seen as ATF 2.x even though it carries a different name. It is the evolution of the previous ATF tools (the tools only), but written in a more modular and flexible way, and with a more reliable codebase. Therefore, you should consider this project as the update of ATF to a newer version.

The atf-run and atf-report tools have effectively been in maintenance mode for over a year already. None of the desired features (see the list of open PRs) have been implemented on top of them, mostly because doing so is building upon a broken implementation. Additionally, several developers have had to implement their own test results dashboards due to defficiencies in atf-report, effectively reinventing the wheel.

This update will permit the real removal of the obsolete tools, thus allowing us to build additional features on top of Kyua without having to worry about being compatible with atf-run (and thus adjusting this tool to behave in the same manner).

Some possible answers to "Why not?" can be found later in this page.

And why now?

Kyua has existed for almost 3 years already, so you may be wondering why this import is being proposed now. The major reasons are:

• Since 0.5, Kyua has now feature-parity with ATF. Replacing the ATF tools with Kyua should not introduce functionality regressions.

• NetBSD-6 has been branched relatively recently, so we can expect NetBSD-7 to be far away enough in the future to provide plenty of time to stabilize Kyua in NetBSD-current.

• The existence of the deprecated atf-run and atf-report tools hinders the development of new features that require changes to the inteface of ATF test programs. As an example: a highly requested feature is the ability to change the timeout settings of the test programs (for the benefit of old, slow platforms); doing this in the ATF codebase is tricky.

• Some developers continue to add old-style tests to the tree. While this is suboptimal and against the policy dictated by core@, Kyua brings in a mechanism to allow running these tests unmodified (i.e. without adding any ATF calls in their code). This, in turn, will let such developers add their tests more easily, and thus increase the tests coverage.

The plan

This section details what the transition plan looks like and an estimate timeline. As all things that depend on free time (opensource software hacking, in this case), take the estimates with a grain of salt.

Discuss this plan in tech-userlevel@

If you are here it's possibly because a review request for this plan has already been published and thus the plan has already begun.

This plan will be sent to the tech-userlevel@ mailing list asking for comments. Two weeks shall be allowed for initial discussion. Depending on the outcome, the plan and/or the software will need to be adjusted (which in turn may require significant amounts of time not yet accounted for). I'll be optimistic for now.

Get core@ approval

By policy, the import of any new software component into src requires core@ approval. Even though this will have been discussed at length in tech-userlevel@ (as per the previous step), the final decider on this issue will be core@. The corollary of this is that, if no consensus can be reached in tech-userlevel@ regarding this plan, core@ will be asked to come up with a decision.

If core@ approves the plan, the next steps shall start immediately. If core@ disagrees, core@ will be asked to provide advice on the corrections that should be made before the plan can be approved.

It is hard to tell how long this step will last, but possibly account for 2 to 4 weeks.

Address feedback as a new release

Publish a new Kyua release that collects all the feedback from the reviews above.

The list of issues to be addressed can be found by querying the bug tracker. In particular, the following are the issues that have arisen from the review:

• Issue 36: Fix the help command to not fail if the configuration file is bogus. DONE

• Issue 37: Simplify the syntax definition of configuration and Kyuafile files by removing the format name. DONE

• Issue 40: Provide manpages instead of an info document. DONE

• Issue 47: Implement independent testers, which reduces the amount of C++ code and avoids the need of modifying bsd.dep.mk. DONE

• Issue 57: Generalized support for all metadata properties to plain test programs. This is to make plain test programs more versatile by bringing them closer to feature parity with ATF test programs. DONE

Import Kyua into src

As the introductory page to Kyua describes, Kyua has been available in pkgsrc for a while and can be readily installed and used to run the tests from /usr/tests.

However, because ATF lives in src, and because NetBSD aims to provide the best environment for testing "out of the box", Kyua should be imported into src just like ATF was. The major reasons for this, as have been explained in the past, are: first, to allow any new deployment of NetBSD to be validated right after installation and continuously afterwards; and, second, to permit the execution of tests during development without having to install any additional software.

The specific steps to perform this import are:

1. Import Lutok into external/bsd/lutok/. This is a shared library that wraps the native Lua C interface in C++. Lutok was originally part of Kyua, and was split into its own package per the request of some users that found this component useful on its own.

2. Import the Kyua testers into external/bsd/kyua-testers/. This yields two new binaries in /usr/libexec (kyua-atf-tester and kyua-plain-tester) and a bunch of tests in /usr/tests/kyua-testers.

3. Import the Kyua frontend into external/bsd/kyua-cli/. This yields a new kyua binary in /usr/bin, a lot of test programs in /usr/tests/kyua-cli (around 100) and some auxiliary files in /usr/share/kyua.

4. Protect all products of Lutok and Kyua with a new MKKYUA knob. Set MKKYUA=no by default. Once the ATF tools are removed, the existence of both the MKATF and MKKYUA knows will probably be confusing. When that happens, we can revisit this decision by possibly replacing both with an MKTESTS.

5. Update bsd.test.mk to generate Kyuafiles in addition to Atffiles.

There is no real need to do this import in a branch given that this import only adds new functionality without touching existing stuff, and the new code is disabled by default.

All the preparatory work for the import can be done offline (in about two weeks at most, given that I have mot of this ready). Aside from the code changes, this will involve the validation of NetBSD/amd64, NetBSD/i386 and NetBSD/macppc builds (which are the ports I have access to). If you consider that some other tricky architecture should be build-tested (sparc64?), let me know and I'll include it in the list.

The submission to CVS will be prepared locally and performed on a package basis (i.e. lutok, kyua-testers and kyua-cli, in this order). These are to be imported separately to simplify the review of the changes and to allow me to better test every individual change locally. There may be an arbitrary amount of time between the submission of each package: this should not be a problem because these modules are still disabled due to MKKYUA being set to no by default.

Adjust continuous testing systems to use Kyua

With kyua being part of the release sets, it is possible to adjust the continuous test systems to make use of this tool in the test environments without having to take any additional step.

I'll work with gson@ and pgoyette@ to adapt their continuous testing machines to use the new built-in kyua binary instead of atf-run and atf-report. I'm planning to do the necessary work to change anita myself, and I expect to help them deploy the changes to their own systems. Because Kyua and ATF will cohexist in the base system at this point, migrating the continuous testing systems to Kyua can happen at its own peace.

It might happen that Kyua misses some little detail needed by these systems. In that case, this may require a new release of Kyua and a reimport into the tree. Incremental imports with new features are much easier than the original import described in here. Also, it will be possible to cherry-pick any external changes into the tree without a reimport (as has often been done in ATF).

This step can take a few weeks of time, mostly due to the back and forth between different people in different timezones.

Flip MKKYUA to yes

The previous steps imported Kyua but didn't enable its build by default so that proper testing can be performed by the only people that care. Once basic testing (particularly build testing on a variety of platforms) is performed, flip MKKYUA=yes.

Update documentation

The Kyua: An introduction for NetBSD users wiki page will be updated to explain how Kyua is bundled in NetBSD and how to use the bundled version.

The Creating atf-based tests for NetBSD src wiki page will be updated to account for the differences in test programs execution with Kyua instead of ATF.

The afterboot(8) and tests(7) manpages will be adjusted to mention kyua instead of the ATF tools.

User validation period

At this stage, at least one month shall be given to the community to test the new tools and the new test results dashboards. Collect feedback and address requests as appropriate (possibly by releasing and importing a new version of Kyua).

One important thing to validate is that the results reported by atf-run and kyua test match with each other. I have already been validating this with every public release of Kyua, but it has been a manual process. To make this more reliable, I will set up a continuous testing machine of my own in which I will execute atf-run and kyua test in sequence (possibly within anita) and will add an automatic comparison of the exit status of each other.

Because the old atf-run and atf-report tools have not yet been dropped at this point, we can spend as much time as necessary on this phase to get things right.

Replace atf-run and atf-report with kyua-atf-compat

Import the Kyua-based atf-run and atf-report compatibility tools and stop building the deprecated versions of these. The compatibility versions are shipped in a separate kyua-atf-compat package, and thus this will be imported into external/bsd/kyua-atf-compat/.

Once this is done, change bsd.test.mk to not generate Atffiles any more, as the compatibility tools do not need them.

Drop atf-run and atf-report's code

Delete the native atf-run and atf-report utilities from the source tree (and from the upstream repository). This also gets rid of a lot of supporting helper code, which makes the various ATF libraries leaner (and therefore benefits all test programs!).

Get rid of some ATF wrappers as a proof of concept

There are some test programs (e.g. the ones in the ipf test suite) that are not ATF-based. To plug them into the NetBSD test suite, we have implemented ATF-based wrappers that invoke the original plain tests. This is certainly suboptimal, as this level of indirection hinders readability and makes development harder. (In particular, it prevents some key developers from contributing tests in the first place.)

Because Kyua has the ability to run these "plain" (non-ATF) test programs directly, and because a bunch of developers are really looking forward to this feature, I will convert a few tests to not include ATF wrappers as a proof of concept of this feature. The ipf tests are probably a good choice for this.

Remove atf-run and atf-report compatibility tools

This is still really far away in the future (needs to happen after NetBSD 7.0 is branched), but I'm listing it for completeness of the plan. The idea is to get rid of any ATF compatibility scripts by NetBSD 8.0. This gives end users a full release cycle (that of 7) to adapt to Kyua while at the same time being able to use the old-style tools.

Possible concerns

This section attempts to collect the list of possible concerns and/or objections that may come up during the review, together with an attempt of rebuttal from my side.

Increased compilation time

The addition of any new component to src increases the build time of the whole system. In the case of Kyua, this increase might be noticeable because of the large amount of test programs provided (roughly 100), all of which are in C++.

Note that ATF also had a relatively large codebase and a bunch of tests (although not as many). The "trick" was that the majority of these tests were written in shell, and as such they did not increase the build time by much. However, they significantly increased the run time of the test suite, and they were less detailed (mostly integration tests, few unit tests) than the Kyua tests.

In order to mitigate this issue, the build of all pieces of Kyua will be protected by MKKYUA so that people allergic to C++ can avoid the whole thing. Even more, there are some other additional provisions described below.

Introduction of more C++ code in base

First of all, why did I use C++? To make the implementation simpler and safer (the RAII programming pattern is really useful in avoiding memory and resource leaks with minimal effort). And C++ is part of the base system and a supported language, so there was no reason not to do so. But that's not the point of this item: if you dislike like C++, this is not going to make you think I did right.

It's true that, if we count the number of lines, Kyua brings in more C++ code than what will eventually be dropped by the removal of the ATF tools. However, because ATF was also C++, the import of Kyua itself does not make the situation significantly worse.

Additionally, the two compilers we can really use (GCC and LLVM) already use, or will soon use, C++ in their code base. It is unlikely that we will be able to remove all C++ support from base anytime soon due to this, while at the same time keeping support for all the ports that NetBSD has.

Thanks to the testers project, and starting with Kyua 0.6, a lot of the tricky OS-specific code in Kyua has been rewritten in plain C. This paves the way to rewriting parts of the now-simpler frontend in C or Lua, if the use of C++ proves to be a serious problem in the future.

In the short term, the replacement of ATF with Kyua does not make things worse: this project just changes one chunk of code with another.

No need for Lutok as a public library

Kyua depends on Lutok, which is a C++ interface to Lua. The code of this library was originally part of Kyua, but I split it into its own project because some users asked for it.

If there is no desire to ship Lutok as a shared library in /usr/lib, we can build Lutok as a static private library and link Kyua against it. There is no need to install this as a shared library.