Contents
The Automated Testing Framework, or ATF for short, is a software package composed of two parts: the ATF libraries and the ATF tools. The ATF libraries provide a toolkit for developers to implement test cases in a variety of languages: C, C++ and POSIX shell. The ATF tools provide the utilities to run such test cases in an automated way and to generate reports.
The ATF tools have some design and, particularly, implementation problems that make it hard to add support for highly desired features such as parallel execution of test cases, unified dashboards covering multiple test runs (like this one), the ability to run legacy or third-party test programs that do not use the ATF libraries, and the ability to tune the timeout of test cases.
Kyua's current goal is to reimplement only the ATF tools while maintaining backwards compatibility with the tests written with the ATF libraries (i.e. with the NetBSD test suite).
Because Kyua is a replacement of some ATF components, the end goal is to integrate Kyua into the NetBSD base system (just as ATF is) and remove the deprecated ATF components. Removing the deprecated components will allow us to make the above-mentioned improvements to Kyua, as well as many others, without having to deal with the obsolete ATF code base. Discussing how and when this transition might happen is out of the scope of this document at the moment.
This page provides instructions on how to use Kyua with the current NetBSD test suite so that you can experiment with the tool, familiarize yourself with it and provide feedback early on.
If you would like more details on how the import of Kyua into NetBSD will happen, please see Kyua: The way into NetBSD.
What's in the name?
You should really think of Kyua as ATF 2.x. Then, why isn't it just ATF 2.x?
To be honest, I never liked the ATF name: it was picked for me as part of the Google Summer of Code 2007 program and I did not think about changing it at that time. A year later, I learned that the ATF acronym is severely overloaded, which makes it hard to find the project on popular search engines, and has unpopular connotations in specific countries.
So, as part of the rewrite, I decided to choose a new name: a name that is not an acronym and thus can be easily pronounced, and a name that is quite unique in search results. The name is Kyua, which is just a play on the pronounciation of the Q.A. acronym. Originally, my intention was to pronounce Kyua as Q.A., but in reality this never happened. Today, just read the name as your instinct would: "Kyu-ah".
Why is Kyua a third-party project?
Kyua's main consumer is NetBSD. One could argue that Kyua should be developed within NetBSD and maintained in the NetBSD source tree. However, there is nothing in the Kyua project that inherently depends on NetBSD, and maintaining it as a third-party package is a way to keep the developers honest regarding portability.
Ideally, other projects (such as FreeBSD) would make use of Kyua too for their testing needs, and if that happened we would be able to share tests with them pretty easily. Forcing a portable codebase in the upstream repository helps in this. (Be aware that different individuals from FreeBSD and Minix have shown interest in adopting Kyua for their respective systems!)
Main differences (aka "what to expect")
As of version 0.5, Kyua has (or is supposed to have) feature parity with the ATF tools. That said, having feature parity does not imply that they are the same. This section outlines a few of the differences that you should be aware of before continuing.
Results database
Kyua collects the results of the execution of a test suite into an SQLite database. User-friendly reports are later generated by extracting data from this same database.
In ATF, the results of the execution were written to an internal format that only atf-report could understand. Despite of the database, Kyua still maintains the separation of "tests execution" from "report generation".
The contents of the database are immutable and incremental. This means that, in the future, the Kyua tools will be able to provide historical data for particular test cases, or for whole test runs (which is what other NetBSD developers have ended up implementing multiple times outside of ATF because the framework did not provide such functionality by itself).
Support for multiple test interfaces
Kyua has support for different "test interfaces", which means that Kyua can execute test programs written using different paradigms and collect their results into a single report. At the moment, two interfaces are supported:
The "atf" interface provides compatibility with those test programs that use the ATF libraries. This is the only interface currently used by the NetBSD test suite, as there is no way to run any other test program in an automated manner.
The "plain" interface permits the execution of legacy test programs that do not use any testing library. Such test programs are those that just return 0 or non-0 to indicate the success or failure of the test (respectively). This feature will allow the NetBSD test suite to transparently execute third-party test suites (such as the IPF or GCC test suites) without having to implement ATF-based wrappers. It will also lower the barrier of entry to writing test programs for NetBSD, as using the ATF libraries will become optional.
These interfaces are implemented as independent binaries, called testers,
that are fully scriptable. The testers are provided in the kyua-testers
package. The idea behind having these as independent programs is to
restrict the OS-specific code to a small subset of Kyua written in C, and
thus to allow the higher-level layers to be written in other languages
(possibly Lua).
Lua configuration files
Kyua has two kind of configuration files: the Kyuafiles, which are the files shipped with a test suite that describe what test programs need to be run; and the user configuration files, which specify the run-time settings of Kyua and the test suites. ATF had this same split of configuration files, and they were written in a custom language, with a custom parser.
The Kyua configuration files are all Lua scripts. The major advantage of this at the moment is that their syntax will be familiar to end users, and that the parser for these files is well-tested. In the future, the use of Lua will allow the implementation of more-intelligent test (and maybe even build) scripts.
Direct HTML output
All of the NetBSD continous build and testing systems provide status
reports through the releng web interface. In the case of ATF, this has
traditionally been tricky because ATF cannot generate HTML contents
directly; instead, atf-report generates XML output which later must be
postprocessed with xsltproc to create the HTML pages.
Kyua has the ability to generate HTML reports straight from the tool, without having to go through any XML toolchain. This means that NetBSD, out of the box, can generate such reports and publish them with the builtin httpd(8) server.
Heavier code base
If you take a look at the Kyua distribution file, you may notice that it is about the same size as the distribution file of ATF, yet Kyua does not currently replace the ATF libraries. This may be surprising because it seems to imply that the codebase of Kyua is bigger because it "just" reimplements atf-run and atf-report: i.e. by just reimplementing parts of ATF, it is already as big as the whole of ATF.
This is true, for two reasons.
The first is that Kyua is more featureful and flexible: the features outlined above have a cost in terms of implementation, and the codebase of Kyua is more carefully crafted to allow for later growth. In particular, all OS-specific details have been abstracted for easier portability, and the SQLite and Lua libraries have been wrapped for safety.
The second is that Kyua is much better tested (which is very important for a software package that you will rely on to validate your own software!). To give you some numbers, ATF 0.16 contains around 400 test cases for both atf-run and atf-report while Kyua 0.5 contains around 1100 test cases.
Components
Kyua, as a project, is made up of a variety of components (which include ATF, because the ATF libraries are not being rewritten). All of these components exist in pkgsrc, and are:
pkgsrc/devel/atf: The ATF tools, namely atf-run and atf-report. These are deprecated and this package should eventually disappear.
pkgsrc/devel/kyua: The Kyua command-line interface, which provides a superset of the functionality of atf-run and atf-report.
Running the NetBSD test suite
There are two ways to run the NetBSD test suite with Kyua. The easy (or trivial) way is to use the backwards compatibility ATF tools, and the more sophisticated way is to convert the test suite to Kyua and use the native Kyua binary. This section explains both approaches.
Using the ATF compatibility tools
The easiest (but also the least "future-proof") way to run the NetBSD test suite with Kyua is to use the backwards compatibility ATF tools provided by the kyua-atf-compat module. First of all, install the package:
$ cd /usr/pkgsrc/devel/kyua-atf-compat
$ make install && make clean
And then, running the test suite is as easy as:
$ cd /usr/tests
$ /usr/pkg/bin/atf-run | /usr/pkg/bin/atf-report
Please be aware that if the atf-run and atf-report tools provided by kyua-atf-compat appear in your PATH before the real atf-run and atf-report tools shipped by NetBSD, you will experience test failures for all the tests in /usr/tests/atf/atf-run and /usr/tests/atf/atf-report. This is expected: while the compatibility tools behave similarly to the real tools from a user's perspective, they are not fully interchangeable. (For example, the serialization format between atf-run and atf-report is different.)
One property of the atf-run wrapper is that it uses the default results database in ~/.kyua/store.db to record the execution of the tests. This means that, once the execution of the tests is done with the compatibility tools, you can still use the native Kyua binary to poke at the results database. More on this below.
Using the native Kyua command-line interface
The preferred way to run the NetBSD test suite with Kyua is to use the native Kyua command-line binary. This is the preferred method because it trains you to use the new interface rather than relying on the old pipeline and because it exposes you to all the new features of Kyua. Regardless, this and the previous approach will yield the same results for a particular execution.
Using the native command-line interface is a multi-step process because the existing NetBSD test suite is not prepared for Kyua. Let's take a look at these steps.
To get started, install the Kyua package:
$ cd /usr/pkgsrc/devel/kyua
$ make install clean
Once this is done, configure Kyua in the same way ATF is configured "out of the box" in NetBSD. Create a /usr/pkg/etc/kyua/kyua.conf file containing:
syntax('config', 1)
unprivileged_user = '_tests'
The next step is to populate /usr/tests with Kyuafiles, as Kyua is unable to read existing Atffiles. This is easy to do with the atf2kyua(1) tool shipped in the kyua-atf-compat package:
# atf2kyua /usr/tests
And that is it. You can now execute the test suite using Kyua with any of the following two forms:
$ cd /usr/tests && kyua test
$ kyua test -k /usr/tests/Kyuafile
Note that none of these will generate "pretty" reports. These commands will only record the results of the execution into the database. In order to generate reports, keep reading.
Generating reports
Once you have ran the NetBSD test suite with any of the mechanisms above, the results of the execution have been stored in the "Kyua store", which is a database located in ~/.kyua/store.db by default. (This path can be changed at any time with the --store flag.)
To extract a report from the database using the results of the latest tests run, you can run any of the following:
$ kyua report -o my-report.txt
$ kyua html-report -o /var/www/results/
Support and feedback
The Kyua manual is available in the GNU Info format and can be accessed by running:
$ info kyua
Alternatively, use the help subcommand to get built-in documentation. The following invocation will print all the available subcommands:
$ kyua help
And an invocation like this will show you all the possible options for a given subcommand:
$ kyua help report-html
If you have gone through the instructions above and started playing with Kyua, please do not hesitate to report your experiences (either good or bad) to Julio Merino! Any comments will be highly appreciated and will be taken into account for the near future of Kyua.