File:  [NetBSD Developer Wiki] / wikisrc / kyua.mdwn
Revision 1.7: download - view: text, annotated - select for diffs
Tue Nov 27 06:45:22 2012 UTC (16 months, 3 weeks ago) by riz
Branches: MAIN
CVS tags: HEAD
typo correction:  /usr/pkgsrc/deve/ -> /usr/pkgsrc/devel/

[[!meta title="Kyua: An introduction for NetBSD users"]]
[[!toc levels=2]]

The [Automated Testing Framework](ATF), or ATF for short, is a software
package composed of two parts: the *ATF libraries* and the *ATF tools*.
The ATF libraries provide a toolkit for developers to implement test cases
in a variety of languages: C, C++ and POSIX shell.  The ATF tools provide
the utilities to run such test cases in an automated way and to generate
reports.

The ATF tools have some
[design and, particularly, implementation problems](http://mail-index.netbsd.org/atf-devel/2010/11/13/msg000206.html)
that make it hard to add support for highly desired features such as
parallel execution of test cases, unified dashboards covering multiple test
runs ([like this one](http://releng.netbsd.org/test-results.html)), the
ability to run legacy or third-party test programs that do not use the ATF
libraries, and the ability to tune the timeout of test cases.

*Kyua's current goal is to reimplement _only_ the ATF tools* while
maintaining backwards compatibility with the tests written with the ATF
libraries (i.e. with the NetBSD test suite).

Because Kyua is a replacement of some ATF components, the end goal is to
integrate Kyua into the NetBSD base system (just as ATF is) and remove the
deprecated ATF components.  Removing the deprecated components will allow
us to make the above-mentioned improvements to Kyua, as well as many
others, without having to deal with the obsolete ATF code base.
*Discussing how and when this transition might happen is out of the scope
of this document at the moment.*

This page provides instructions on how to use Kyua with the current NetBSD
test suite so that you can experiment with the tool, familiarize yourself
with it and provide feedback early on.

**If you would like more details on how the import of Kyua into NetBSD will
  happen, please see [[Kyua: The way into NetBSD|/kyua/import]].**

# What's in the name?

You should really think of Kyua as ATF 2.x.  Then, why isn't it just ATF
2.x?

To be honest, I never liked the ATF name: it was picked for me as part of
the Google Summer of Code 2007 program and I did not think about changing
it at that time.  A year later, I learned that the ATF acronym is severely
overloaded, which makes it hard to find the project on popular search
engines, and has unpopular connotations in specific countries.

So, as part of the rewrite, I decided to choose a new name: a name that is
not an acronym and thus can be easily pronounced, and a name that is quite
unique in search results.  The name is Kyua, which is just a play on the
pronounciation of the Q.A. acronym.  Originally, my intention was to
pronounce Kyua as Q.A., but in reality this never happened.  Today, just
read the name as your instinct would: "Kyu-ah".

# Why is Kyua a third-party project?

Kyua's main consumer is NetBSD.  One could argue that Kyua should be
developed within NetBSD and maintained in the NetBSD source tree.  However,
there is nothing in the Kyua project that inherently depends on NetBSD, and
maintaining it as a third-party package is a way to keep the developers
honest regarding portability.

Ideally, other projects (such as FreeBSD) would make use of Kyua too for
their testing needs, and if that happened we would be able to share tests
with them pretty easily.  Forcing a portable codebase in the upstream
repository helps in this.  (Be aware that different individuals from
FreeBSD and Minix have shown interest in adopting Kyua for their respective
systems!)

# Main differences (aka "what to expect")

As of version 0.5, Kyua has (or is supposed to have) feature parity with
the ATF tools.  That said, having feature parity does not imply that they
are the same.  This section outlines a few of the differences that you
should be aware of before continuing.

## Results database

Kyua collects the results of the execution of a test suite into an SQLite
database.  User-friendly reports are later generated by extracting data
from this same database.

In ATF, the results of the execution were written to an internal format
that only atf-report could understand.  Despite of the database, Kyua still
maintains the separation of "tests execution" from "report generation".

The contents of the database are immutable and incremental.  This means
that, in the future, the Kyua tools will be able to provide historical data
for particular test cases, or for whole test runs (which is what other
NetBSD developers have ended up implementing multiple times outside of ATF
because the framework did not provide such functionality by itself).

## Support for multiple test interfaces

Kyua has support for different "test interfaces", which means that Kyua can
execute test programs written using different paradigms and collect their
results into a single report.  At the moment, two interfaces are supported:

* The "atf" interface provides compatibility with those test programs that
  use the ATF libraries.  This is the only interface currently used by the
  NetBSD test suite, as there is no way to run any other test program in an
  automated manner.

* The "plain" interface permits the execution of legacy test programs that
  do not use any testing library.  Such test programs are those that just
  return 0 or non-0 to indicate the success or failure of the test
  (respectively).  This feature will allow the NetBSD test suite to
  transparently execute third-party test suites (such as the IPF or GCC
  test suites) without having to implement ATF-based wrappers.  It will
  also lower the barrier of entry to writing test programs for NetBSD, as
  using the ATF libraries will become optional.

## Lua configuration files

Kyua has two kind of configuration files: the Kyuafiles, which are the
files shipped with a test suite that describe what test programs need to be
run; and the user configuration files, which specify the run-time settings
of Kyua and the test suites.  ATF had this same split of configuration
files, and they were written in a custom language, with a custom parser.

The Kyua configuration files are all Lua scripts.  The major advantage of
this at the moment is that their syntax will be familiar to end users, and
that the parser for these files is well-tested.  In the future, the use of
Lua will allow the implementation of more-intelligent test (and maybe even
build) scripts.

## Direct HTML output

All of the NetBSD continous build and testing systems provide status
reports through the releng web interface.  In the case of ATF, this has
traditionally been tricky because ATF cannot generate HTML contents
directly; instead, `atf-report` generates XML output which later must be
postprocessed with `xsltproc` to create the HTML pages.

Kyua has the ability to generate HTML reports straight from the tool,
without having to go through any XML toolchain.  This means that NetBSD,
out of the box, can generate such reports and publish them with the builtin
httpd(8) server.

## Heavier code base

If you take a look at the Kyua distribution file, you may notice that it is
about the same size as the distribution file of ATF, yet Kyua does not
currently replace the ATF libraries.  This may be surprising because it
seems to imply that the codebase of Kyua is bigger because it "just"
reimplements atf-run and atf-report: i.e. by just reimplementing parts of
ATF, it is already as big as the whole of ATF.

This is true, for two reasons.

The first is that Kyua is more featureful and flexible: the features
outlined above have a cost in terms of implementation, and the codebase of
Kyua is more carefully crafted to allow for later growth.  In particular,
all OS-specific details have been abstracted for easier portability, and
the SQLite and Lua libraries have been wrapped for safety.

The second is that Kyua is much better tested (which is very important for
a software package that you will rely on to validate your own software!).
To give you some numbers, ATF 0.16 contains around 400 test cases for both
atf-run and atf-report while Kyua 0.5 contains around 1100 test cases.

# Components

Kyua, as a project, is made up of a variety of components (which *include*
ATF, because the ATF libraries are *not* being rewritten).  All of these
components exist in pkgsrc, and are:

* pkgsrc/devel/atf-libs: The C, C++ and POSIX shell libraries provided by
  ATF.  These are *NOT* meant to be replaced by Kyua.

* pkgsrc/devel/atf: The ATF tools, namely atf-run and atf-report.  These
  are deprecated and this package should eventually disappear.

* pkgsrc/devel/kyua-cli: The Kyua command-line interface, which provides a
  superset of the functionality of atf-run and atf-report.

* pkgsrc/devel/kyua-atf-compat: Drop-in replacements for atf-run and
  atf-report that use kyua-cli in the backend.

# Running the NetBSD test suite

There are two ways to run the NetBSD test suite with Kyua.  The easy (or
trivial) way is to use the backwards compatibility ATF tools, and the more
sophisticated way is to convert the test suite to Kyua and use the native
Kyua binary.  This section explains both approaches.

## Using the ATF compatibility tools

The easiest (but also the least "future-proof") way to run the NetBSD test
suite with Kyua is to use the backwards compatibility ATF tools provided by
the kyua-atf-compat module.  First of all, install the package:

    $ cd /usr/pkgsrc/devel/kyua-atf-compat
    $ make install && make clean

And then, running the test suite is as easy as:

    $ cd /usr/tests
    $ /usr/pkg/bin/atf-run | /usr/pkg/bin/atf-report

Please be aware that if the atf-run and atf-report tools provided by
kyua-atf-compat appear in your PATH before the real atf-run and atf-report
tools shipped by NetBSD, you will experience test failures for all the
tests in /usr/tests/atf/atf-run and /usr/tests/atf/atf-report.  This is
expected: while the compatibility tools behave similarly to the real tools
from a user's perspective, they are not fully interchangeable.  (For
example, the serialization format between atf-run and atf-report is
different.)

One property of the atf-run wrapper is that it uses the default results
database in ~/.kyua/store.db to record the execution of the tests.  This
means that, once the execution of the tests is done with the compatibility
tools, you can still use the native Kyua binary to poke at the results
database.  More on this below.

## Using the native Kyua command-line interface

The preferred way to run the NetBSD test suite with Kyua is to use the
native Kyua command-line binary.  This is the preferred method because it
trains you to use the new interface rather than relying on the old pipeline
and because it exposes you to all the new features of Kyua.  Regardless,
this and the previous approach will yield the same results for a particular
execution.

Using the native command-line interface is a multi-step process because
the existing NetBSD test suite is not prepared for Kyua.  Let's take a look
at these steps.

To get started, install the Kyua packages:

    $ cd /usr/pkgsrc/devel/kyua-cli
    $ make install && make clean
    $ cd /usr/pkgsrc/devel/kyua-atf-compat
    $ make install && make clean

Once this is done, configure Kyua in the same way ATF is configured "out of
the box" in NetBSD.  Create the /usr/pkg/etc/kyua/kyua.conf file with these
contents:

    syntax('config', 1)
    unprivileged_user = '_tests'

The next step is to populate /usr/tests with Kyuafiles, as Kyua is unable
to read existing Atffiles.  This is easy to do with the atf2kyua(1) tool
shipped in the kyua-atf-compat package:

    # atf2kyua /usr/tests

And that is it.  You can now execute the test suite using Kyua with any of
the following two forms:

    $ cd /usr/tests && kyua test
    $ kyua test -k /usr/tests/Kyuafile

Note that none of these will generate "pretty" reports.  These commands
will only record the results of the execution into the database.  In order
to generate reports, keep reading.

# Generating reports

Once you have ran the NetBSD test suite with any of the mechanisms above,
the results of the execution have been stored in the "Kyua store", which is
a database located in ~/.kyua/store.db by default.  (This path can be
changed at any time with the --store flag.)

To extract a report from the database using the results of the latest tests
run, you can run any of the following:

    $ kyua report -o my-report.txt
    $ kyua html-report -o /var/www/results/

# Support and feedback

The Kyua manual is available in the GNU Info format and can be accessed by
running:

    $ info kyua

Alternatively, use the help subcommand to get built-in documentation.  The
following invocation will print all the available subcommands:

    $ kyua help

And an invocation like this will show you all the possible options for a
given subcommand:

    $ kyua help report-html

If you have gone through the instructions above and started playing with
Kyua, please do not hesitate to report your experiences (either good or
bad) to [Julio Merino](mailto:jmmv@NetBSD.org)!  Any comments will be
highly appreciated and will be taken into account for the near future of
Kyua.

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb