1: [[!meta title="Kyua: An introduction for NetBSD users"]]
5: [[!toc levels=2]]
7: The [Automated Testing Framework](ATF), or ATF for short, is a software
8: package composed of two parts: the *ATF libraries* and the *ATF tools*.
9: The ATF libraries provide a toolkit for developers to implement test cases
10: in a variety of languages: C, C++ and POSIX shell. The ATF tools provide
11: the utilities to run such test cases in an automated way and to generate
14: The ATF tools have some
15: [design and, particularly, implementation problems](http://mail-index.netbsd.org/atf-devel/2010/11/13/msg000206.html)
16: that make it hard to add support for highly desired features such as
17: parallel execution of test cases, unified dashboards covering multiple test
18: runs ([like this one](http://releng.netbsd.org/test-results.html)), the
19: ability to run legacy or third-party test programs that do not use the ATF
20: libraries, and the ability to tune the timeout of test cases.
22: *Kyua's current goal is to reimplement _only_ the ATF tools* while
23: maintaining backwards compatibility with the tests written with the ATF
24: libraries (i.e. with the NetBSD test suite).
26: Because Kyua is a replacement of some ATF components, the end goal is to
27: integrate Kyua into the NetBSD base system (just as ATF is) and remove the
28: deprecated ATF components. Removing the deprecated components will allow
29: us to make the above-mentioned improvements to Kyua, as well as many
30: others, without having to deal with the obsolete ATF code base.
31: *Discussing how and when this transition might happen is out of the scope
32: of this document at the moment.*
34: This page provides instructions on how to use Kyua with the current NetBSD
35: test suite so that you can experiment with the tool, familiarize yourself
36: with it and provide feedback early on.
38: **If you would like more details on how the import of Kyua into NetBSD will
39: happen, please see [[Kyua: The way into NetBSD|/kyua/import]].**
41: # What's in the name?
43: You should really think of Kyua as ATF 2.x. Then, why isn't it just ATF
46: To be honest, I never liked the ATF name: it was picked for me as part of
47: the Google Summer of Code 2007 program and I did not think about changing
48: it at that time. A year later, I learned that the ATF acronym is severely
49: overloaded, which makes it hard to find the project on popular search
50: engines, and has unpopular connotations in specific countries.
52: So, as part of the rewrite, I decided to choose a new name: a name that is
53: not an acronym and thus can be easily pronounced, and a name that is quite
54: unique in search results. The name is Kyua, which is just a play on the
55: pronounciation of the Q.A. acronym. Originally, my intention was to
56: pronounce Kyua as Q.A., but in reality this never happened. Today, just
57: read the name as your instinct would: "Kyu-ah".
59: # Why is Kyua a third-party project?
61: Kyua's main consumer is NetBSD. One could argue that Kyua should be
62: developed within NetBSD and maintained in the NetBSD source tree. However,
63: there is nothing in the Kyua project that inherently depends on NetBSD, and
64: maintaining it as a third-party package is a way to keep the developers
65: honest regarding portability.
67: Ideally, other projects (such as FreeBSD) would make use of Kyua too for
68: their testing needs, and if that happened we would be able to share tests
69: with them pretty easily. Forcing a portable codebase in the upstream
70: repository helps in this. (Be aware that different individuals from
71: FreeBSD and Minix have shown interest in adopting Kyua for their respective
74: # Main differences (aka "what to expect")
76: As of version 0.5, Kyua has (or is supposed to have) feature parity with
77: the ATF tools. That said, having feature parity does not imply that they
78: are the same. This section outlines a few of the differences that you
79: should be aware of before continuing.
81: ## Results database
83: Kyua collects the results of the execution of a test suite into an SQLite
84: database. User-friendly reports are later generated by extracting data
85: from this same database.
87: In ATF, the results of the execution were written to an internal format
88: that only atf-report could understand. Despite of the database, Kyua still
89: maintains the separation of "tests execution" from "report generation".
91: The contents of the database are immutable and incremental. This means
92: that, in the future, the Kyua tools will be able to provide historical data
93: for particular test cases, or for whole test runs (which is what other
94: NetBSD developers have ended up implementing multiple times outside of ATF
95: because the framework did not provide such functionality by itself).
97: ## Support for multiple test interfaces
99: Kyua has support for different "test interfaces", which means that Kyua can
100: execute test programs written using different paradigms and collect their
101: results into a single report. At the moment, two interfaces are supported:
103: * The "atf" interface provides compatibility with those test programs that
104: use the ATF libraries. This is the only interface currently used by the
105: NetBSD test suite, as there is no way to run any other test program in an
106: automated manner.
108: * The "plain" interface permits the execution of legacy test programs that
109: do not use any testing library. Such test programs are those that just
110: return 0 or non-0 to indicate the success or failure of the test
111: (respectively). This feature will allow the NetBSD test suite to
112: transparently execute third-party test suites (such as the IPF or GCC
113: test suites) without having to implement ATF-based wrappers. It will
114: also lower the barrier of entry to writing test programs for NetBSD, as
115: using the ATF libraries will become optional.
117: These interfaces are implemented as independent binaries, called testers,
118: that are fully scriptable. The testers are provided in the `kyua-testers`
119: package. The idea behind having these as independent programs is to
120: restrict the OS-specific code to a small subset of Kyua written in C, and
121: thus to allow the higher-level layers to be written in other languages
122: (possibly Lua).
124: ## Lua configuration files
126: Kyua has two kind of configuration files: the Kyuafiles, which are the
127: files shipped with a test suite that describe what test programs need to be
128: run; and the user configuration files, which specify the run-time settings
129: of Kyua and the test suites. ATF had this same split of configuration
130: files, and they were written in a custom language, with a custom parser.
132: The Kyua configuration files are all Lua scripts. The major advantage of
133: this at the moment is that their syntax will be familiar to end users, and
134: that the parser for these files is well-tested. In the future, the use of
135: Lua will allow the implementation of more-intelligent test (and maybe even
136: build) scripts.
138: ## Direct HTML output
140: All of the NetBSD continous build and testing systems provide status
141: reports through the releng web interface. In the case of ATF, this has
142: traditionally been tricky because ATF cannot generate HTML contents
143: directly; instead, `atf-report` generates XML output which later must be
144: postprocessed with `xsltproc` to create the HTML pages.
146: Kyua has the ability to generate HTML reports straight from the tool,
147: without having to go through any XML toolchain. This means that NetBSD,
148: out of the box, can generate such reports and publish them with the builtin
149: httpd(8) server.
151: ## Heavier code base
153: If you take a look at the Kyua distribution file, you may notice that it is
154: about the same size as the distribution file of ATF, yet Kyua does not
155: currently replace the ATF libraries. This may be surprising because it
156: seems to imply that the codebase of Kyua is bigger because it "just"
157: reimplements atf-run and atf-report: i.e. by just reimplementing parts of
158: ATF, it is already as big as the whole of ATF.
160: This is true, for two reasons.
162: The first is that Kyua is more featureful and flexible: the features
163: outlined above have a cost in terms of implementation, and the codebase of
164: Kyua is more carefully crafted to allow for later growth. In particular,
165: all OS-specific details have been abstracted for easier portability, and
166: the SQLite and Lua libraries have been wrapped for safety.
168: The second is that Kyua is much better tested (which is very important for
169: a software package that you will rely on to validate your own software!).
170: To give you some numbers, ATF 0.16 contains around 400 test cases for both
171: atf-run and atf-report while Kyua 0.5 contains around 1100 test cases.
173: # Components
175: Kyua, as a project, is made up of a variety of components (which *include*
176: ATF, because the ATF libraries are *not* being rewritten). All of these
177: components exist in pkgsrc, and are:
179: * pkgsrc/devel/atf-libs: The C, C++ and POSIX shell libraries provided by
180: ATF. These are *NOT* meant to be replaced by Kyua.
182: * pkgsrc/devel/atf: The ATF tools, namely atf-run and atf-report. These
183: are deprecated and this package should eventually disappear.
185: * pkgsrc/devel/kyua-testers: The Kyua testers, which provide the
186: `kyua-atf-tester` and `kyua-plain-tester` helper binaries. These
187: binaries implement the logic to execute test cases in an isolated manner
188: and to expose the test programs using a common and abstract command-line
191: * pkgsrc/devel/kyua-cli: The Kyua command-line interface, which provides a
192: superset of the functionality of atf-run and atf-report.
194: * pkgsrc/devel/kyua-atf-compat: Drop-in replacements for atf-run and
195: atf-report that use kyua-cli in the backend.
197: # Running the NetBSD test suite
199: There are two ways to run the NetBSD test suite with Kyua. The easy (or
200: trivial) way is to use the backwards compatibility ATF tools, and the more
201: sophisticated way is to convert the test suite to Kyua and use the native
202: Kyua binary. This section explains both approaches.
204: ## Using the ATF compatibility tools
206: The easiest (but also the least "future-proof") way to run the NetBSD test
207: suite with Kyua is to use the backwards compatibility ATF tools provided by
208: the kyua-atf-compat module. First of all, install the package:
210: $ cd /usr/pkgsrc/devel/kyua-atf-compat
211: $ make install && make clean
213: And then, running the test suite is as easy as:
215: $ cd /usr/tests
216: $ /usr/pkg/bin/atf-run | /usr/pkg/bin/atf-report
218: Please be aware that if the atf-run and atf-report tools provided by
219: kyua-atf-compat appear in your PATH before the real atf-run and atf-report
220: tools shipped by NetBSD, you will experience test failures for all the
221: tests in /usr/tests/atf/atf-run and /usr/tests/atf/atf-report. This is
222: expected: while the compatibility tools behave similarly to the real tools
223: from a user's perspective, they are not fully interchangeable. (For
224: example, the serialization format between atf-run and atf-report is
227: One property of the atf-run wrapper is that it uses the default results
228: database in ~/.kyua/store.db to record the execution of the tests. This
229: means that, once the execution of the tests is done with the compatibility
230: tools, you can still use the native Kyua binary to poke at the results
231: database. More on this below.
233: ## Using the native Kyua command-line interface
235: The preferred way to run the NetBSD test suite with Kyua is to use the
236: native Kyua command-line binary. This is the preferred method because it
237: trains you to use the new interface rather than relying on the old pipeline
238: and because it exposes you to all the new features of Kyua. Regardless,
239: this and the previous approach will yield the same results for a particular
242: Using the native command-line interface is a multi-step process because
243: the existing NetBSD test suite is not prepared for Kyua. Let's take a look
244: at these steps.
246: To get started, install the Kyua packages:
248: $ cd /usr/pkgsrc/devel/kyua-cli
249: $ make install && make clean
250: $ cd /usr/pkgsrc/devel/kyua-atf-compat
251: $ make install && make clean
253: Once this is done, configure Kyua in the same way ATF is configured "out of
254: the box" in NetBSD. Create the /usr/pkg/etc/kyua/kyua.conf file with these
257: syntax('config', 1)
258: unprivileged_user = '_tests'
260: The next step is to populate /usr/tests with Kyuafiles, as Kyua is unable
261: to read existing Atffiles. This is easy to do with the atf2kyua(1) tool
262: shipped in the kyua-atf-compat package:
264: # atf2kyua /usr/tests
266: And that is it. You can now execute the test suite using Kyua with any of
267: the following two forms:
269: $ cd /usr/tests && kyua test
270: $ kyua test -k /usr/tests/Kyuafile
272: Note that none of these will generate "pretty" reports. These commands
273: will only record the results of the execution into the database. In order
274: to generate reports, keep reading.
276: # Generating reports
278: Once you have ran the NetBSD test suite with any of the mechanisms above,
279: the results of the execution have been stored in the "Kyua store", which is
280: a database located in ~/.kyua/store.db by default. (This path can be
281: changed at any time with the --store flag.)
283: To extract a report from the database using the results of the latest tests
284: run, you can run any of the following:
286: $ kyua report -o my-report.txt
287: $ kyua html-report -o /var/www/results/
289: # Support and feedback
291: The Kyua manual is available in the GNU Info format and can be accessed by
294: $ info kyua
296: Alternatively, use the help subcommand to get built-in documentation. The
297: following invocation will print all the available subcommands:
299: $ kyua help
301: And an invocation like this will show you all the possible options for a
302: given subcommand:
304: $ kyua help report-html
306: If you have gone through the instructions above and started playing with
307: Kyua, please do not hesitate to report your experiences (either good or
308: bad) to [Julio Merino](mailto:jmmv@NetBSD.org)! Any comments will be
309: highly appreciated and will be taken into account for the near future of
CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb