Annotation of wikisrc/kyua.mdwn, revision 1.1
1.1 ! jmmv 1: [[!meta title="Kyua: An introduction for NetBSD users"]]
! 2: [[!toc ]]
! 3:
! 4: The [Automated Testing Framework](ATF), or ATF for short, is a software
! 5: package composed of two parts: the *ATF libraries* and the *ATF tools*.
! 6: The ATF libraries provide a toolkit for developers to implement test cases
! 7: in a variety of languages: C, C++ and POSIX shell. The ATF tools provide
! 8: the utilities to run such test cases in an automated way and to generate
! 9: reports.
! 10:
! 11: The ATF tools have some
! 12: [design and, particularly, implementation problems](http://mail-index.netbsd.org/atf-devel/2010/11/13/msg000206.html)
! 13: that make it hard to add support for highly desired features such as
! 14: parallel execution of test cases, unified dashboards covering multiple test
! 15: runs ([like this one](http://releng.netbsd.org/test-results.html)), the
! 16: ability to run legacy or third-party test programs that do not use the ATF
! 17: libraries, and the ability to tune the timeout of test cases.
! 18:
! 19: *Kyua's current goal is to reimplement _only_ the ATF tools* while
! 20: maintaining backwards compatibility with the tests written with the ATF
! 21: libraries (i.e. with the NetBSD test suite).
! 22:
! 23: Because Kyua is a replacement of some ATF components, the end goal is to
! 24: integrate Kyua into the NetBSD base system (just as ATF is) and remove the
! 25: deprecated ATF components. Removing the deprecated components will allow
! 26: us to make the above-mentioned improvements to Kyua, as well as many
! 27: others, without having to deal with the obsolete ATF code base.
! 28: *Discussing how and when this transition might happen is out of the scope
! 29: of this document at the moment.*
! 30:
! 31: This page provides instructions on how to use Kyua with the current NetBSD
! 32: test suite so that you can experiment with the tool, familiarize yourself
! 33: with it and provide feedback early on.
! 34:
! 35: # Main differences (aka "what to expect")
! 36:
! 37: As of version 0.5, Kyua has (or is supposed to have) feature parity with
! 38: the ATF tools. That said, having feature parity does not imply that they
! 39: are the same. This section outlines a few of the differences that you
! 40: should be aware of before continuing.
! 41:
! 42: ## Results database
! 43:
! 44: Kyua collects the results of the execution of a test suite into an SQLite
! 45: database. User-friendly reports are later generated by extracting data
! 46: from this same database.
! 47:
! 48: In ATF, the results of the execution were written to an internal format
! 49: that only atf-report could understand. Despite of the database, Kyua still
! 50: maintains the separation of "tests execution" from "report generation".
! 51:
! 52: The contents of the database are immutable and incremental. This means
! 53: that, in the future, the Kyua tools will be able to provide historical data
! 54: for particular test cases, or for whole test runs (which is what other
! 55: NetBSD developers have ended up implementing multiple times outside of ATF
! 56: because the framework did not provide such functionality by itself).
! 57:
! 58: ## Support for multiple test interfaces
! 59:
! 60: Kyua has support for different "test interfaces", which means that Kyua can
! 61: execute test programs written using different paradigms and collect their
! 62: results into a single report. At the moment, two interfaces are supported:
! 63:
! 64: * The "atf" interface provides compatibility with those test programs that
! 65: use the ATF libraries. This is the only interface currently used by the
! 66: NetBSD test suite, as there is no way to run any other test program in an
! 67: automated manner.
! 68:
! 69: * The "plain" interface permits the execution of legacy test programs that
! 70: do not use any testing library. Such test programs are those that just
! 71: return 0 or non-0 to indicate the success or failure of the test
! 72: (respectively). This feature will allow the NetBSD test suite to
! 73: transparently execute third-party test suites (such as the IPF or GCC
! 74: test suites) without having to implement ATF-based wrappers. It will
! 75: also lower the barrier of entry to writing test programs for NetBSD, as
! 76: using the ATF libraries will become optional.
! 77:
! 78: ## Lua configuration files
! 79:
! 80: Kyua has two kind of configuration files: the Kyuafiles, which are the
! 81: files shipped with a test suite that describe what test programs need to be
! 82: run; and the user configuration files, which specify the run-time settings
! 83: of Kyua and the test suites. ATF had this same split of configuration
! 84: files, and they were written in a custom language, with a custom parser.
! 85:
! 86: The Kyua configuration files are all Lua scripts. The major advantage of
! 87: this at the moment is that their syntax will be familiar to end users, and
! 88: that the parser for these files is well-tested. In the future, the use of
! 89: Lua will allow the implementation of more-intelligent test (and maybe even
! 90: build) scripts.
! 91:
! 92: ## Heavier code base
! 93:
! 94: If you take a look at the Kyua distribution file, you may notice that it is
! 95: about the same size as the distribution file of ATF, yet Kyua does not
! 96: currently replace the ATF libraries. This may be surprising because it
! 97: seems to imply that the codebase of Kyua is bigger because it "just"
! 98: reimplements atf-run and atf-report: i.e. by just reimplementing parts of
! 99: ATF, it is already as big as the whole of ATF.
! 100:
! 101: This is true, for two reasons.
! 102:
! 103: The first is that Kyua is more featureful and flexible: the features
! 104: outlined above have a cost in terms of implementation, and the codebase of
! 105: Kyua is more carefully crafted to allow for later growth. In particular,
! 106: all OS-specific details have been abstracted for easier portability, and
! 107: the SQLite and Lua libraries have been wrapped for safety.
! 108:
! 109: The second is that Kyua is much better tested (which is very important for
! 110: a software package that you will rely on to validate your own software!).
! 111: To give you some numbers, ATF 0.16 contains around 400 test cases for both
! 112: atf-run and atf-report while Kyua 0.5 contains around 1100 test cases.
! 113:
! 114: # Components
! 115:
! 116: Kyua, as a project, is made up of a variety of components (which *include*
! 117: ATF, because the ATF libraries are *not* being rewritten). All of these
! 118: components exist in pkgsrc, and are:
! 119:
! 120: * pkgsrc/devel/atf-libs: The C, C++ and POSIX shell libraries provided by
! 121: ATF. These are *NOT* meant to be replaced by Kyua.
! 122:
! 123: * pkgsrc/devel/atf: The ATF tools, namely atf-run and atf-report. These
! 124: are deprecated and this package should eventually disappear.
! 125:
! 126: * pkgsrc/devel/kyua-cli: The Kyua command-line interface, which provides a
! 127: superset of the functionality of atf-run and atf-report.
! 128:
! 129: * pkgsrc/devel/kyua-atf-compat: Drop-in replacements for atf-run and
! 130: atf-report that use kyua-cli in the backend.
! 131:
! 132: # Running the NetBSD test suite
! 133:
! 134: There are two ways to run the NetBSD test suite with Kyua. The easy (or
! 135: trivial) way is to use the backwards compatibility ATF tools, and the more
! 136: sophisticated way is to convert the test suite to Kyua and use the native
! 137: Kyua binary. This section explains both approaches.
! 138:
! 139: ## Using the ATF compatibility tools
! 140:
! 141: The easiest (but also the least "future-proof") way to run the NetBSD test
! 142: suite with Kyua is to use the backwards compatibility ATF tools provided by
! 143: the kyua-atf-compat module. First of all, install the package:
! 144:
! 145: $ cd /usr/pkgsrc/deve/kyua-atf-compat
! 146: $ make install && make clean
! 147:
! 148: And then, running the test suite is as easy as:
! 149:
! 150: $ cd /usr/tests
! 151: $ /usr/pkg/bin/atf-run | /usr/pkg/bin/atf-report
! 152:
! 153: Please be aware that if the atf-run and atf-report tools provided by
! 154: kyua-atf-compat appear in your PATH before the real atf-run and atf-report
! 155: tools shipped by NetBSD, you will experience test failures for all the
! 156: tests in /usr/tests/atf/atf-run and /usr/tests/atf/atf-report. This is
! 157: expected: while the compatibility tools behave similarly to the real tools
! 158: from a user's perspective, they are not fully interchangeable. (For
! 159: example, the serialization format between atf-run and atf-report is
! 160: different.)
! 161:
! 162: One property of the atf-run wrapper is that it uses the default results
! 163: database in ~/.kyua/store.db to record the execution of the tests. This
! 164: means that, once the execution of the tests is done with the compatibility
! 165: tools, you can still use the native Kyua binary to poke at the results
! 166: database. More on this below.
! 167:
! 168: ## Using the native Kyua command-line interface
! 169:
! 170: The preferred way to run the NetBSD test suite with Kyua is to use the
! 171: native Kyua command-line binary. This is the preferred method because it
! 172: trains you to use the new interface rather than relying on the old pipeline
! 173: and because it exposes you to all the new features of Kyua. Regardless,
! 174: this and the previous approach will yield the same results for a particular
! 175: execution.
! 176:
! 177: Using the native command-line interface is a multi-step process because
! 178: the existing NetBSD test suite is not prepared for Kyua. Let's take a look
! 179: at these steps.
! 180:
! 181: To get started, install the Kyua packages:
! 182:
! 183: $ cd /usr/pkgsrc/deve/kyua-cli
! 184: $ make install && make clean
! 185: $ cd /usr/pkgsrc/deve/kyua-atf-compat
! 186: $ make install && make clean
! 187:
! 188: Once this is done, configure Kyua in the same way ATF is configured "out of
! 189: the box" in NetBSD. Create the /usr/pkg/etc/kyua/kyua.conf file with these
! 190: contents:
! 191:
! 192: syntax('kyuafile', 1)
! 193: unprivileged_user = '_tests'
! 194:
! 195: The next step is to populate /usr/tests with Kyuafiles, as Kyua is unable
! 196: to read existing Atffiles. This is easy to do with the atf2kyua(1) tool
! 197: shipped in the kyua-atf-compat package:
! 198:
! 199: # atf2kyua /usr/tests
! 200:
! 201: And that is it. You can now execute the test suite using Kyua with any of
! 202: the following two forms:
! 203:
! 204: $ cd /usr/tests && kyua test
! 205: $ kyua test -k /usr/tests/Kyuafile
! 206:
! 207: Note that none of these will generate "pretty" reports. These commands
! 208: will only record the results of the execution into the database. In order
! 209: to generate reports, keep reading.
! 210:
! 211: # Generating reports
! 212:
! 213: Once you have ran the NetBSD test suite with any of the mechanisms above,
! 214: the results of the execution have been stored in the "Kyua store", which is
! 215: a database located in ~/.kyua/store.db by default. (This path can be
! 216: changed at any time with the --store flag.)
! 217:
! 218: To extract a report from the database using the results of the latest tests
! 219: run, you can run any of the following:
! 220:
! 221: $ kyua report -o my-report.txt
! 222: $ kyua html-report -o /var/www/results/
! 223:
! 224: # Support and feedback
! 225:
! 226: The Kyua manual is available in the GNU Info format and can be accessed by
! 227: running:
! 228:
! 229: $ info kyua
! 230:
! 231: Alternatively, use the help subcommand to get built-in documentation. The
! 232: following invocation will print all the available subcommands:
! 233:
! 234: $ kyua help
! 235:
! 236: And an invocation like this will show you all the possible options for a
! 237: given subcommand:
! 238:
! 239: $ kyua help report-html
! 240:
! 241: If you have gone through the instructions above and started playing with
! 242: Kyua, please do not hesitate to report your experiences (either good or
! 243: bad) to [Julio Merino](mailto:jmmv@NetBSD.org)! Any comments will be
! 244: highly appreciated and will be taken into account for the near future of
! 245: Kyua.
CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb
![[BACK]](/icons/back.gif){kind=icon}
Return to kyua.mdwn CVS log ![[TXT]](/icons/text.gif){kind=icon}
![[DIR]](/icons/dir.gif){kind=icon}
Up to [NetBSD Developer Wiki] / wikisrc