Annotation of wikisrc/kyua.mdwn, revision 1.9

1.1       jmmv        1: [[!meta title="Kyua: An introduction for NetBSD users"]]
1.9     ! wiki        2: 
        !             3: **Contents**
        !             4: 
1.4       jmmv        5: [[!toc levels=2]]
1.1       jmmv        6: 
                      7: The [Automated Testing Framework](ATF), or ATF for short, is a software
                      8: package composed of two parts: the *ATF libraries* and the *ATF tools*.
                      9: The ATF libraries provide a toolkit for developers to implement test cases
                     10: in a variety of languages: C, C++ and POSIX shell.  The ATF tools provide
                     11: the utilities to run such test cases in an automated way and to generate
                     12: reports.
                     13: 
                     14: The ATF tools have some
                     15: [design and, particularly, implementation problems](http://mail-index.netbsd.org/atf-devel/2010/11/13/msg000206.html)
                     16: that make it hard to add support for highly desired features such as
                     17: parallel execution of test cases, unified dashboards covering multiple test
                     18: runs ([like this one](http://releng.netbsd.org/test-results.html)), the
                     19: ability to run legacy or third-party test programs that do not use the ATF
                     20: libraries, and the ability to tune the timeout of test cases.
                     21: 
                     22: *Kyua's current goal is to reimplement _only_ the ATF tools* while
                     23: maintaining backwards compatibility with the tests written with the ATF
                     24: libraries (i.e. with the NetBSD test suite).
                     25: 
                     26: Because Kyua is a replacement of some ATF components, the end goal is to
                     27: integrate Kyua into the NetBSD base system (just as ATF is) and remove the
                     28: deprecated ATF components.  Removing the deprecated components will allow
                     29: us to make the above-mentioned improvements to Kyua, as well as many
                     30: others, without having to deal with the obsolete ATF code base.
                     31: *Discussing how and when this transition might happen is out of the scope
                     32: of this document at the moment.*
                     33: 
                     34: This page provides instructions on how to use Kyua with the current NetBSD
                     35: test suite so that you can experiment with the tool, familiarize yourself
                     36: with it and provide feedback early on.
                     37: 
1.5       jmmv       38: **If you would like more details on how the import of Kyua into NetBSD will
1.6       jmmv       39:   happen, please see [[Kyua: The way into NetBSD|/kyua/import]].**
1.5       jmmv       40: 
1.2       jmmv       41: # What's in the name?
                     42: 
                     43: You should really think of Kyua as ATF 2.x.  Then, why isn't it just ATF
                     44: 2.x?
                     45: 
                     46: To be honest, I never liked the ATF name: it was picked for me as part of
                     47: the Google Summer of Code 2007 program and I did not think about changing
                     48: it at that time.  A year later, I learned that the ATF acronym is severely
                     49: overloaded, which makes it hard to find the project on popular search
                     50: engines, and has unpopular connotations in specific countries.
                     51: 
                     52: So, as part of the rewrite, I decided to choose a new name: a name that is
                     53: not an acronym and thus can be easily pronounced, and a name that is quite
                     54: unique in search results.  The name is Kyua, which is just a play on the
                     55: pronounciation of the Q.A. acronym.  Originally, my intention was to
                     56: pronounce Kyua as Q.A., but in reality this never happened.  Today, just
                     57: read the name as your instinct would: "Kyu-ah".
                     58: 
1.4       jmmv       59: # Why is Kyua a third-party project?
                     60: 
                     61: Kyua's main consumer is NetBSD.  One could argue that Kyua should be
                     62: developed within NetBSD and maintained in the NetBSD source tree.  However,
                     63: there is nothing in the Kyua project that inherently depends on NetBSD, and
                     64: maintaining it as a third-party package is a way to keep the developers
                     65: honest regarding portability.
                     66: 
                     67: Ideally, other projects (such as FreeBSD) would make use of Kyua too for
                     68: their testing needs, and if that happened we would be able to share tests
                     69: with them pretty easily.  Forcing a portable codebase in the upstream
                     70: repository helps in this.  (Be aware that different individuals from
                     71: FreeBSD and Minix have shown interest in adopting Kyua for their respective
                     72: systems!)
                     73: 
1.1       jmmv       74: # Main differences (aka "what to expect")
                     75: 
                     76: As of version 0.5, Kyua has (or is supposed to have) feature parity with
                     77: the ATF tools.  That said, having feature parity does not imply that they
                     78: are the same.  This section outlines a few of the differences that you
                     79: should be aware of before continuing.
                     80: 
                     81: ## Results database
                     82: 
                     83: Kyua collects the results of the execution of a test suite into an SQLite
                     84: database.  User-friendly reports are later generated by extracting data
                     85: from this same database.
                     86: 
                     87: In ATF, the results of the execution were written to an internal format
                     88: that only atf-report could understand.  Despite of the database, Kyua still
                     89: maintains the separation of "tests execution" from "report generation".
                     90: 
                     91: The contents of the database are immutable and incremental.  This means
                     92: that, in the future, the Kyua tools will be able to provide historical data
                     93: for particular test cases, or for whole test runs (which is what other
                     94: NetBSD developers have ended up implementing multiple times outside of ATF
                     95: because the framework did not provide such functionality by itself).
                     96: 
                     97: ## Support for multiple test interfaces
                     98: 
                     99: Kyua has support for different "test interfaces", which means that Kyua can
                    100: execute test programs written using different paradigms and collect their
                    101: results into a single report.  At the moment, two interfaces are supported:
                    102: 
                    103: * The "atf" interface provides compatibility with those test programs that
                    104:   use the ATF libraries.  This is the only interface currently used by the
                    105:   NetBSD test suite, as there is no way to run any other test program in an
                    106:   automated manner.
                    107: 
                    108: * The "plain" interface permits the execution of legacy test programs that
                    109:   do not use any testing library.  Such test programs are those that just
                    110:   return 0 or non-0 to indicate the success or failure of the test
                    111:   (respectively).  This feature will allow the NetBSD test suite to
                    112:   transparently execute third-party test suites (such as the IPF or GCC
                    113:   test suites) without having to implement ATF-based wrappers.  It will
                    114:   also lower the barrier of entry to writing test programs for NetBSD, as
                    115:   using the ATF libraries will become optional.
                    116: 
1.8       jmmv      117: These interfaces are implemented as independent binaries, called testers,
                    118: that are fully scriptable.  The testers are provided in the `kyua-testers`
                    119: package.  The idea behind having these as independent programs is to
                    120: restrict the OS-specific code to a small subset of Kyua written in C, and
                    121: thus to allow the higher-level layers to be written in other languages
                    122: (possibly Lua).
                    123: 
1.1       jmmv      124: ## Lua configuration files
                    125: 
                    126: Kyua has two kind of configuration files: the Kyuafiles, which are the
                    127: files shipped with a test suite that describe what test programs need to be
                    128: run; and the user configuration files, which specify the run-time settings
                    129: of Kyua and the test suites.  ATF had this same split of configuration
                    130: files, and they were written in a custom language, with a custom parser.
                    131: 
                    132: The Kyua configuration files are all Lua scripts.  The major advantage of
                    133: this at the moment is that their syntax will be familiar to end users, and
                    134: that the parser for these files is well-tested.  In the future, the use of
                    135: Lua will allow the implementation of more-intelligent test (and maybe even
                    136: build) scripts.
                    137: 
1.4       jmmv      138: ## Direct HTML output
                    139: 
                    140: All of the NetBSD continous build and testing systems provide status
                    141: reports through the releng web interface.  In the case of ATF, this has
                    142: traditionally been tricky because ATF cannot generate HTML contents
                    143: directly; instead, `atf-report` generates XML output which later must be
                    144: postprocessed with `xsltproc` to create the HTML pages.
                    145: 
                    146: Kyua has the ability to generate HTML reports straight from the tool,
                    147: without having to go through any XML toolchain.  This means that NetBSD,
                    148: out of the box, can generate such reports and publish them with the builtin
                    149: httpd(8) server.
                    150: 
1.1       jmmv      151: ## Heavier code base
                    152: 
                    153: If you take a look at the Kyua distribution file, you may notice that it is
                    154: about the same size as the distribution file of ATF, yet Kyua does not
                    155: currently replace the ATF libraries.  This may be surprising because it
                    156: seems to imply that the codebase of Kyua is bigger because it "just"
                    157: reimplements atf-run and atf-report: i.e. by just reimplementing parts of
                    158: ATF, it is already as big as the whole of ATF.
                    159: 
                    160: This is true, for two reasons.
                    161: 
                    162: The first is that Kyua is more featureful and flexible: the features
                    163: outlined above have a cost in terms of implementation, and the codebase of
                    164: Kyua is more carefully crafted to allow for later growth.  In particular,
                    165: all OS-specific details have been abstracted for easier portability, and
                    166: the SQLite and Lua libraries have been wrapped for safety.
                    167: 
                    168: The second is that Kyua is much better tested (which is very important for
                    169: a software package that you will rely on to validate your own software!).
                    170: To give you some numbers, ATF 0.16 contains around 400 test cases for both
                    171: atf-run and atf-report while Kyua 0.5 contains around 1100 test cases.
                    172: 
                    173: # Components
                    174: 
                    175: Kyua, as a project, is made up of a variety of components (which *include*
                    176: ATF, because the ATF libraries are *not* being rewritten).  All of these
                    177: components exist in pkgsrc, and are:
                    178: 
                    179: * pkgsrc/devel/atf-libs: The C, C++ and POSIX shell libraries provided by
                    180:   ATF.  These are *NOT* meant to be replaced by Kyua.
                    181: 
                    182: * pkgsrc/devel/atf: The ATF tools, namely atf-run and atf-report.  These
                    183:   are deprecated and this package should eventually disappear.
                    184: 
1.8       jmmv      185: * pkgsrc/devel/kyua-testers: The Kyua testers, which provide the
                    186:   `kyua-atf-tester` and `kyua-plain-tester` helper binaries.  These
                    187:   binaries implement the logic to execute test cases in an isolated manner
                    188:   and to expose the test programs using a common and abstract command-line
                    189:   interface.
                    190: 
1.1       jmmv      191: * pkgsrc/devel/kyua-cli: The Kyua command-line interface, which provides a
                    192:   superset of the functionality of atf-run and atf-report.
                    193: 
                    194: * pkgsrc/devel/kyua-atf-compat: Drop-in replacements for atf-run and
                    195:   atf-report that use kyua-cli in the backend.
                    196: 
                    197: # Running the NetBSD test suite
                    198: 
                    199: There are two ways to run the NetBSD test suite with Kyua.  The easy (or
                    200: trivial) way is to use the backwards compatibility ATF tools, and the more
                    201: sophisticated way is to convert the test suite to Kyua and use the native
                    202: Kyua binary.  This section explains both approaches.
                    203: 
                    204: ## Using the ATF compatibility tools
                    205: 
                    206: The easiest (but also the least "future-proof") way to run the NetBSD test
                    207: suite with Kyua is to use the backwards compatibility ATF tools provided by
                    208: the kyua-atf-compat module.  First of all, install the package:
                    209: 
1.7       riz       210:     $ cd /usr/pkgsrc/devel/kyua-atf-compat
1.1       jmmv      211:     $ make install && make clean
                    212: 
                    213: And then, running the test suite is as easy as:
                    214: 
                    215:     $ cd /usr/tests
                    216:     $ /usr/pkg/bin/atf-run | /usr/pkg/bin/atf-report
                    217: 
                    218: Please be aware that if the atf-run and atf-report tools provided by
                    219: kyua-atf-compat appear in your PATH before the real atf-run and atf-report
                    220: tools shipped by NetBSD, you will experience test failures for all the
                    221: tests in /usr/tests/atf/atf-run and /usr/tests/atf/atf-report.  This is
                    222: expected: while the compatibility tools behave similarly to the real tools
                    223: from a user's perspective, they are not fully interchangeable.  (For
                    224: example, the serialization format between atf-run and atf-report is
                    225: different.)
                    226: 
                    227: One property of the atf-run wrapper is that it uses the default results
                    228: database in ~/.kyua/store.db to record the execution of the tests.  This
                    229: means that, once the execution of the tests is done with the compatibility
                    230: tools, you can still use the native Kyua binary to poke at the results
                    231: database.  More on this below.
                    232: 
                    233: ## Using the native Kyua command-line interface
                    234: 
                    235: The preferred way to run the NetBSD test suite with Kyua is to use the
                    236: native Kyua command-line binary.  This is the preferred method because it
                    237: trains you to use the new interface rather than relying on the old pipeline
                    238: and because it exposes you to all the new features of Kyua.  Regardless,
                    239: this and the previous approach will yield the same results for a particular
                    240: execution.
                    241: 
                    242: Using the native command-line interface is a multi-step process because
                    243: the existing NetBSD test suite is not prepared for Kyua.  Let's take a look
                    244: at these steps.
                    245: 
                    246: To get started, install the Kyua packages:
                    247: 
1.7       riz       248:     $ cd /usr/pkgsrc/devel/kyua-cli
1.1       jmmv      249:     $ make install && make clean
1.7       riz       250:     $ cd /usr/pkgsrc/devel/kyua-atf-compat
1.1       jmmv      251:     $ make install && make clean
                    252: 
                    253: Once this is done, configure Kyua in the same way ATF is configured "out of
                    254: the box" in NetBSD.  Create the /usr/pkg/etc/kyua/kyua.conf file with these
                    255: contents:
                    256: 
1.3       jmmv      257:     syntax('config', 1)
1.1       jmmv      258:     unprivileged_user = '_tests'
                    259: 
                    260: The next step is to populate /usr/tests with Kyuafiles, as Kyua is unable
                    261: to read existing Atffiles.  This is easy to do with the atf2kyua(1) tool
                    262: shipped in the kyua-atf-compat package:
                    263: 
                    264:     # atf2kyua /usr/tests
                    265: 
                    266: And that is it.  You can now execute the test suite using Kyua with any of
                    267: the following two forms:
                    268: 
                    269:     $ cd /usr/tests && kyua test
                    270:     $ kyua test -k /usr/tests/Kyuafile
                    271: 
                    272: Note that none of these will generate "pretty" reports.  These commands
                    273: will only record the results of the execution into the database.  In order
                    274: to generate reports, keep reading.
                    275: 
                    276: # Generating reports
                    277: 
                    278: Once you have ran the NetBSD test suite with any of the mechanisms above,
                    279: the results of the execution have been stored in the "Kyua store", which is
                    280: a database located in ~/.kyua/store.db by default.  (This path can be
                    281: changed at any time with the --store flag.)
                    282: 
                    283: To extract a report from the database using the results of the latest tests
                    284: run, you can run any of the following:
                    285: 
                    286:     $ kyua report -o my-report.txt
                    287:     $ kyua html-report -o /var/www/results/
                    288: 
                    289: # Support and feedback
                    290: 
                    291: The Kyua manual is available in the GNU Info format and can be accessed by
                    292: running:
                    293: 
                    294:     $ info kyua
                    295: 
                    296: Alternatively, use the help subcommand to get built-in documentation.  The
                    297: following invocation will print all the available subcommands:
                    298: 
                    299:     $ kyua help
                    300: 
                    301: And an invocation like this will show you all the possible options for a
                    302: given subcommand:
                    303: 
                    304:     $ kyua help report-html
                    305: 
                    306: If you have gone through the instructions above and started playing with
                    307: Kyua, please do not hesitate to report your experiences (either good or
                    308: bad) to [Julio Merino](mailto:jmmv@NetBSD.org)!  Any comments will be
                    309: highly appreciated and will be taken into account for the near future of
                    310: Kyua.

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb