File:  [NetBSD Developer Wiki] / wikisrc / kyua.mdwn
Revision 1.8: download - view: text, annotated - select for diffs
Fri Feb 1 22:20:02 2013 UTC (7 years ago) by jmmv
Branches: MAIN
CVS tags: HEAD
Update the description of Kyua and the import plan according to the recent
changes regarding the move of the test program execution code to a new
kyua-testers package written in C.

    1: [[!meta title="Kyua: An introduction for NetBSD users"]]
    2: [[!toc levels=2]]
    3: 
    4: The [Automated Testing Framework](ATF), or ATF for short, is a software
    5: package composed of two parts: the *ATF libraries* and the *ATF tools*.
    6: The ATF libraries provide a toolkit for developers to implement test cases
    7: in a variety of languages: C, C++ and POSIX shell.  The ATF tools provide
    8: the utilities to run such test cases in an automated way and to generate
    9: reports.
   10: 
   11: The ATF tools have some
   12: [design and, particularly, implementation problems](http://mail-index.netbsd.org/atf-devel/2010/11/13/msg000206.html)
   13: that make it hard to add support for highly desired features such as
   14: parallel execution of test cases, unified dashboards covering multiple test
   15: runs ([like this one](http://releng.netbsd.org/test-results.html)), the
   16: ability to run legacy or third-party test programs that do not use the ATF
   17: libraries, and the ability to tune the timeout of test cases.
   18: 
   19: *Kyua's current goal is to reimplement _only_ the ATF tools* while
   20: maintaining backwards compatibility with the tests written with the ATF
   21: libraries (i.e. with the NetBSD test suite).
   22: 
   23: Because Kyua is a replacement of some ATF components, the end goal is to
   24: integrate Kyua into the NetBSD base system (just as ATF is) and remove the
   25: deprecated ATF components.  Removing the deprecated components will allow
   26: us to make the above-mentioned improvements to Kyua, as well as many
   27: others, without having to deal with the obsolete ATF code base.
   28: *Discussing how and when this transition might happen is out of the scope
   29: of this document at the moment.*
   30: 
   31: This page provides instructions on how to use Kyua with the current NetBSD
   32: test suite so that you can experiment with the tool, familiarize yourself
   33: with it and provide feedback early on.
   34: 
   35: **If you would like more details on how the import of Kyua into NetBSD will
   36:   happen, please see [[Kyua: The way into NetBSD|/kyua/import]].**
   37: 
   38: # What's in the name?
   39: 
   40: You should really think of Kyua as ATF 2.x.  Then, why isn't it just ATF
   41: 2.x?
   42: 
   43: To be honest, I never liked the ATF name: it was picked for me as part of
   44: the Google Summer of Code 2007 program and I did not think about changing
   45: it at that time.  A year later, I learned that the ATF acronym is severely
   46: overloaded, which makes it hard to find the project on popular search
   47: engines, and has unpopular connotations in specific countries.
   48: 
   49: So, as part of the rewrite, I decided to choose a new name: a name that is
   50: not an acronym and thus can be easily pronounced, and a name that is quite
   51: unique in search results.  The name is Kyua, which is just a play on the
   52: pronounciation of the Q.A. acronym.  Originally, my intention was to
   53: pronounce Kyua as Q.A., but in reality this never happened.  Today, just
   54: read the name as your instinct would: "Kyu-ah".
   55: 
   56: # Why is Kyua a third-party project?
   57: 
   58: Kyua's main consumer is NetBSD.  One could argue that Kyua should be
   59: developed within NetBSD and maintained in the NetBSD source tree.  However,
   60: there is nothing in the Kyua project that inherently depends on NetBSD, and
   61: maintaining it as a third-party package is a way to keep the developers
   62: honest regarding portability.
   63: 
   64: Ideally, other projects (such as FreeBSD) would make use of Kyua too for
   65: their testing needs, and if that happened we would be able to share tests
   66: with them pretty easily.  Forcing a portable codebase in the upstream
   67: repository helps in this.  (Be aware that different individuals from
   68: FreeBSD and Minix have shown interest in adopting Kyua for their respective
   69: systems!)
   70: 
   71: # Main differences (aka "what to expect")
   72: 
   73: As of version 0.5, Kyua has (or is supposed to have) feature parity with
   74: the ATF tools.  That said, having feature parity does not imply that they
   75: are the same.  This section outlines a few of the differences that you
   76: should be aware of before continuing.
   77: 
   78: ## Results database
   79: 
   80: Kyua collects the results of the execution of a test suite into an SQLite
   81: database.  User-friendly reports are later generated by extracting data
   82: from this same database.
   83: 
   84: In ATF, the results of the execution were written to an internal format
   85: that only atf-report could understand.  Despite of the database, Kyua still
   86: maintains the separation of "tests execution" from "report generation".
   87: 
   88: The contents of the database are immutable and incremental.  This means
   89: that, in the future, the Kyua tools will be able to provide historical data
   90: for particular test cases, or for whole test runs (which is what other
   91: NetBSD developers have ended up implementing multiple times outside of ATF
   92: because the framework did not provide such functionality by itself).
   93: 
   94: ## Support for multiple test interfaces
   95: 
   96: Kyua has support for different "test interfaces", which means that Kyua can
   97: execute test programs written using different paradigms and collect their
   98: results into a single report.  At the moment, two interfaces are supported:
   99: 
  100: * The "atf" interface provides compatibility with those test programs that
  101:   use the ATF libraries.  This is the only interface currently used by the
  102:   NetBSD test suite, as there is no way to run any other test program in an
  103:   automated manner.
  104: 
  105: * The "plain" interface permits the execution of legacy test programs that
  106:   do not use any testing library.  Such test programs are those that just
  107:   return 0 or non-0 to indicate the success or failure of the test
  108:   (respectively).  This feature will allow the NetBSD test suite to
  109:   transparently execute third-party test suites (such as the IPF or GCC
  110:   test suites) without having to implement ATF-based wrappers.  It will
  111:   also lower the barrier of entry to writing test programs for NetBSD, as
  112:   using the ATF libraries will become optional.
  113: 
  114: These interfaces are implemented as independent binaries, called testers,
  115: that are fully scriptable.  The testers are provided in the `kyua-testers`
  116: package.  The idea behind having these as independent programs is to
  117: restrict the OS-specific code to a small subset of Kyua written in C, and
  118: thus to allow the higher-level layers to be written in other languages
  119: (possibly Lua).
  120: 
  121: ## Lua configuration files
  122: 
  123: Kyua has two kind of configuration files: the Kyuafiles, which are the
  124: files shipped with a test suite that describe what test programs need to be
  125: run; and the user configuration files, which specify the run-time settings
  126: of Kyua and the test suites.  ATF had this same split of configuration
  127: files, and they were written in a custom language, with a custom parser.
  128: 
  129: The Kyua configuration files are all Lua scripts.  The major advantage of
  130: this at the moment is that their syntax will be familiar to end users, and
  131: that the parser for these files is well-tested.  In the future, the use of
  132: Lua will allow the implementation of more-intelligent test (and maybe even
  133: build) scripts.
  134: 
  135: ## Direct HTML output
  136: 
  137: All of the NetBSD continous build and testing systems provide status
  138: reports through the releng web interface.  In the case of ATF, this has
  139: traditionally been tricky because ATF cannot generate HTML contents
  140: directly; instead, `atf-report` generates XML output which later must be
  141: postprocessed with `xsltproc` to create the HTML pages.
  142: 
  143: Kyua has the ability to generate HTML reports straight from the tool,
  144: without having to go through any XML toolchain.  This means that NetBSD,
  145: out of the box, can generate such reports and publish them with the builtin
  146: httpd(8) server.
  147: 
  148: ## Heavier code base
  149: 
  150: If you take a look at the Kyua distribution file, you may notice that it is
  151: about the same size as the distribution file of ATF, yet Kyua does not
  152: currently replace the ATF libraries.  This may be surprising because it
  153: seems to imply that the codebase of Kyua is bigger because it "just"
  154: reimplements atf-run and atf-report: i.e. by just reimplementing parts of
  155: ATF, it is already as big as the whole of ATF.
  156: 
  157: This is true, for two reasons.
  158: 
  159: The first is that Kyua is more featureful and flexible: the features
  160: outlined above have a cost in terms of implementation, and the codebase of
  161: Kyua is more carefully crafted to allow for later growth.  In particular,
  162: all OS-specific details have been abstracted for easier portability, and
  163: the SQLite and Lua libraries have been wrapped for safety.
  164: 
  165: The second is that Kyua is much better tested (which is very important for
  166: a software package that you will rely on to validate your own software!).
  167: To give you some numbers, ATF 0.16 contains around 400 test cases for both
  168: atf-run and atf-report while Kyua 0.5 contains around 1100 test cases.
  169: 
  170: # Components
  171: 
  172: Kyua, as a project, is made up of a variety of components (which *include*
  173: ATF, because the ATF libraries are *not* being rewritten).  All of these
  174: components exist in pkgsrc, and are:
  175: 
  176: * pkgsrc/devel/atf-libs: The C, C++ and POSIX shell libraries provided by
  177:   ATF.  These are *NOT* meant to be replaced by Kyua.
  178: 
  179: * pkgsrc/devel/atf: The ATF tools, namely atf-run and atf-report.  These
  180:   are deprecated and this package should eventually disappear.
  181: 
  182: * pkgsrc/devel/kyua-testers: The Kyua testers, which provide the
  183:   `kyua-atf-tester` and `kyua-plain-tester` helper binaries.  These
  184:   binaries implement the logic to execute test cases in an isolated manner
  185:   and to expose the test programs using a common and abstract command-line
  186:   interface.
  187: 
  188: * pkgsrc/devel/kyua-cli: The Kyua command-line interface, which provides a
  189:   superset of the functionality of atf-run and atf-report.
  190: 
  191: * pkgsrc/devel/kyua-atf-compat: Drop-in replacements for atf-run and
  192:   atf-report that use kyua-cli in the backend.
  193: 
  194: # Running the NetBSD test suite
  195: 
  196: There are two ways to run the NetBSD test suite with Kyua.  The easy (or
  197: trivial) way is to use the backwards compatibility ATF tools, and the more
  198: sophisticated way is to convert the test suite to Kyua and use the native
  199: Kyua binary.  This section explains both approaches.
  200: 
  201: ## Using the ATF compatibility tools
  202: 
  203: The easiest (but also the least "future-proof") way to run the NetBSD test
  204: suite with Kyua is to use the backwards compatibility ATF tools provided by
  205: the kyua-atf-compat module.  First of all, install the package:
  206: 
  207:     $ cd /usr/pkgsrc/devel/kyua-atf-compat
  208:     $ make install && make clean
  209: 
  210: And then, running the test suite is as easy as:
  211: 
  212:     $ cd /usr/tests
  213:     $ /usr/pkg/bin/atf-run | /usr/pkg/bin/atf-report
  214: 
  215: Please be aware that if the atf-run and atf-report tools provided by
  216: kyua-atf-compat appear in your PATH before the real atf-run and atf-report
  217: tools shipped by NetBSD, you will experience test failures for all the
  218: tests in /usr/tests/atf/atf-run and /usr/tests/atf/atf-report.  This is
  219: expected: while the compatibility tools behave similarly to the real tools
  220: from a user's perspective, they are not fully interchangeable.  (For
  221: example, the serialization format between atf-run and atf-report is
  222: different.)
  223: 
  224: One property of the atf-run wrapper is that it uses the default results
  225: database in ~/.kyua/store.db to record the execution of the tests.  This
  226: means that, once the execution of the tests is done with the compatibility
  227: tools, you can still use the native Kyua binary to poke at the results
  228: database.  More on this below.
  229: 
  230: ## Using the native Kyua command-line interface
  231: 
  232: The preferred way to run the NetBSD test suite with Kyua is to use the
  233: native Kyua command-line binary.  This is the preferred method because it
  234: trains you to use the new interface rather than relying on the old pipeline
  235: and because it exposes you to all the new features of Kyua.  Regardless,
  236: this and the previous approach will yield the same results for a particular
  237: execution.
  238: 
  239: Using the native command-line interface is a multi-step process because
  240: the existing NetBSD test suite is not prepared for Kyua.  Let's take a look
  241: at these steps.
  242: 
  243: To get started, install the Kyua packages:
  244: 
  245:     $ cd /usr/pkgsrc/devel/kyua-cli
  246:     $ make install && make clean
  247:     $ cd /usr/pkgsrc/devel/kyua-atf-compat
  248:     $ make install && make clean
  249: 
  250: Once this is done, configure Kyua in the same way ATF is configured "out of
  251: the box" in NetBSD.  Create the /usr/pkg/etc/kyua/kyua.conf file with these
  252: contents:
  253: 
  254:     syntax('config', 1)
  255:     unprivileged_user = '_tests'
  256: 
  257: The next step is to populate /usr/tests with Kyuafiles, as Kyua is unable
  258: to read existing Atffiles.  This is easy to do with the atf2kyua(1) tool
  259: shipped in the kyua-atf-compat package:
  260: 
  261:     # atf2kyua /usr/tests
  262: 
  263: And that is it.  You can now execute the test suite using Kyua with any of
  264: the following two forms:
  265: 
  266:     $ cd /usr/tests && kyua test
  267:     $ kyua test -k /usr/tests/Kyuafile
  268: 
  269: Note that none of these will generate "pretty" reports.  These commands
  270: will only record the results of the execution into the database.  In order
  271: to generate reports, keep reading.
  272: 
  273: # Generating reports
  274: 
  275: Once you have ran the NetBSD test suite with any of the mechanisms above,
  276: the results of the execution have been stored in the "Kyua store", which is
  277: a database located in ~/.kyua/store.db by default.  (This path can be
  278: changed at any time with the --store flag.)
  279: 
  280: To extract a report from the database using the results of the latest tests
  281: run, you can run any of the following:
  282: 
  283:     $ kyua report -o my-report.txt
  284:     $ kyua html-report -o /var/www/results/
  285: 
  286: # Support and feedback
  287: 
  288: The Kyua manual is available in the GNU Info format and can be accessed by
  289: running:
  290: 
  291:     $ info kyua
  292: 
  293: Alternatively, use the help subcommand to get built-in documentation.  The
  294: following invocation will print all the available subcommands:
  295: 
  296:     $ kyua help
  297: 
  298: And an invocation like this will show you all the possible options for a
  299: given subcommand:
  300: 
  301:     $ kyua help report-html
  302: 
  303: If you have gone through the instructions above and started playing with
  304: Kyua, please do not hesitate to report your experiences (either good or
  305: bad) to [Julio Merino](mailto:jmmv@NetBSD.org)!  Any comments will be
  306: highly appreciated and will be taken into account for the near future of
  307: Kyua.

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb