Annotation of wikisrc/kyua.mdwn, revision 1.1

1.1     ! jmmv        1: [[!meta title="Kyua: An introduction for NetBSD users"]]
        !             2: [[!toc ]]
        !             3: 
        !             4: The [Automated Testing Framework](ATF), or ATF for short, is a software
        !             5: package composed of two parts: the *ATF libraries* and the *ATF tools*.
        !             6: The ATF libraries provide a toolkit for developers to implement test cases
        !             7: in a variety of languages: C, C++ and POSIX shell.  The ATF tools provide
        !             8: the utilities to run such test cases in an automated way and to generate
        !             9: reports.
        !            10: 
        !            11: The ATF tools have some
        !            12: [design and, particularly, implementation problems](http://mail-index.netbsd.org/atf-devel/2010/11/13/msg000206.html)
        !            13: that make it hard to add support for highly desired features such as
        !            14: parallel execution of test cases, unified dashboards covering multiple test
        !            15: runs ([like this one](http://releng.netbsd.org/test-results.html)), the
        !            16: ability to run legacy or third-party test programs that do not use the ATF
        !            17: libraries, and the ability to tune the timeout of test cases.
        !            18: 
        !            19: *Kyua's current goal is to reimplement _only_ the ATF tools* while
        !            20: maintaining backwards compatibility with the tests written with the ATF
        !            21: libraries (i.e. with the NetBSD test suite).
        !            22: 
        !            23: Because Kyua is a replacement of some ATF components, the end goal is to
        !            24: integrate Kyua into the NetBSD base system (just as ATF is) and remove the
        !            25: deprecated ATF components.  Removing the deprecated components will allow
        !            26: us to make the above-mentioned improvements to Kyua, as well as many
        !            27: others, without having to deal with the obsolete ATF code base.
        !            28: *Discussing how and when this transition might happen is out of the scope
        !            29: of this document at the moment.*
        !            30: 
        !            31: This page provides instructions on how to use Kyua with the current NetBSD
        !            32: test suite so that you can experiment with the tool, familiarize yourself
        !            33: with it and provide feedback early on.
        !            34: 
        !            35: # Main differences (aka "what to expect")
        !            36: 
        !            37: As of version 0.5, Kyua has (or is supposed to have) feature parity with
        !            38: the ATF tools.  That said, having feature parity does not imply that they
        !            39: are the same.  This section outlines a few of the differences that you
        !            40: should be aware of before continuing.
        !            41: 
        !            42: ## Results database
        !            43: 
        !            44: Kyua collects the results of the execution of a test suite into an SQLite
        !            45: database.  User-friendly reports are later generated by extracting data
        !            46: from this same database.
        !            47: 
        !            48: In ATF, the results of the execution were written to an internal format
        !            49: that only atf-report could understand.  Despite of the database, Kyua still
        !            50: maintains the separation of "tests execution" from "report generation".
        !            51: 
        !            52: The contents of the database are immutable and incremental.  This means
        !            53: that, in the future, the Kyua tools will be able to provide historical data
        !            54: for particular test cases, or for whole test runs (which is what other
        !            55: NetBSD developers have ended up implementing multiple times outside of ATF
        !            56: because the framework did not provide such functionality by itself).
        !            57: 
        !            58: ## Support for multiple test interfaces
        !            59: 
        !            60: Kyua has support for different "test interfaces", which means that Kyua can
        !            61: execute test programs written using different paradigms and collect their
        !            62: results into a single report.  At the moment, two interfaces are supported:
        !            63: 
        !            64: * The "atf" interface provides compatibility with those test programs that
        !            65:   use the ATF libraries.  This is the only interface currently used by the
        !            66:   NetBSD test suite, as there is no way to run any other test program in an
        !            67:   automated manner.
        !            68: 
        !            69: * The "plain" interface permits the execution of legacy test programs that
        !            70:   do not use any testing library.  Such test programs are those that just
        !            71:   return 0 or non-0 to indicate the success or failure of the test
        !            72:   (respectively).  This feature will allow the NetBSD test suite to
        !            73:   transparently execute third-party test suites (such as the IPF or GCC
        !            74:   test suites) without having to implement ATF-based wrappers.  It will
        !            75:   also lower the barrier of entry to writing test programs for NetBSD, as
        !            76:   using the ATF libraries will become optional.
        !            77: 
        !            78: ## Lua configuration files
        !            79: 
        !            80: Kyua has two kind of configuration files: the Kyuafiles, which are the
        !            81: files shipped with a test suite that describe what test programs need to be
        !            82: run; and the user configuration files, which specify the run-time settings
        !            83: of Kyua and the test suites.  ATF had this same split of configuration
        !            84: files, and they were written in a custom language, with a custom parser.
        !            85: 
        !            86: The Kyua configuration files are all Lua scripts.  The major advantage of
        !            87: this at the moment is that their syntax will be familiar to end users, and
        !            88: that the parser for these files is well-tested.  In the future, the use of
        !            89: Lua will allow the implementation of more-intelligent test (and maybe even
        !            90: build) scripts.
        !            91: 
        !            92: ## Heavier code base
        !            93: 
        !            94: If you take a look at the Kyua distribution file, you may notice that it is
        !            95: about the same size as the distribution file of ATF, yet Kyua does not
        !            96: currently replace the ATF libraries.  This may be surprising because it
        !            97: seems to imply that the codebase of Kyua is bigger because it "just"
        !            98: reimplements atf-run and atf-report: i.e. by just reimplementing parts of
        !            99: ATF, it is already as big as the whole of ATF.
        !           100: 
        !           101: This is true, for two reasons.
        !           102: 
        !           103: The first is that Kyua is more featureful and flexible: the features
        !           104: outlined above have a cost in terms of implementation, and the codebase of
        !           105: Kyua is more carefully crafted to allow for later growth.  In particular,
        !           106: all OS-specific details have been abstracted for easier portability, and
        !           107: the SQLite and Lua libraries have been wrapped for safety.
        !           108: 
        !           109: The second is that Kyua is much better tested (which is very important for
        !           110: a software package that you will rely on to validate your own software!).
        !           111: To give you some numbers, ATF 0.16 contains around 400 test cases for both
        !           112: atf-run and atf-report while Kyua 0.5 contains around 1100 test cases.
        !           113: 
        !           114: # Components
        !           115: 
        !           116: Kyua, as a project, is made up of a variety of components (which *include*
        !           117: ATF, because the ATF libraries are *not* being rewritten).  All of these
        !           118: components exist in pkgsrc, and are:
        !           119: 
        !           120: * pkgsrc/devel/atf-libs: The C, C++ and POSIX shell libraries provided by
        !           121:   ATF.  These are *NOT* meant to be replaced by Kyua.
        !           122: 
        !           123: * pkgsrc/devel/atf: The ATF tools, namely atf-run and atf-report.  These
        !           124:   are deprecated and this package should eventually disappear.
        !           125: 
        !           126: * pkgsrc/devel/kyua-cli: The Kyua command-line interface, which provides a
        !           127:   superset of the functionality of atf-run and atf-report.
        !           128: 
        !           129: * pkgsrc/devel/kyua-atf-compat: Drop-in replacements for atf-run and
        !           130:   atf-report that use kyua-cli in the backend.
        !           131: 
        !           132: # Running the NetBSD test suite
        !           133: 
        !           134: There are two ways to run the NetBSD test suite with Kyua.  The easy (or
        !           135: trivial) way is to use the backwards compatibility ATF tools, and the more
        !           136: sophisticated way is to convert the test suite to Kyua and use the native
        !           137: Kyua binary.  This section explains both approaches.
        !           138: 
        !           139: ## Using the ATF compatibility tools
        !           140: 
        !           141: The easiest (but also the least "future-proof") way to run the NetBSD test
        !           142: suite with Kyua is to use the backwards compatibility ATF tools provided by
        !           143: the kyua-atf-compat module.  First of all, install the package:
        !           144: 
        !           145:     $ cd /usr/pkgsrc/deve/kyua-atf-compat
        !           146:     $ make install && make clean
        !           147: 
        !           148: And then, running the test suite is as easy as:
        !           149: 
        !           150:     $ cd /usr/tests
        !           151:     $ /usr/pkg/bin/atf-run | /usr/pkg/bin/atf-report
        !           152: 
        !           153: Please be aware that if the atf-run and atf-report tools provided by
        !           154: kyua-atf-compat appear in your PATH before the real atf-run and atf-report
        !           155: tools shipped by NetBSD, you will experience test failures for all the
        !           156: tests in /usr/tests/atf/atf-run and /usr/tests/atf/atf-report.  This is
        !           157: expected: while the compatibility tools behave similarly to the real tools
        !           158: from a user's perspective, they are not fully interchangeable.  (For
        !           159: example, the serialization format between atf-run and atf-report is
        !           160: different.)
        !           161: 
        !           162: One property of the atf-run wrapper is that it uses the default results
        !           163: database in ~/.kyua/store.db to record the execution of the tests.  This
        !           164: means that, once the execution of the tests is done with the compatibility
        !           165: tools, you can still use the native Kyua binary to poke at the results
        !           166: database.  More on this below.
        !           167: 
        !           168: ## Using the native Kyua command-line interface
        !           169: 
        !           170: The preferred way to run the NetBSD test suite with Kyua is to use the
        !           171: native Kyua command-line binary.  This is the preferred method because it
        !           172: trains you to use the new interface rather than relying on the old pipeline
        !           173: and because it exposes you to all the new features of Kyua.  Regardless,
        !           174: this and the previous approach will yield the same results for a particular
        !           175: execution.
        !           176: 
        !           177: Using the native command-line interface is a multi-step process because
        !           178: the existing NetBSD test suite is not prepared for Kyua.  Let's take a look
        !           179: at these steps.
        !           180: 
        !           181: To get started, install the Kyua packages:
        !           182: 
        !           183:     $ cd /usr/pkgsrc/deve/kyua-cli
        !           184:     $ make install && make clean
        !           185:     $ cd /usr/pkgsrc/deve/kyua-atf-compat
        !           186:     $ make install && make clean
        !           187: 
        !           188: Once this is done, configure Kyua in the same way ATF is configured "out of
        !           189: the box" in NetBSD.  Create the /usr/pkg/etc/kyua/kyua.conf file with these
        !           190: contents:
        !           191: 
        !           192:     syntax('kyuafile', 1)
        !           193:     unprivileged_user = '_tests'
        !           194: 
        !           195: The next step is to populate /usr/tests with Kyuafiles, as Kyua is unable
        !           196: to read existing Atffiles.  This is easy to do with the atf2kyua(1) tool
        !           197: shipped in the kyua-atf-compat package:
        !           198: 
        !           199:     # atf2kyua /usr/tests
        !           200: 
        !           201: And that is it.  You can now execute the test suite using Kyua with any of
        !           202: the following two forms:
        !           203: 
        !           204:     $ cd /usr/tests && kyua test
        !           205:     $ kyua test -k /usr/tests/Kyuafile
        !           206: 
        !           207: Note that none of these will generate "pretty" reports.  These commands
        !           208: will only record the results of the execution into the database.  In order
        !           209: to generate reports, keep reading.
        !           210: 
        !           211: # Generating reports
        !           212: 
        !           213: Once you have ran the NetBSD test suite with any of the mechanisms above,
        !           214: the results of the execution have been stored in the "Kyua store", which is
        !           215: a database located in ~/.kyua/store.db by default.  (This path can be
        !           216: changed at any time with the --store flag.)
        !           217: 
        !           218: To extract a report from the database using the results of the latest tests
        !           219: run, you can run any of the following:
        !           220: 
        !           221:     $ kyua report -o my-report.txt
        !           222:     $ kyua html-report -o /var/www/results/
        !           223: 
        !           224: # Support and feedback
        !           225: 
        !           226: The Kyua manual is available in the GNU Info format and can be accessed by
        !           227: running:
        !           228: 
        !           229:     $ info kyua
        !           230: 
        !           231: Alternatively, use the help subcommand to get built-in documentation.  The
        !           232: following invocation will print all the available subcommands:
        !           233: 
        !           234:     $ kyua help
        !           235: 
        !           236: And an invocation like this will show you all the possible options for a
        !           237: given subcommand:
        !           238: 
        !           239:     $ kyua help report-html
        !           240: 
        !           241: If you have gone through the instructions above and started playing with
        !           242: Kyua, please do not hesitate to report your experiences (either good or
        !           243: bad) to [Julio Merino](mailto:jmmv@NetBSD.org)!  Any comments will be
        !           244: highly appreciated and will be taken into account for the near future of
        !           245: Kyua.

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb