Diff for /wikisrc/tutorials/atf.mdwn between versions 1.2 and 1.3

version 1.2, 2010/09/02 22:59:38 version 1.3, 2010/09/03 13:35:32
Line 67  Then, you could define the following tes Line 67  Then, you could define the following tes
 Try to keep your test case names as descriptive as possible so that they do not  Try to keep your test case names as descriptive as possible so that they do not
 require comments to explain what they intend to test.  require comments to explain what they intend to test.
   
   ## Test case parts
   
   A test case is composed by three parts: the *head*, the *body* and the
   *cleanup*.  Only the body is required; the other two routines are optional.
   
   ### The head
   
   The *head* is used **for the sole purpose** to define meta-data properties for
   the test case.  (Eventually, this would not be specified programmatically, but
   is how we deal with the information right now.)
   
   The following properties are commonly useful:
   
   * descr: A textual description of the purpose of the test case.
   
   * require.user: Set to 'root' to mark the test case as root-specific.  It is
     nice not to abuse this; see puffs and rump.
   
   * use.fs: Set to 'true' if the test case creates temporary files in the "current
     directory".  Otherwise the atf runtime will isolate the test case in such a
     way to forbid this, which will misteriously make your test to fail.
   
   ### The body
   
   The *body* is the actual meat of the test case.  This is just a regular function
   that executes any code you want and calls special atf functions to report
   failures; see below.
   
   In particular, be aware that the atf run-time **isolates** the execution of
   every test case to prevent side-effects (such as temporary file leftovers,
   in-memory data corruption, etc.).  In particular:
   
   * A test case is **always executed as a subprocess** that is separate from the
     head and the cleanup.
   
   * The current working directory of a test case is changed to a temporary
     location that gets cleaned up later on automatically.  (Set the use.fs
     property to true in the head if you need to write to this temporary
     directory.)
   
   * The environment of the test case is "sanitized" to get rid of variables that
     can cause side-effects; e.g. LC_ALL, TZ, etc.
   
 ## Installation of test programs: the why and the where  ## Installation of test programs: the why and the where
   
 Test programs get installed into the /usr/tests/ hierarchy.  The main reason for  Test programs get installed into the /usr/tests/ hierarchy.  The main reason for
Line 128  If the subdirectory does not exist: Line 171  If the subdirectory does not exist:
   
 ### Makefile template  ### Makefile template
   
     # $NetBSD: atf.mdwn,v 1.1 2010/09/01 20:59:24 jmmv Exp $      # $NetBSD: atf.mdwn,v 1.2 2010/09/02 22:59:38 jmmv Exp $
   
     .include <bsd.own.mk>      .include <bsd.own.mk>
   
Line 226  For example: Line 269  For example:
   
     .include <bsd.test.mk>      .include <bsd.test.mk>
   
   ### Common functions
   
   The following functions are commonly used from within a test case body:
   
   * ATF_CHECK(boolean_expression): Checks if the given boolean expression is true
     and, if not, records a failure for the test case but *execution continues*.
     Using ATF_REQUIRE aborts execution immediately after a failure.
   
   * ATF_CHECK_EQ(expected_expression, actual_expression): Checks if the two
     expressions match and, if not, records a failure.  Similarly, ATF_REQUIRE_EQ
     aborts immediately if the check fails.
   
   * ATF_CHECK_STREQ(expected_string, actual_string): Same as ATF_CHECK_EQ but
     performs string comparisons with strcmp.
   
   * atf_tc_skip(const char *format, ...): Marks the test case as skipped with the
     provided reason and exits.
   
   * atf_tc_fail(const char *format, ...): Marks the test case as failed with the
     provided reason and exits.
   
   * atf_tc_pass(void): Explicitly marks the test case as passed.  This is
     *implied* when the test case function ends, so you should not use this in
     general.
   
   * atf_expect_fail(const char *format, ...): Tells the atf runtime that the code
     following this call is expected to raise one or more failures (be it with
     atf_tc_fail, ATF_CHECK_*, etc.).  Use this to mark a block of code that is
     known to be broken (e.g. a test that reproduces a known bug).  Use the string
     parameter to provide an explanation about why the code is broken; if possible,
     provide a PR number.  Lastly, to terminate the "expected failure" code block
     and reset the runtime to the default functionality, use the atf_expect_pass()
     function.
   
   * atf_expect_death(const char *format, ...): Same as atf_expect_fail but expects
     an abrupt termination of the test case, be it due to a call to exit() or to
     the reception of a signal.
   
   * atf_expect_exit(int exitcode, const char *fomat, ...): Same as atf_expect_fail
     but expects the test case to exit with a specific exitcode.  Provide -1 to
     indicate any exit code.
   
   * atf_expect_signal(int signo, const char *fomat, ...): Same as atf_expect_fail
     but expects the test case to receive a specific signal.  Provide -1 to
     indicate any signal.
   
   * atf_expect_timeout(const char *reason, ...): Same as atf_expect_fail but
     expects the test case to get stuck and time out.
   
 ## Shell test programs  ## Shell test programs
   
 ### Template  ### Template
Line 288  test programs are processed with the atf Line 380  test programs are processed with the atf
 wrapper over /bin/sh that loads the shared atf code and then delegates execution  wrapper over /bin/sh that loads the shared atf code and then delegates execution
 to your source file.  to your source file.
   
   ### Common functions
   
   The following functions are commonly used from within a test case body:
   
   * atf_check: This is probably the most useful function for shell-based tests.
     It may need some experience to get it right, but it allows, in one line, to
     check the execution of a command.  Where check means: validate exit code,
     stdout and stderr.  This is just a wrapper over atf-check, so please refer to
     [atf-check(1)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-check++NetBSD-current)
     for more details.
   
   * atf_check_equal value1 value2: Check that the two values are equal and, if
     not, abort execution.
   
   * atf_expect_*: Same as their C counterparts; see above.
   
   * atf_fail reason: Explicitly marks the test case as failed and aborts it.
   
   * atf_skip reason: Explicitly marks the test case as skipped and exits.
   
   * atf_pass: Explicitly markts the test case as passed and exits.
   
   * atf_get_srcdir: Prints the path to the directory where the test case lives.
     Use as $(atf_get_srcdir)/my-static-data-file.
   
 ## FAQ  ## FAQ
   
 ### How do I atfify a plain test program?  ### How do I atfify a plain test program?
Line 385  Which can later be invoked as any of: Line 502  Which can later be invoked as any of:
     $ ./snprintf_test string_formatter      $ ./snprintf_test string_formatter
     $ atf-run snprintf_test | atf-report      $ atf-run snprintf_test | atf-report
   
   ### How do I write a test case for an unfixed PR?
   
   Use the "expectations" mechanism to define part of the test case as faulty,
   crashy, etc.  For example, suppose we have PR 1 that reports a condition in
   which snprintf() does the wrong formatting when using %s, and PR 2 that mentions
   that another snprintf() call using %d with number 5 causes a segfault.  We could
   do:
   
       #include <atf-c.h>
       #include <signal.h>
       #include <stdio.h>
   
       ATF_TC_WITHOUT_HEAD(tc, integer_formatter);
       ATF_TC_BODY(tc, integer_formatter)
       {
           char buf[1024];
   
           snprintf(buf, sizeof(buf), "Hello %d\n", 1);
           ATF_CHECK_STREQ("Hello 1", buf);
   
           atf_tc_expect_signal(SIGSEGV, "PR 2: %%d with 5 causes a crash");
           snprintf(buf, sizeof(buf), "Hello %d\n", 5);
           atf_tc_expect_pass();
           ATF_CHECK_STREQ("Hello 5", buf);
       }
   
       ATF_TC_WITHOUT_HEAD(tc, string_formatter);
       ATF_TC_BODY(tc, string_formatter)
       {
           char buf[1024];
   
           snprintf(buf, sizeof(buf), "Hello %s\n", "world!");
           atf_tc_expect_failure("PR 1: %%s does not work");
           ATF_CHECK_STREQ("Hello world!", buf);
           atf_tc_expect_pass();
       }
   
       ATF_TP_ADD_TCS(tp)
       {
           ATF_TP_ADD_TC(tp, integer_formatter);
           ATF_TP_ADD_TC(tp, string_formatter);
       }
   
 ### Do I need to remove temporary files?  ### Do I need to remove temporary files?
   
 No.  atf-run does this automatically for you, because it runs every test program  No.  atf-run does this automatically for you, because it runs every test program

Removed from v.1.2  
changed lines
  Added in v.1.3


CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb