Diff for /wikisrc/tutorials/atf.mdwn between versions 1.7 and 1.12

version 1.7, 2010/09/03 15:47:34 version 1.12, 2011/01/10 13:58:07
Line 1 Line 1
 [[!meta title="Creating atf-based tests for NetBSD src"]]  [[!meta title="Creating atf-based tests for NetBSD src"]]
 [[!toc ]]  [[!toc ]]
   
   # Introduction
   
 This quick tutorial provides a guideline on how to start creating new test  This quick tutorial provides a guideline on how to start creating new test
 programs and/or test cases, how these tests are tied to the NetBSD source tree  programs and/or test cases, how these tests are tied to the NetBSD source tree
 and includes a short reference of the most commonly used functions.  and includes a short reference of the most commonly used functions.
Line 69  require comments to explain what they in Line 71  require comments to explain what they in
   
 # Test case parts  # Test case parts
   
 A test case is composed by three parts: the *head*, the *body* and the  
 *cleanup*.  Only the body is required; the other two routines are optional, but  
 you are strongly encouraged to provide the *head*, if only to describe the  
 purpose of your tests by setting the *descr* property.  
   
 ## The head  ## The head
   
 The *head* is used **for the sole purpose** to define meta-data properties for  The *head* is used **for the sole purpose** to define meta-data properties for
Line 86  The following properties are commonly us Line 83  The following properties are commonly us
   
 * require.user: Set to 'root' to tell the atf runtime that this test requires  * require.user: Set to 'root' to tell the atf runtime that this test requires
   root privileges.  The test will later be skipped if you are running atf as    root privileges.  The test will later be skipped if you are running atf as
   non-root, and the test will be executed otherwise.    non-root, and the test will be executed otherwise.  Please do not use this
     unless absolutely necessary!  You can most likely make your tests run as a
     regular user if you use puffs and rump.
   
 * use.fs: Set to 'true' if the test case creates temporary files in the "current  * use.fs: Set to 'true' if the test case creates temporary files in the "current
   directory".  If set to false, the atf runtime will set the "current directory"    directory".  If set to false, the atf runtime will set the "current directory"
   to an unwritable directory, which will disallow the creation of the temporary    to an unwritable directory, which will disallow the creation of the temporary
   files and will make your test misteriously fail.    files and will make your test mysteriously fail.
   
 ## The body  ## The body
   
Line 104  every test case to prevent side-effects  Line 103  every test case to prevent side-effects 
 in-memory data corruption, etc.).  In particular:  in-memory data corruption, etc.).  In particular:
   
 * A test case is **always executed as a subprocess** that is separate from the  * A test case is **always executed as a subprocess** that is separate from the
   head and the cleanup.  This implies that you cannot pass any in-memory state    head.  This implies that you cannot pass any in-memory state between the
   between the three parts.  The only exception is that the body and the cleanup    parts.
   share the same temporary directory, so you can use the file system to pass  
   information between them.  
   
 * The current working directory of a test case is changed to a temporary  * The current working directory of a test case is changed to a temporary
   location that gets cleaned up later on automatically.  (Set the use.fs    location that gets cleaned up later on automatically.  (Set the use.fs
Line 172  to execute the tests of that particular  Line 169  to execute the tests of that particular 
 tests have been installed into the destdir.  tests have been installed into the destdir.
   
 Please note that this is only provided for convenience but it is completely  Please note that this is only provided for convenience but it is completely
 unsupported.  Tests run this way may fail misteriously, and that is perfectly  unsupported.  Tests run this way may fail mysteriously, and that is perfectly
 fine as long as they work from their canonical locations in /usr/tests.  fine as long as they work from their canonical locations in /usr/tests.
   
 # Adding a new test  # Adding a new test
Line 206  If the subdirectory does not exist: Line 203  If the subdirectory does not exist:
   
 1. Edit the parent Makefile to recurse into the new subdirectory.  1. Edit the parent Makefile to recurse into the new subdirectory.
   
 1. Edit src/etc/mtree/NetBSD.base.dist to register the new subdirectory.  Your  1. Edit src/etc/mtree/NetBSD.dist.tests to register the new subdirectory.  Your
    test will be installed under /usr/tests/.     test will be installed under /usr/tests/.
   
 1. Edit src/distrib/sets/lists/tests/mi to register the new test program.  Do  1. Edit src/distrib/sets/lists/tests/mi to register the new test program.  Do
Line 282  specific details as to how this works fo Line 279  specific details as to how this works fo
             atf_tc_fail("Condition not met!"); /* Abort execution. */              atf_tc_fail("Condition not met!"); /* Abort execution. */
     }      }
   
     ATF_TC_WITHOUT_HEAD(tc, another_test_case);      ATF_TC(tc, another_test_case);
       ATF_TC_HEAD(tc, another_test_case)
       {
           atf_tc_set_md_var(tc, "descr", "This test case ensures that...");
       }
     ATF_TC_BODY(tc, another_test_case)      ATF_TC_BODY(tc, another_test_case)
     {      {
         /* Do more tests here... */          /* Do more tests here... */
Line 296  specific details as to how this works fo Line 297  specific details as to how this works fo
   
 This program needs to be built with the Makefile shown below.  Once built, the  This program needs to be built with the Makefile shown below.  Once built, the
 program automatically gains a main() method that provides a consistent user  program automatically gains a main() method that provides a consistent user
 interface to all test programs.  You are simply not inteded to provide your own  interface to all test programs.  You are simply not intended to provide your own
 main method, nor to deal with the command-line of the invocation.  main method, nor to deal with the command-line of the invocation.
   
 ## How to build  ## How to build
Line 318  For example: Line 319  For example:
   
 The following functions are commonly used from within a test case body:  The following functions are commonly used from within a test case body:
   
 * ATF_CHECK(boolean_expression): Checks if the given boolean expression is true  * ATF_REQUIRE(boolean_expression): Checks if the given boolean expression is
   and, if not, records a failure for the test case but *execution continues*.    true and, if not, aborts execution and marks the test as failed.  Similarly
   Using ATF_REQUIRE aborts execution immediately after a failure.    ATF_CHECK performs the same test but does not abort execution: it records the
     failure but keeps processing the test case.  For an explanation on when to use
 * ATF_CHECK_EQ(expected_expression, actual_expression): Checks if the two    which, refer to the FAQ question below.
   expressions match and, if not, records a failure.  Similarly, ATF_REQUIRE_EQ  
   aborts immediately if the check fails.  * ATF_REQUIRE_EQ(expected_expression, actual_expression): Checks if the two
     expressions match and, if not, aborts marking the test as failed.  Similarly,
     ATF_CHECK_EQ records the error but does not abort execution.
   
 * ATF_CHECK_STREQ(expected_string, actual_string): Same as ATF_CHECK_EQ but  * ATF_REQUIRE_STREQ(expected_string, actual_string): Same as ATF_REQUIRE_EQ but
   performs string comparisons with strcmp.    performs string comparisons with strcmp.
   
 * atf_tc_skip(const char *format, ...): Marks the test case as skipped with the  * atf_tc_skip(const char *format, ...): Marks the test case as skipped with the
Line 341  The following functions are commonly use Line 344  The following functions are commonly use
   
 * atf_expect_fail(const char *format, ...): Tells the atf runtime that the code  * atf_expect_fail(const char *format, ...): Tells the atf runtime that the code
   following this call is expected to raise one or more failures (be it with    following this call is expected to raise one or more failures (be it with
   atf_tc_fail, ATF_CHECK_*, etc.).  Use this to mark a block of code that is    atf_tc_fail, ATF_REQUIRE_*, etc.).  Use this to mark a block of code that is
   known to be broken (e.g. a test that reproduces a known bug).  Use the string    known to be broken (e.g. a test that reproduces a known bug).  Use the string
   parameter to provide an explanation about why the code is broken; if possible,    parameter to provide an explanation about why the code is broken; if possible,
   provide a PR number.  Lastly, to terminate the "expected failure" code block    provide a PR number.  Lastly, to terminate the "expected failure" code block
Line 352  The following functions are commonly use Line 355  The following functions are commonly use
   an abrupt termination of the test case, be it due to a call to exit() or to    an abrupt termination of the test case, be it due to a call to exit() or to
   the reception of a signal.    the reception of a signal.
   
 * atf_expect_exit(int exitcode, const char *fomat, ...): Same as atf_expect_fail  * atf_expect_exit(int exitcode, const char *format, ...): Same as atf_expect_fail
   but expects the test case to exit with a specific exitcode.  Provide -1 to    but expects the test case to exit with a specific exitcode.  Provide -1 to
   indicate any exit code.    indicate any exit code.
   
 * atf_expect_signal(int signo, const char *fomat, ...): Same as atf_expect_fail  * atf_expect_signal(int signo, const char *format, ...): Same as atf_expect_fail
   but expects the test case to receive a specific signal.  Provide -1 to    but expects the test case to receive a specific signal.  Provide -1 to
   indicate any signal.    indicate any signal.
   
 * atf_expect_timeout(const char *reason, ...): Same as atf_expect_fail but  * atf_expect_timeout(const char *reason, ...): Same as atf_expect_fail but
   expects the test case to get stuck and time out.    expects the test case to get stuck and time out.
   
   * atf_tc_get_config_var("srcdir"): Returns the path to the directory containing
     the test program binary.  This must be used to locate any data/auxiliary files
     stored alongside the binary.
   
   * RL(integer_expression, integer): Used to evaluate a call to a libc function
     that updates errno when it returns an error and to provide correct error
     reporting.  The integer expression is the call to such function, and the
     literal integer provides the expected return value when there is an error.
     For example: RL(open("foo", O_RDONLY), -1).  This would fail the test case if
     open returns -1, and would record the correct error message returned by libc.
   
 # Shell test programs  # Shell test programs
   
 ## Template  ## Template
Line 396  The details on how this works are provid Line 410  The details on how this works are provid
     }      }
   
     atf_test_case another_test_case      atf_test_case another_test_case
       another_test_case_head() {
           atf_set "descr" "This test case ensures that..."
       }
     another_test_case_body() {      another_test_case_body() {
         # Do more tests...          # Do more tests...
     }      }
Line 407  The details on how this works are provid Line 424  The details on how this works are provid
   
 This program needs to be built with the Makefile shown below.  The program  This program needs to be built with the Makefile shown below.  The program
 automatically gains an entry point that provides a consistent user interface to  automatically gains an entry point that provides a consistent user interface to
 all test programs.  You are simply not inteded to provide your own "main  all test programs.  You are simply not intended to provide your own "main
 method", nor to deal with the command-line of the invocation.  method", nor to deal with the command-line of the invocation.
   
 ## How to build  ## How to build
Line 450  The following functions are commonly use Line 467  The following functions are commonly use
   
 * atf_skip reason: Explicitly marks the test case as skipped and exits.  * atf_skip reason: Explicitly marks the test case as skipped and exits.
   
 * atf_pass: Explicitly markts the test case as passed and exits.  * atf_pass: Explicitly marks the test case as passed and exits.
   
 * atf_get_srcdir: Prints the path to the directory where the test case lives.  * atf_get_srcdir: Prints the path to the directory where the test case lives.
   Use as $(atf_get_srcdir)/my-static-data-file.    Use as $(atf_get_srcdir)/my-static-data-file.
Line 502  keep this in mind: Line 519  keep this in mind:
   
 * Use the ATF_CHECK* and/or atf_tc_fail functions to do the comparisons and  * Use the ATF_CHECK* and/or atf_tc_fail functions to do the comparisons and
   report errors.  Neither errx nor any other error reporting and program    report errors.  Neither errx nor any other error reporting and program
   termination functions (read: err, errx, warn, warnx, exit, abor) are to be    termination functions (read: err, errx, warn, warnx, exit, abort) are to be
   used at all.    used at all.
   
 The result would look like:  The result would look like:
Line 559  Which can later be invoked as any of: Line 576  Which can later be invoked as any of:
 Use the "expectations" mechanism to define part of the test case as faulty,  Use the "expectations" mechanism to define part of the test case as faulty,
 crashy, etc.  This is for two reasons:  crashy, etc.  This is for two reasons:
   
 * As long as the bug still exists, he test case will be reported as an "expected  * As long as the bug still exists, the test case will be reported as an
   failure".  Such expected failures do not count towards the success or failure    "expected failure".  Such expected failures do not count towards the success
   of the whole test suite.    or failure of the whole test suite.
   
 * When the bug gets fixed, the bug will not trigger any more in the test case,  * When the bug gets fixed, the bug will not trigger any more in the test case,
   and thus the expectation of failure will not be met any more.  At this point    and thus the expectation of failure will not be met any more.  At this point
   the test case will start raising a regular failure, which is usually addressed    the test case will start raising a regular failure, which is usually addressed
   by either just removing the expect_* calls.    by just removing the expect_* calls (but add a comment with the PR number!).
   
 For example, suppose we have PR lib/1 that reports a condition in which  For example, suppose we have PR lib/1 that reports a condition in which
 snprintf() does the wrong formatting when using %s, and PR lib/2 that mentions  snprintf() does the wrong formatting when using %s, and PR lib/2 that mentions
Line 577  do: Line 594  do:
     #include <signal.h>      #include <signal.h>
     #include <stdio.h>      #include <stdio.h>
   
     ATF_TC_WITHOUT_HEAD(tc, integer_formatter);      ATF_TC(tc, integer_formatter);
       ATF_TC_HEAD(tc, integer_formatter)
       {
           atf_tc_set_md_var(tc, "descr", "Tests the %d formatter for snprintf");
       }
     ATF_TC_BODY(tc, integer_formatter)      ATF_TC_BODY(tc, integer_formatter)
     {      {
         char buf[1024];          char buf[1024];
Line 591  do: Line 612  do:
         ATF_CHECK_STREQ("Hello 5", buf);          ATF_CHECK_STREQ("Hello 5", buf);
     }      }
   
     ATF_TC_WITHOUT_HEAD(tc, string_formatter);      ATF_TC(tc, string_formatter);
       ATF_TC_HEAD(tc, string_formatter)
       {
           atf_tc_set_md_var(tc, "descr", "Tests the %s formatter for snprintf");
       }
     ATF_TC_BODY(tc, string_formatter)      ATF_TC_BODY(tc, string_formatter)
     {      {
         char buf[1024];          char buf[1024];

Removed from v.1.7  
changed lines
  Added in v.1.12


CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb