Diff for /wikisrc/tutorials/atf.mdwn between versions 1.9 and 1.10

version 1.9, 2010/09/03 16:32:26 version 1.10, 2010/09/03 21:29:36
Line 1 Line 1
 [[!meta title="Creating atf-based tests for NetBSD src"]]  [[!meta title="Creating atf-based tests for NetBSD src"]]
 [[!toc ]]  [[!toc ]]
   
   # Introduction
   
 This quick tutorial provides a guideline on how to start creating new test  This quick tutorial provides a guideline on how to start creating new test
 programs and/or test cases, how these tests are tied to the NetBSD source tree  programs and/or test cases, how these tests are tied to the NetBSD source tree
 and includes a short reference of the most commonly used functions.  and includes a short reference of the most commonly used functions.
Line 81  The following properties are commonly us Line 83  The following properties are commonly us
   
 * require.user: Set to 'root' to tell the atf runtime that this test requires  * require.user: Set to 'root' to tell the atf runtime that this test requires
   root privileges.  The test will later be skipped if you are running atf as    root privileges.  The test will later be skipped if you are running atf as
   non-root, and the test will be executed otherwise.    non-root, and the test will be executed otherwise.  Please do not use this
     unless absolutely necessary!  You can most likely make your tests run as a
     regular user if you use puffs and rump.
   
 * use.fs: Set to 'true' if the test case creates temporary files in the "current  * use.fs: Set to 'true' if the test case creates temporary files in the "current
   directory".  If set to false, the atf runtime will set the "current directory"    directory".  If set to false, the atf runtime will set the "current directory"
   to an unwritable directory, which will disallow the creation of the temporary    to an unwritable directory, which will disallow the creation of the temporary
   files and will make your test misteriously fail.    files and will make your test mysteriously fail.
   
 ## The body  ## The body
   
Line 165  to execute the tests of that particular  Line 169  to execute the tests of that particular 
 tests have been installed into the destdir.  tests have been installed into the destdir.
   
 Please note that this is only provided for convenience but it is completely  Please note that this is only provided for convenience but it is completely
 unsupported.  Tests run this way may fail misteriously, and that is perfectly  unsupported.  Tests run this way may fail mysteriously, and that is perfectly
 fine as long as they work from their canonical locations in /usr/tests.  fine as long as they work from their canonical locations in /usr/tests.
   
 # Adding a new test  # Adding a new test
Line 315  For example: Line 319  For example:
   
 The following functions are commonly used from within a test case body:  The following functions are commonly used from within a test case body:
   
 * ATF_CHECK(boolean_expression): Checks if the given boolean expression is true  * ATF_REQUIRE(boolean_expression): Checks if the given boolean expression is
   and, if not, records a failure for the test case but *execution continues*.    true and, if not, aborts exectuion and markts the test as failed.  Similarly
   Using ATF_REQUIRE aborts execution immediately after a failure.    ATF_CHECK performs the same test but does not abort execution: it records the
     failure but keeps processing the test case.  For an explanation on when to use
 * ATF_CHECK_EQ(expected_expression, actual_expression): Checks if the two    which, refer to the FAQ question below.
   expressions match and, if not, records a failure.  Similarly, ATF_REQUIRE_EQ  
   aborts immediately if the check fails.  * ATF_REQUIRE_EQ(expected_expression, actual_expression): Checks if the two
     expressions match and, if not, aborts marking the test as failed.  Similarly,
     ATF_CHECK_EQ records the error but does not abort execution.
   
 * ATF_CHECK_STREQ(expected_string, actual_string): Same as ATF_CHECK_EQ but  * ATF_REQUIRE_STREQ(expected_string, actual_string): Same as ATF_REQUIRE_EQ but
   performs string comparisons with strcmp.    performs string comparisons with strcmp.
   
 * atf_tc_skip(const char *format, ...): Marks the test case as skipped with the  * atf_tc_skip(const char *format, ...): Marks the test case as skipped with the
Line 338  The following functions are commonly use Line 344  The following functions are commonly use
   
 * atf_expect_fail(const char *format, ...): Tells the atf runtime that the code  * atf_expect_fail(const char *format, ...): Tells the atf runtime that the code
   following this call is expected to raise one or more failures (be it with    following this call is expected to raise one or more failures (be it with
   atf_tc_fail, ATF_CHECK_*, etc.).  Use this to mark a block of code that is    atf_tc_fail, ATF_REQUIRE_*, etc.).  Use this to mark a block of code that is
   known to be broken (e.g. a test that reproduces a known bug).  Use the string    known to be broken (e.g. a test that reproduces a known bug).  Use the string
   parameter to provide an explanation about why the code is broken; if possible,    parameter to provide an explanation about why the code is broken; if possible,
   provide a PR number.  Lastly, to terminate the "expected failure" code block    provide a PR number.  Lastly, to terminate the "expected failure" code block
Line 364  The following functions are commonly use Line 370  The following functions are commonly use
   the test program binary.  This must be used to locate any data/auxiliary files    the test program binary.  This must be used to locate any data/auxiliary files
   stored alongside the binary.    stored alongside the binary.
   
   * RL(integer_expression, integer): Used to evaluate a call to a libc function
     that updates errno when it returns an error and to provide correct error
     reporting.  The integer expression is the call to such function, and the
     literal integer provides the expected return value when there is an error.
     For example: RL(open("foo", O_RDONLY), -1).  This would fail the test case if
     open returns -1, and would record the correct error message returned by libc.
   
 # Shell test programs  # Shell test programs
   
 ## Template  ## Template
Line 563  Which can later be invoked as any of: Line 576  Which can later be invoked as any of:
 Use the "expectations" mechanism to define part of the test case as faulty,  Use the "expectations" mechanism to define part of the test case as faulty,
 crashy, etc.  This is for two reasons:  crashy, etc.  This is for two reasons:
   
 * As long as the bug still exists, he test case will be reported as an "expected  * As long as the bug still exists, the test case will be reported as an
   failure".  Such expected failures do not count towards the success or failure    "expected failure".  Such expected failures do not count towards the success
   of the whole test suite.    or failure of the whole test suite.
   
 * When the bug gets fixed, the bug will not trigger any more in the test case,  * When the bug gets fixed, the bug will not trigger any more in the test case,
   and thus the expectation of failure will not be met any more.  At this point    and thus the expectation of failure will not be met any more.  At this point
   the test case will start raising a regular failure, which is usually addressed    the test case will start raising a regular failure, which is usually addressed
   by either just removing the expect_* calls.    by just removing the expect_* calls (but add a comment with the PR number!).
   
 For example, suppose we have PR lib/1 that reports a condition in which  For example, suppose we have PR lib/1 that reports a condition in which
 snprintf() does the wrong formatting when using %s, and PR lib/2 that mentions  snprintf() does the wrong formatting when using %s, and PR lib/2 that mentions

Removed from v.1.9  
changed lines
  Added in v.1.10


CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb