Diff for /wikisrc/tutorials/atf.mdwn between versions 1.6 and 1.14

version 1.6, 2010/09/03 15:04:51 version 1.14, 2012/08/26 04:21:13
Line 1 Line 1
 [[!meta title="Creating atf-based tests for NetBSD src"]]  [[!meta title="Creating atf-based tests for NetBSD src"]]
 [[!toc ]]  [[!toc ]]
   
 This quick tutorial is an attempt to workaround the lack of proper documentation  # Introduction
 in atf.  The tutorial provides a guideline on how to start creating new test  
   This quick tutorial provides a guideline on how to start creating new test
 programs and/or test cases, how these tests are tied to the NetBSD source tree  programs and/or test cases, how these tests are tied to the NetBSD source tree
 and a short reference of the most commonly used functions.  and includes a short reference of the most commonly used functions.
   
 You should start by reading the  You should start by reading the
 [tests(7)](http://netbsd.gw.com/cgi-bin/man-cgi?tests++NetBSD-current) manual  [tests(7)](http://netbsd.gw.com/cgi-bin/man-cgi?tests++NetBSD-current) manual
 page, which is probably the only sane document in the whole documentation.  Any  page, which provides a user-level overview on how to run the tests included in
 other attempts at reading the atf-* manual pages are probably doomed unless you  NetBSD.  While reading this tutorial, you may also want to refer to these pages
 are already familiar with atf itself and its internals.  Still, you may be able  on a need-to-know basis:
 to get some useful information out of  
 [atf-run(1)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-run++NetBSD-current),  [atf-run(1)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-run++NetBSD-current),
 [atf-report(1)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-report++NetBSD-current),  [atf-report(1)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-report++NetBSD-current),
 [atf-test-program(1)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-test-program++NetBSD-current),  [atf-test-program(1)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-test-program++NetBSD-current),
 [atf-c-api(3)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-c-api++NetBSD-current)  [atf-c-api(3)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-c-api++NetBSD-current),
   [atf-sh-api(3)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-sh-api++NetBSD-current)
 and  and
 [atf-sh-api(3)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-sh-api++NetBSD-current).  [atf-check(1)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-check++NetBSD-current).
   
 **IMPORTANT: Do not take anything for granted, SPECIALLY if you have previously  **IMPORTANT: Do not take anything for granted, SPECIALLY if you have previously
 worked with and/or have seen src/regress/.  Your assumptions are most likely  worked with and/or have seen src/regress/.  Your assumptions are most likely
Line 56  tool: Line 57  tool:
 Then, you could define the following test programs and test cases:  Then, you could define the following test programs and test cases:
   
 * bin/ls/fs_test.c: Provides test cases for list_files and stat_files.  These  * bin/ls/fs_test.c: Provides test cases for list_files and stat_files.  These
   would be named list_files__empty_directory, list_files__one_file,    would be named list_files\_\_empty_directory, list_files\_\_one_file,
   list_files__multiple_files, stat_files__directory, stat_files__symlink, etc.    list_files\_\_multiple_files, stat_files\_\_directory, stat_files\_\_symlink, etc.
   
 * bin/ls/ui_test.c: Provides test cases for the format_columns function.  These  * bin/ls/ui_test.c: Provides test cases for the format_columns function.  These
   would be named format_columns__no_files, format_columns__multiple_files, etc.    would be named format_columns\_\_no_files, format_columns\_\_multiple_files, etc.
   
 * bin/ls/integration_test.sh: Provides "black box" test cases for the binary  * bin/ls/integration_test.sh: Provides "black box" test cases for the binary
   itself.  These would be named lflag, lflag_and_Fflag, no_flags, no_files, etc.    itself.  These would be named lflag, lflag_and_Fflag, no_flags, no_files, etc.
Line 70  require comments to explain what they in Line 71  require comments to explain what they in
   
 # Test case parts  # Test case parts
   
 A test case is composed by three parts: the *head*, the *body* and the  
 *cleanup*.  Only the body is required; the other two routines are optional.  
   
 ## The head  ## The head
   
 The *head* is used **for the sole purpose** to define meta-data properties for  The *head* is used **for the sole purpose** to define meta-data properties for
Line 83  The following properties are commonly us Line 81  The following properties are commonly us
   
 * descr: A textual description of the purpose of the test case.  * descr: A textual description of the purpose of the test case.
   
 * require.user: Set to 'root' to mark the test case as root-specific.  It is  * require.user: Set to 'root' to tell the atf runtime that this test requires
   nice not to abuse this; see puffs and rump.    root privileges.  The test will later be skipped if you are running atf as
     non-root, and the test will be executed otherwise.  Please do not use this
     unless absolutely necessary!  You can most likely make your tests run as a
     regular user if you use puffs and rump.
   
 * use.fs: Set to 'true' if the test case creates temporary files in the "current  * use.fs: Set to 'true' if the test case creates temporary files in the "current
   directory".  Otherwise the atf runtime will isolate the test case in such a    directory".  If set to false, the atf runtime will set the "current directory"
   way to forbid this, which will misteriously make your test to fail.    to an unwritable directory, which will disallow the creation of the temporary
     files and will make your test mysteriously fail.
   
 ## The body  ## The body
   
Line 101  every test case to prevent side-effects  Line 103  every test case to prevent side-effects 
 in-memory data corruption, etc.).  In particular:  in-memory data corruption, etc.).  In particular:
   
 * A test case is **always executed as a subprocess** that is separate from the  * A test case is **always executed as a subprocess** that is separate from the
   head and the cleanup.    head.  This implies that you cannot pass any in-memory state between the
     parts.
   
 * The current working directory of a test case is changed to a temporary  * The current working directory of a test case is changed to a temporary
   location that gets cleaned up later on automatically.  (Set the use.fs    location that gets cleaned up later on automatically.  (Set the use.fs
Line 111  in-memory data corruption, etc.).  In pa Line 114  in-memory data corruption, etc.).  In pa
 * The environment of the test case is "sanitized" to get rid of variables that  * The environment of the test case is "sanitized" to get rid of variables that
   can cause side-effects; e.g. LC_ALL, TZ, etc.    can cause side-effects; e.g. LC_ALL, TZ, etc.
   
 # Installation of test programs: the why and the where  # Running the test programs
   
   Do:
   
       $ cd /usr/tests/
       $ atf-run | atf-report
   
   Why?
   
 Test programs get installed into the /usr/tests/ hierarchy.  The main reason for  Test programs get installed into the /usr/tests/ hierarchy.  The main reason for
 doing that is to allow *any* user to test his system and to be able to convince  doing that is to allow *any* user to test his system and to be able to convince
Line 133  This Makefile in src/tests/bin/ls/ will  Line 143  This Makefile in src/tests/bin/ls/ will 
 ui_test binary.  The Makefile will also generate an Atffile.  Both files (the  ui_test binary.  The Makefile will also generate an Atffile.  Both files (the
 ui_test binary and the Atffile) will later be installed to /usr/tests/bin/ls/  ui_test binary and the Atffile) will later be installed to /usr/tests/bin/ls/
   
   ## Executing a single test
   
   In general, you **do not want to run a test program by hand**.  If you do so,
   you do not take advantage of any of the isolation provided by the atf runtime.
   This means that the test program will probably leave some temporary files behind
   or will raise some false negatives.
   
   To run a test, use atf-run.  In general:
   
       $ atf-run | atf-report  # To run all the test programs in a directory.
       $ atf-run some_test | atf-report  # To run only the some_test program.
   
   The only "legitimate" case in which you should be running test cases by hand is
   to debug them:
   
       $ gdb --args ./some_test the_broken_test_case
   
   ... but be sure to clean up any leftover files if you do that.
   
   ## Executing tests during development
   
   When you are in a subdirectory of src/tests/, you can generally run "make test"
   to execute the tests of that particular subdirectory.  This assumes that the
   tests have been installed into the destdir.
   
   Please note that this is only provided for convenience but it is completely
   unsupported.  Tests run this way may fail mysteriously, and that is perfectly
   fine as long as they work from their canonical locations in /usr/tests.
   
 # Adding a new test  # Adding a new test
   
 To add a new *test case* to the source tree, look for any test program in  To add a new *test case* to the source tree, look for any test program in
Line 164  If the subdirectory does not exist: Line 203  If the subdirectory does not exist:
   
 1. Edit the parent Makefile to recurse into the new subdirectory.  1. Edit the parent Makefile to recurse into the new subdirectory.
   
 1. Edit src/etc/mtree/NetBSD.base.dist to register the new subdirectory.  Your  1. Edit src/etc/mtree/NetBSD.dist.tests to register the new subdirectory.  Your
    test will be installed under /usr/tests/.     test will be installed under /usr/tests/.
   
 1. Edit src/distrib/sets/lists/tests/mi to register the new test program.  Do  1. Edit src/distrib/sets/lists/tests/mi to register the new test program.  Do
Line 172  If the subdirectory does not exist: Line 211  If the subdirectory does not exist:
   
 ## Makefile template  ## Makefile template
   
     # $NetBSD: atf.mdwn,v 1.5 2010/09/03 15:04:05 jmmv Exp $  Follow this template to create your Makefile:
   
     .include <bsd.own.mk>      .include <bsd.own.mk>
   
     # This must always be defined.      # This must always be defined.
     TESTSDIR= ${TESTSBASE}/bin/ls      TESTSDIR= ${TESTSBASE}/bin/ls
   
     # Define only the variables you actually need for the directory.      # These correspond to the test programs you have in the directory.
     TESTS_C+= c1_test c2_test  # Correspond to c1_test.c and c2_test.c.      TESTS_C+= c1_test c2_test  # Correspond to c1_test.c and c2_test.c.
     TESTS_SH+= sh1_test sh2_test  # Correspond to sh1_test.c and sh2_test.c      TESTS_SH+= sh1_test sh2_test  # Correspond to sh1_test.c and sh2_test.c
   
Line 191  If the subdirectory does not exist: Line 230  If the subdirectory does not exist:
   
 ## Atffile template  ## Atffile template
   
   *Atffiles are automatically generated by bsd.test.mk, so in general you will not
   have to deal with them.*
   
 What is an Atffile?  An Atffile is the atf-run counterpart of a "Makefile".  What is an Atffile?  An Atffile is the atf-run counterpart of a "Makefile".
 Given that atf tests *do not rely on a toolchain*, they cannot use make(1) to  Given that atf tests *do not rely on a toolchain*, they cannot use make(1) to
 script their execution as the old tests in src/regress/ did.  script their execution as the old tests in src/regress/ did.
Line 198  script their execution as the old tests  Line 240  script their execution as the old tests 
 The Atffiles, in general, just provide a list of test programs in a particular  The Atffiles, in general, just provide a list of test programs in a particular
 directory and the list of the subdirectories to descend into.  directory and the list of the subdirectories to descend into.
   
 Atffiles are automatically generated by bsd.test.mk, so in general you will not  If you have to provide an Atffile explicitly because the automatic generation
 have to deal with them.  However, if you have to provide one explicitly, they  does not suit your needs, follow this format:
 follow the following format:  
   
     Content-Type: application/X-atf-atffile; version="1"      Content-Type: application/X-atf-atffile; version="1"
   
Line 216  follow the following format: Line 257  follow the following format:
   
 ## Template  ## Template
   
 The following code snippet provides a C test program with two test cases:  The following code snippet provides a C test program with two test cases.  The
   specific details as to how this works follow later:
   
     #include <atf-c.h>      #include <atf-c.h>
   
Line 227  The following code snippet provides a C  Line 269  The following code snippet provides a C 
     }      }
     ATF_TC_BODY(tc, my_test_case)      ATF_TC_BODY(tc, my_test_case)
     {      {
         ATF_CHECK(returns_a_boolean()); /* Non-fatal test. */          ATF_CHECK(true); /* Success; continue execution. */
         ATF_REQUIRE(returns_a_boolean()); /* Non-fatal test. */          ATF_CHECK(false); /* Failure; continue execution. */
   
         ATF_CHECK_EQ(4, 2 + 2); /* Non-fatal test. */          ATF_CHECK_EQ(5, 2 + 2); /* Failure; continue execution. */
         ATF_REQUIRE_EQ(4, 2 + 2); /* Fatal test. */          ATF_REQUIRE_EQ(5, 2 + 2); /* Failure; abort execution. */
   
         if (!condition)          if (!condition)
             atf_tc_fail("Condition not met!"); /* Explicit failure. */              atf_tc_fail("Condition not met!"); /* Abort execution. */
     }      }
   
     ATF_TC_WITHOUT_HEAD(tc, another_test_case);      ATF_TC(tc, another_test_case);
       ATF_TC_HEAD(tc, another_test_case)
       {
           atf_tc_set_md_var(tc, "descr", "This test case ensures that...");
       }
     ATF_TC_BODY(tc, another_test_case)      ATF_TC_BODY(tc, another_test_case)
     {      {
         /* Do more tests here... */          /* Do more tests here... */
Line 249  The following code snippet provides a C  Line 295  The following code snippet provides a C 
         ATF_TP_ADD_TC(tp, another_test_case);          ATF_TP_ADD_TC(tp, another_test_case);
     }      }
   
 This program needs to be linked against libatf-c as described below.  Once  This program needs to be built with the Makefile shown below.  Once built, the
 linked, the program automatically gains a main() method that provides a  program automatically gains a main() method that provides a consistent user
 consistent user interface to all test programs.  You are simply not inteded to  interface to all test programs.  You are simply not intended to provide your own
 provide your own main method, nor to deal with the command-line of the  main method, nor to deal with the command-line of the invocation.
 invocation.  
   
 ## How to build  ## How to build
   
Line 274  For example: Line 319  For example:
   
 The following functions are commonly used from within a test case body:  The following functions are commonly used from within a test case body:
   
 * ATF_CHECK(boolean_expression): Checks if the given boolean expression is true  * ATF_REQUIRE(boolean_expression): Checks if the given boolean expression is
   and, if not, records a failure for the test case but *execution continues*.    true and, if not, aborts execution and marks the test as failed.  Similarly
   Using ATF_REQUIRE aborts execution immediately after a failure.    ATF_CHECK performs the same test but does not abort execution: it records the
     failure but keeps processing the test case.  For an explanation on when to use
 * ATF_CHECK_EQ(expected_expression, actual_expression): Checks if the two    which, refer to the FAQ question below.
   expressions match and, if not, records a failure.  Similarly, ATF_REQUIRE_EQ  
   aborts immediately if the check fails.  * ATF_REQUIRE_EQ(expected_expression, actual_expression): Checks if the two
     expressions match and, if not, aborts marking the test as failed.  Similarly,
     ATF_CHECK_EQ records the error but does not abort execution.
   
 * ATF_CHECK_STREQ(expected_string, actual_string): Same as ATF_CHECK_EQ but  * ATF_REQUIRE_STREQ(expected_string, actual_string): Same as ATF_REQUIRE_EQ but
   performs string comparisons with strcmp.    performs string comparisons with strcmp.
   
 * atf_tc_skip(const char *format, ...): Marks the test case as skipped with the  * atf_tc_skip(const char *format, ...): Marks the test case as skipped with the
Line 297  The following functions are commonly use Line 344  The following functions are commonly use
   
 * atf_expect_fail(const char *format, ...): Tells the atf runtime that the code  * atf_expect_fail(const char *format, ...): Tells the atf runtime that the code
   following this call is expected to raise one or more failures (be it with    following this call is expected to raise one or more failures (be it with
   atf_tc_fail, ATF_CHECK_*, etc.).  Use this to mark a block of code that is    atf_tc_fail, ATF_REQUIRE_*, etc.).  Use this to mark a block of code that is
   known to be broken (e.g. a test that reproduces a known bug).  Use the string    known to be broken (e.g. a test that reproduces a known bug).  Use the string
   parameter to provide an explanation about why the code is broken; if possible,    parameter to provide an explanation about why the code is broken; if possible,
   provide a PR number.  Lastly, to terminate the "expected failure" code block    provide a PR number.  Lastly, to terminate the "expected failure" code block
Line 308  The following functions are commonly use Line 355  The following functions are commonly use
   an abrupt termination of the test case, be it due to a call to exit() or to    an abrupt termination of the test case, be it due to a call to exit() or to
   the reception of a signal.    the reception of a signal.
   
 * atf_expect_exit(int exitcode, const char *fomat, ...): Same as atf_expect_fail  * atf_expect_exit(int exitcode, const char *format, ...): Same as atf_expect_fail
   but expects the test case to exit with a specific exitcode.  Provide -1 to    but expects the test case to exit with a specific exitcode.  Provide -1 to
   indicate any exit code.    indicate any exit code.
   
 * atf_expect_signal(int signo, const char *fomat, ...): Same as atf_expect_fail  * atf_expect_signal(int signo, const char *format, ...): Same as atf_expect_fail
   but expects the test case to receive a specific signal.  Provide -1 to    but expects the test case to receive a specific signal.  Provide -1 to
   indicate any signal.    indicate any signal.
   
 * atf_expect_timeout(const char *reason, ...): Same as atf_expect_fail but  * atf_expect_timeout(const char *reason, ...): Same as atf_expect_fail but
   expects the test case to get stuck and time out.    expects the test case to get stuck and time out.
   
   * atf_tc_get_config_var("srcdir"): Returns the path to the directory containing
     the test program binary.  This must be used to locate any data/auxiliary files
     stored alongside the binary.
   
   * RL(integer_expression, integer): Used to evaluate a call to a libc function
     that updates errno when it returns an error and to provide correct error
     reporting.  The integer expression is the call to such function, and the
     literal integer provides the expected return value when there is an error.
     For example: RL(open("foo", O_RDONLY), -1).  This would fail the test case if
     open returns -1, and would record the correct error message returned by libc.
   
 # Shell test programs  # Shell test programs
   
 ## Template  ## Template
   
 The following code snippet provides a shell test program with two test cases:  The following code snippet provides a shell test program with two test cases.
   The details on how this works are provided later:
   
     atf_test_case my_test_case      atf_test_case my_test_case
     my_test_case_head() {      my_test_case_head() {
Line 336  The following code snippet provides a sh Line 395  The following code snippet provides a sh
     file1      file1
     file2      file2
     EOF      EOF
           # The following call validates that the 'ls' command returns an
           # exit code of 0, that its stdout matches exactly the contents
           # previously stored in the 'expout' file and that its stderr is
           # completely empty.  See atf-check(1) for details, which is the
           # auxiliary tool invoked by the atf_check wrapper function.
         atf_check -s eq:0 -o file:expout -e empty 'ls'          atf_check -s eq:0 -o file:expout -e empty 'ls'
   
         atf_check_equal 4 $((2 + 2))          atf_check_equal 4 $((2 + 2))
Line 346  The following code snippet provides a sh Line 410  The following code snippet provides a sh
     }      }
   
     atf_test_case another_test_case      atf_test_case another_test_case
       another_test_case_head() {
           atf_set "descr" "This test case ensures that..."
       }
     another_test_case_body() {      another_test_case_body() {
         # Do more tests...          # Do more tests...
     }      }
Line 355  The following code snippet provides a sh Line 422  The following code snippet provides a sh
         atf_add_test_case another_test_case          atf_add_test_case another_test_case
     }      }
   
 This program needs to be be executed with the atf-sh(1) interpreter as described  This program needs to be built with the Makefile shown below.  The program
 below.  The program automatically gains an entry point that provides a  automatically gains an entry point that provides a consistent user interface to
 consistent user interface to all test programs.  You are simply not inteded to  all test programs.  You are simply not intended to provide your own "main
 provide your own "main method", nor to deal with the command-line of the  method", nor to deal with the command-line of the invocation.
 invocation.  
   
 ## How to build  ## How to build
   
Line 401  The following functions are commonly use Line 467  The following functions are commonly use
   
 * atf_skip reason: Explicitly marks the test case as skipped and exits.  * atf_skip reason: Explicitly marks the test case as skipped and exits.
   
 * atf_pass: Explicitly markts the test case as passed and exits.  * atf_pass: Explicitly marks the test case as passed and exits.
   
 * atf_get_srcdir: Prints the path to the directory where the test case lives.  * atf_get_srcdir: Prints the path to the directory where the test case lives.
   Use as $(atf_get_srcdir)/my-static-data-file.    Use as $(atf_get_srcdir)/my-static-data-file.
Line 452  keep this in mind: Line 518  keep this in mind:
   integer formatter, one for the string formatter, etc.    integer formatter, one for the string formatter, etc.
   
 * Use the ATF_CHECK* and/or atf_tc_fail functions to do the comparisons and  * Use the ATF_CHECK* and/or atf_tc_fail functions to do the comparisons and
   report errors.  errx should not be used.    report errors.  Neither errx nor any other error reporting and program
     termination functions (read: err, errx, warn, warnx, exit, abort) are to be
     used at all.
   
 The result would look like:  The result would look like:
   
Line 499  The result would look like: Line 567  The result would look like:
   
 Which can later be invoked as any of:  Which can later be invoked as any of:
   
     $ ./snprintf_test integer_formatter      $ atf-run snprintf_test | atf-report  # Normal execution method.
     $ ./snprintf_test string_formatter      $ ./snprintf_test integer_formatter  # For DEBUGGING only.
     $ atf-run snprintf_test | atf-report      $ ./snprintf_test string_formatter  # For DEBUGGING only.
   
 ## How do I write a test case for an unfixed PR?  ## How do I write a test case for an unfixed PR?
   
 Use the "expectations" mechanism to define part of the test case as faulty,  Use the "expectations" mechanism to define part of the test case as faulty,
 crashy, etc.  For example, suppose we have PR 1 that reports a condition in  crashy, etc.  This is for two reasons:
 which snprintf() does the wrong formatting when using %s, and PR 2 that mentions  
   * As long as the bug still exists, the test case will be reported as an
     "expected failure".  Such expected failures do not count towards the success
     or failure of the whole test suite.
   
   * When the bug gets fixed, the bug will not trigger any more in the test case,
     and thus the expectation of failure will not be met any more.  At this point
     the test case will start raising a regular failure, which is usually addressed
     by just removing the expect_* calls (but add a comment with the PR number!).
   
   For example, suppose we have PR lib/1 that reports a condition in which
   snprintf() does the wrong formatting when using %s, and PR lib/2 that mentions
 that another snprintf() call using %d with number 5 causes a segfault.  We could  that another snprintf() call using %d with number 5 causes a segfault.  We could
 do:  do:
   
Line 515  do: Line 594  do:
     #include <signal.h>      #include <signal.h>
     #include <stdio.h>      #include <stdio.h>
   
     ATF_TC_WITHOUT_HEAD(tc, integer_formatter);      ATF_TC(tc, integer_formatter);
       ATF_TC_HEAD(tc, integer_formatter)
       {
           atf_tc_set_md_var(tc, "descr", "Tests the %d formatter for snprintf");
       }
     ATF_TC_BODY(tc, integer_formatter)      ATF_TC_BODY(tc, integer_formatter)
     {      {
         char buf[1024];          char buf[1024];
Line 523  do: Line 606  do:
         snprintf(buf, sizeof(buf), "Hello %d\n", 1);          snprintf(buf, sizeof(buf), "Hello %d\n", 1);
         ATF_CHECK_STREQ("Hello 1", buf);          ATF_CHECK_STREQ("Hello 1", buf);
   
         atf_tc_expect_signal(SIGSEGV, "PR 2: %%d with 5 causes a crash");          atf_tc_expect_signal(SIGSEGV, "PR lib/2: %%d with 5 causes a crash");
         snprintf(buf, sizeof(buf), "Hello %d\n", 5);          snprintf(buf, sizeof(buf), "Hello %d\n", 5);
         atf_tc_expect_pass();          atf_tc_expect_pass();
         ATF_CHECK_STREQ("Hello 5", buf);          ATF_CHECK_STREQ("Hello 5", buf);
     }      }
   
     ATF_TC_WITHOUT_HEAD(tc, string_formatter);      ATF_TC(tc, string_formatter);
       ATF_TC_HEAD(tc, string_formatter)
       {
           atf_tc_set_md_var(tc, "descr", "Tests the %s formatter for snprintf");
       }
     ATF_TC_BODY(tc, string_formatter)      ATF_TC_BODY(tc, string_formatter)
     {      {
         char buf[1024];          char buf[1024];
   
         snprintf(buf, sizeof(buf), "Hello %s\n", "world!");          snprintf(buf, sizeof(buf), "Hello %s\n", "world!");
         atf_tc_expect_failure("PR 1: %%s does not work");          atf_tc_expect_failure("PR lib/1: %%s does not work");
         ATF_CHECK_STREQ("Hello world!", buf);          ATF_CHECK_STREQ("Hello world!", buf);
         atf_tc_expect_pass();          atf_tc_expect_pass();
     }      }
Line 550  do: Line 637  do:
   
 No.  atf-run does this automatically for you, because it runs every test program  No.  atf-run does this automatically for you, because it runs every test program
 in its own temporary subdirectory.  in its own temporary subdirectory.
   
   ## When do I use ATF_CHECK and when ATF_REQUIRE?
   
   ATF_CHECK logs errors but does not abort the execution of the test program.
   ATF_REQUIRE logs errors in a similar way but immediately terminates the
   execution.
   
   You can use this distinction in the following way: use ATF_REQUIRE to check the
   code that "prepares" your test case.  Use ATF_CHECK to do the actual
   functionality tests once all the set up has been performed.  For example:
   
       ATF_TC_BODY(getline) {
           FILE *f;
           char buf[1024];
   
           /* Opening the file is not part of the functionality under test, but it
            * must succeed before we actually test the relevant code. */
           ATF_REQUIRE((f = fopen("foo")) != NULL);
   
           ATF_CHECK(getline(f, buf, sizeof(buf)) > 0);
           ATF_CHECK_STREQ("line 1", buf);
   
           ATF_CHECK(getline(f, buf, sizeof(buf)) > 0);
           ATF_CHECK_STREQ("line 2", buf);
       }

Removed from v.1.6  
changed lines
  Added in v.1.14


CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb