Diff for /wikisrc/tutorials/atf.mdwn between versions 1.4 and 1.5

version 1.4, 2010/09/03 14:53:05 version 1.5, 2010/09/03 15:04:05
Line 1 Line 1
 # Creating atf-based tests for NetBSD src  [[!meta title="Creating atf-based tests for NetBSD src"]]
   
 [[!toc ]]  [[!toc ]]
   
   # Creating atf-based tests for NetBSD src
   
 This quick tutorial is an attempt to workaround the lack of proper documentation  This quick tutorial is an attempt to workaround the lack of proper documentation
 in atf.  The tutorial provides a guideline on how to start creating new test  in atf.  The tutorial provides a guideline on how to start creating new test
 programs and/or test cases, how these tests are tied to the NetBSD source tree  programs and/or test cases, how these tests are tied to the NetBSD source tree
Line 24  and Line 25  and
 worked with and/or have seen src/regress/.  Your assumptions are most likely  worked with and/or have seen src/regress/.  Your assumptions are most likely
 incorrect.**  incorrect.**
   
 ## Test programs vs. test cases  # Test programs vs. test cases
   
 So, what is what and how do you organize your tests?  So, what is what and how do you organize your tests?
   
Line 69  Then, you could define the following tes Line 70  Then, you could define the following tes
 Try to keep your test case names as descriptive as possible so that they do not  Try to keep your test case names as descriptive as possible so that they do not
 require comments to explain what they intend to test.  require comments to explain what they intend to test.
   
 ## Test case parts  # Test case parts
   
 A test case is composed by three parts: the *head*, the *body* and the  A test case is composed by three parts: the *head*, the *body* and the
 *cleanup*.  Only the body is required; the other two routines are optional.  *cleanup*.  Only the body is required; the other two routines are optional.
   
 ### The head  ## The head
   
 The *head* is used **for the sole purpose** to define meta-data properties for  The *head* is used **for the sole purpose** to define meta-data properties for
 the test case.  (Eventually, this would not be specified programmatically, but  the test case.  (Eventually, this would not be specified programmatically, but
Line 91  The following properties are commonly us Line 92  The following properties are commonly us
   directory".  Otherwise the atf runtime will isolate the test case in such a    directory".  Otherwise the atf runtime will isolate the test case in such a
   way to forbid this, which will misteriously make your test to fail.    way to forbid this, which will misteriously make your test to fail.
   
 ### The body  ## The body
   
 The *body* is the actual meat of the test case.  This is just a regular function  The *body* is the actual meat of the test case.  This is just a regular function
 that executes any code you want and calls special atf functions to report  that executes any code you want and calls special atf functions to report
Line 112  in-memory data corruption, etc.).  In pa Line 113  in-memory data corruption, etc.).  In pa
 * The environment of the test case is "sanitized" to get rid of variables that  * The environment of the test case is "sanitized" to get rid of variables that
   can cause side-effects; e.g. LC_ALL, TZ, etc.    can cause side-effects; e.g. LC_ALL, TZ, etc.
   
 ## Installation of test programs: the why and the where  # Installation of test programs: the why and the where
   
 Test programs get installed into the /usr/tests/ hierarchy.  The main reason for  Test programs get installed into the /usr/tests/ hierarchy.  The main reason for
 doing that is to allow *any* user to test his system and to be able to convince  doing that is to allow *any* user to test his system and to be able to convince
Line 134  This Makefile in src/tests/bin/ls/ will  Line 135  This Makefile in src/tests/bin/ls/ will 
 ui_test binary.  The Makefile will also generate an Atffile.  Both files (the  ui_test binary.  The Makefile will also generate an Atffile.  Both files (the
 ui_test binary and the Atffile) will later be installed to /usr/tests/bin/ls/  ui_test binary and the Atffile) will later be installed to /usr/tests/bin/ls/
   
 ## Adding a new test  # Adding a new test
   
 To add a new *test case* to the source tree, look for any test program in  To add a new *test case* to the source tree, look for any test program in
 src/tests/ that can assimilate it.  If you find such a program, just add the  src/tests/ that can assimilate it.  If you find such a program, just add the
Line 171  If the subdirectory does not exist: Line 172  If the subdirectory does not exist:
 1. Edit src/distrib/sets/lists/tests/mi to register the new test program.  Do  1. Edit src/distrib/sets/lists/tests/mi to register the new test program.  Do
    not forget to add .debug entries if your test program is a C/C++ binary.     not forget to add .debug entries if your test program is a C/C++ binary.
   
 ### Makefile template  ## Makefile template
   
     # $NetBSD: atf.mdwn,v 1.3 2010/09/03 13:35:32 jmmv Exp $      # $NetBSD: atf.mdwn,v 1.4 2010/09/03 14:53:05 jmmv Exp $
   
     .include <bsd.own.mk>      .include <bsd.own.mk>
   
Line 190  If the subdirectory does not exist: Line 191  If the subdirectory does not exist:
   
     .include <bsd.test.mk>      .include <bsd.test.mk>
   
 ### Atffile template  ## Atffile template
   
 What is an Atffile?  An Atffile is the atf-run counterpart of a "Makefile".  What is an Atffile?  An Atffile is the atf-run counterpart of a "Makefile".
 Given that atf tests *do not rely on a toolchain*, they cannot use make(1) to  Given that atf tests *do not rely on a toolchain*, they cannot use make(1) to
Line 213  follow the following format: Line 214  follow the following format:
     tp: subdir1      tp: subdir1
     tp: subdir2      tp: subdir2
   
 ## C test programs  # C test programs
   
 ### Template  ## Template
   
 The following code snippet provides a C test program with two test cases:  The following code snippet provides a C test program with two test cases:
   
Line 256  consistent user interface to all test pr Line 257  consistent user interface to all test pr
 provide your own main method, nor to deal with the command-line of the  provide your own main method, nor to deal with the command-line of the
 invocation.  invocation.
   
 ### How to build  ## How to build
   
 To build a C test program, append the name of the test program (without the .c  To build a C test program, append the name of the test program (without the .c
 extension) to the TESTS_C variable in the Makefile.  extension) to the TESTS_C variable in the Makefile.
Line 271  For example: Line 272  For example:
   
     .include <bsd.test.mk>      .include <bsd.test.mk>
   
 ### Common functions  ## Common functions
   
 The following functions are commonly used from within a test case body:  The following functions are commonly used from within a test case body:
   
Line 320  The following functions are commonly use Line 321  The following functions are commonly use
 * atf_expect_timeout(const char *reason, ...): Same as atf_expect_fail but  * atf_expect_timeout(const char *reason, ...): Same as atf_expect_fail but
   expects the test case to get stuck and time out.    expects the test case to get stuck and time out.
   
 ## Shell test programs  # Shell test programs
   
 ### Template  ## Template
   
 The following code snippet provides a shell test program with two test cases:  The following code snippet provides a shell test program with two test cases:
   
Line 362  consistent user interface to all test pr Line 363  consistent user interface to all test pr
 provide your own "main method", nor to deal with the command-line of the  provide your own "main method", nor to deal with the command-line of the
 invocation.  invocation.
   
 ### How to build  ## How to build
   
 To build a shell test program, append the name of the test program (without the  To build a shell test program, append the name of the test program (without the
 .sh extension) to the TESTS_SH variable in the Makefile.  .sh extension) to the TESTS_SH variable in the Makefile.
Line 382  test programs are processed with the atf Line 383  test programs are processed with the atf
 wrapper over /bin/sh that loads the shared atf code and then delegates execution  wrapper over /bin/sh that loads the shared atf code and then delegates execution
 to your source file.  to your source file.
   
 ### Common functions  ## Common functions
   
 The following functions are commonly used from within a test case body:  The following functions are commonly used from within a test case body:
   
Line 407  The following functions are commonly use Line 408  The following functions are commonly use
 * atf_get_srcdir: Prints the path to the directory where the test case lives.  * atf_get_srcdir: Prints the path to the directory where the test case lives.
   Use as $(atf_get_srcdir)/my-static-data-file.    Use as $(atf_get_srcdir)/my-static-data-file.
   
 ## FAQ  # FAQ
   
 ### How do I atfify a plain test program?  ## How do I atfify a plain test program?
   
 Let's suppose you have a program to exercise a particular piece of code.  Let's suppose you have a program to exercise a particular piece of code.
 Conceptually this implements a test but it does not use atf at all.  For  Conceptually this implements a test but it does not use atf at all.  For
Line 504  Which can later be invoked as any of: Line 505  Which can later be invoked as any of:
     $ ./snprintf_test string_formatter      $ ./snprintf_test string_formatter
     $ atf-run snprintf_test | atf-report      $ atf-run snprintf_test | atf-report
   
 ### How do I write a test case for an unfixed PR?  ## How do I write a test case for an unfixed PR?
   
 Use the "expectations" mechanism to define part of the test case as faulty,  Use the "expectations" mechanism to define part of the test case as faulty,
 crashy, etc.  For example, suppose we have PR 1 that reports a condition in  crashy, etc.  For example, suppose we have PR 1 that reports a condition in
Line 547  do: Line 548  do:
         ATF_TP_ADD_TC(tp, string_formatter);          ATF_TP_ADD_TC(tp, string_formatter);
     }      }
   
 ### Do I need to remove temporary files?  ## Do I need to remove temporary files?
   
 No.  atf-run does this automatically for you, because it runs every test program  No.  atf-run does this automatically for you, because it runs every test program
 in its own temporary subdirectory.  in its own temporary subdirectory.

Removed from v.1.4  
changed lines
  Added in v.1.5


CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb