Annotation of wikisrc/tutorials/atf.mdwn, revision 1.2

1.1       jmmv        1: # Creating atf-based tests for NetBSD src
                      2: 
                      3: This quick tutorial is an attempt to workaround the lack of proper documentation
                      4: in atf.  The tutorial provides a guideline on how to start creating new test
                      5: programs and/or test cases, how these tests are tied to the NetBSD source tree
                      6: and a short reference of the most commonly used functions.
                      7: 
                      8: You should start by reading the
                      9: [tests(7)](http://netbsd.gw.com/cgi-bin/man-cgi?tests++NetBSD-current) manual
                     10: page, which is probably the only sane document in the whole documentation.  Any
                     11: other attempts at reading the atf-* manual pages are probably doomed unless you
                     12: are already familiar with atf itself and its internals.  Still, you may be able
                     13: to get some useful information out of
                     14: [atf-run(1)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-run++NetBSD-current),
                     15: [atf-report(1)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-report++NetBSD-current),
                     16: [atf-test-program(1)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-test-program++NetBSD-current),
                     17: [atf-c-api(3)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-c-api++NetBSD-current)
                     18: and
                     19: [atf-sh-api(3)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-sh-api++NetBSD-current).
                     20: 
                     21: **IMPORTANT: Do not take anything for granted, SPECIALLY if you have previously
                     22: worked with and/or have seen src/regress/.  Your assumptions are most likely
1.2     ! jmmv       23: incorrect.**
1.1       jmmv       24: 
                     25: ## Test programs vs. test cases
                     26: 
                     27: So, what is what and how do you organize your tests?
                     28: 
                     29: A **test case** is a piece of code that exercises a particular functionality of
                     30: another piece of code.  Commonly, test cases validate the outcome of a
                     31: particular source function or class method, the validity of the execution of a
                     32: command with a particular combination of flags/arguments, etc.  Test cases are
                     33: supposed to be very concise, in the sense that they should just be testing *one
                     34: behavior*.
                     35: 
                     36: A **test program** is a binary that collects and exposes a group of test cases.
                     37: Typically, these test programs expose conceptually-related tests or all the
                     38: tests for a particular source file.
                     39: 
                     40: In general, having many test programs with **just one test case** in them is
                     41: **wrong** and smells from the previous layout of src/regress/.  Think about some
                     42: other organization.  And don't blame atf for this separation: this is extremely
                     43: common in (almost?) all other test frameworks and, when used wisely, becomes an
                     44: invaluable classification.
                     45: 
                     46: For example, suppose you have the following fictitious source files for the ls
                     47: tool:
                     48: 
                     49: * bin/ls/fs.c: Provides the list_files() and stat_files() functions.
                     50: 
                     51: * bin/ls/ui.c: Provides the format_columns() function.
                     52: 
                     53: * bin/ls/main.c: The main method for ls.
                     54: 
                     55: Then, you could define the following test programs and test cases:
                     56: 
                     57: * bin/ls/fs_test.c: Provides test cases for list_files and stat_files.  These
                     58:   would be named list_files__empty_directory, list_files__one_file,
                     59:   list_files__multiple_files, stat_files__directory, stat_files__symlink, etc.
                     60: 
                     61: * bin/ls/ui_test.c: Provides test cases for the format_columns function.  These
                     62:   would be named format_columns__no_files, format_columns__multiple_files, etc.
                     63: 
                     64: * bin/ls/integration_test.sh: Provides "black box" test cases for the binary
                     65:   itself.  These would be named lflag, lflag_and_Fflag, no_flags, no_files, etc.
                     66: 
1.2     ! jmmv       67: Try to keep your test case names as descriptive as possible so that they do not
        !            68: require comments to explain what they intend to test.
        !            69: 
        !            70: ## Installation of test programs: the why and the where
        !            71: 
        !            72: Test programs get installed into the /usr/tests/ hierarchy.  The main reason for
        !            73: doing that is to allow *any* user to test his system and to be able to convince
        !            74: himself that everything is working correctly.
        !            75: 
        !            76: Imagine that you install NetBSD-current on a public-facing machine that has some
        !            77: particular hardware only supported in the bleeding-edge source tree.  In this
        !            78: scenario, you, as the administrator, could just go into /usr/tests/, run the
        !            79: tests and know immediately if everything is working correctly in your
        !            80: software+hardware combination or not.  No need to rely on promises from the
        !            81: vendor, no need to deal with a source tree, no need to have a compiler
        !            82: installed...
        !            83: 
        !            84: So, that's the theory.  Now, how does this map to our source tree?
        !            85: 
        !            86: At the moment, the source test programs are located somewhere under src/tests/.
        !            87: Say, for example, that you have the src/tests/bin/ls/ui_test.c source file.
        !            88: This Makefile in src/tests/bin/ls/ will take this source file and generate a
        !            89: ui_test binary.  The Makefile will also generate an Atffile.  Both files (the
        !            90: ui_test binary and the Atffile) will later be installed to /usr/tests/bin/ls/
1.1       jmmv       91: 
                     92: ## Adding a new test
                     93: 
                     94: To add a new *test case* to the source tree, look for any test program in
                     95: src/tests/ that can assimilate it.  If you find such a program, just add the
1.2     ! jmmv       96: test case to it: no other changes are required so your life is easy.  Otherwise,
        !            97: you will have to create a new test program.
1.1       jmmv       98: 
                     99: To add a new *test program* to the source tree:
                    100: 
                    101: 1. Locate the appropriate subdirectory in which to put your test program.  It is
                    102: OK (and **expected**) to have multiple test programs into the same directory.
                    103: **Restrain yourself from creating one directory per test program.**
                    104: 
                    105: If the subdirectory exists:
                    106: 
                    107: 1. Choose a sane name for the test program; the name must not be so specific
                    108:    that it restricts the addition of future test cases into it.
                    109: 
                    110: 1. Create the test program source file using one of the templates below.
                    111:    E.g. src/tests/tutorial/sample_test.c.
                    112: 
                    113: 1. Add the new test program to the Makefile.
                    114: 
                    115: If the subdirectory does not exist:
                    116: 
                    117: 1. Do the same as above.
                    118: 
                    119: 1. Create the Makefile for the directory using the templates below.
                    120: 
                    121: 1. Edit the parent Makefile to recurse into the new subdirectory.
                    122: 
                    123: 1. Edit src/etc/mtree/NetBSD.base.dist to register the new subdirectory.  Your
                    124:    test will be installed under /usr/tests/.
                    125: 
                    126: 1. Edit src/distrib/sets/lists/tests/mi to register the new test program.  Do
                    127:    not forget to add .debug entries if your test program is a C/C++ binary.
                    128: 
                    129: ### Makefile template
                    130: 
1.2     ! jmmv      131:     # $NetBSD: atf.mdwn,v 1.1 2010/09/01 20:59:24 jmmv Exp $
1.1       jmmv      132: 
                    133:     .include <bsd.own.mk>
                    134: 
                    135:     # This must always be defined.
                    136:     TESTSDIR= ${TESTSBASE}/bin/ls
                    137: 
                    138:     # Define only the variables you actually need for the directory.
                    139:     TESTS_C+= c1_test c2_test  # Correspond to c1_test.c and c2_test.c.
                    140:     TESTS_SH+= sh1_test sh2_test  # Correspond to sh1_test.c and sh2_test.c
                    141: 
                    142:     # Define only if your tests need any data files.
                    143:     FILESDIR= ${TESTSDIR}
                    144:     FILES= testdata1.txt testdata2.bin  # Any necessary data files.
                    145: 
                    146:     .include <bsd.test.mk>
                    147: 
1.2     ! jmmv      148: ### Atffile template
        !           149: 
        !           150: What is an Atffile?  An Atffile is the atf-run counterpart of a "Makefile".
        !           151: Given that atf tests *do not rely on a toolchain*, they cannot use make(1) to
        !           152: script their execution as the old tests in src/regress/ did.
        !           153: 
        !           154: The Atffiles, in general, just provide a list of test programs in a particular
        !           155: directory and the list of the subdirectories to descend into.
        !           156: 
        !           157: Atffiles are automatically generated by bsd.test.mk, so in general you will not
        !           158: have to deal with them.  However, if you have to provide one explicitly, they
        !           159: follow the following format:
        !           160: 
        !           161:     Content-Type: application/X-atf-atffile; version="1"
        !           162: 
        !           163:     prop: test-suite = NetBSD
        !           164: 
        !           165:     tp: first_test
        !           166:     tp: second_test
        !           167:     tp-glob: optional_*_test
        !           168:     tp: subdir1
        !           169:     tp: subdir2
        !           170: 
1.1       jmmv      171: ## C test programs
                    172: 
                    173: ### Template
                    174: 
1.2     ! jmmv      175: The following code snippet provides a C test program with two test cases:
        !           176: 
1.1       jmmv      177:     #include <atf-c.h>
                    178: 
                    179:     ATF_TC(tc, my_test_case);
                    180:     ATF_TC_HEAD(tc, my_test_case)
                    181:     {
                    182:         atf_tc_set_md_var(tc, "descr", "This test case ensures that...");
                    183:     }
                    184:     ATF_TC_BODY(tc, my_test_case)
                    185:     {
                    186:         ATF_CHECK(returns_a_boolean()); /* Non-fatal test. */
                    187:         ATF_REQUIRE(returns_a_boolean()); /* Non-fatal test. */
                    188: 
                    189:         ATF_CHECK_EQ(4, 2 + 2); /* Non-fatal test. */
                    190:         ATF_REQUIRE_EQ(4, 2 + 2); /* Fatal test. */
                    191: 
                    192:         if (!condition)
                    193:             atf_tc_fail("Condition not met!"); /* Explicit failure. */
                    194:     }
                    195: 
1.2     ! jmmv      196:     ATF_TC_WITHOUT_HEAD(tc, another_test_case);
        !           197:     ATF_TC_BODY(tc, another_test_case)
        !           198:     {
        !           199:         /* Do more tests here... */
        !           200:     }
        !           201: 
1.1       jmmv      202:     ATF_TP_ADD_TCS(tp)
                    203:     {
                    204:         ATF_TP_ADD_TC(tp, my_test_case);
1.2     ! jmmv      205:         ATF_TP_ADD_TC(tp, another_test_case);
1.1       jmmv      206:     }
                    207: 
1.2     ! jmmv      208: This program needs to be linked against libatf-c as described below.  Once
        !           209: linked, the program automatically gains a main() method that provides a
        !           210: consistent user interface to all test programs.  You are simply not inteded to
        !           211: provide your own main method, nor to deal with the command-line of the
        !           212: invocation.
        !           213: 
1.1       jmmv      214: ### How to build
                    215: 
                    216: To build a C test program, append the name of the test program (without the .c
                    217: extension) to the TESTS_C variable in the Makefile.
                    218: 
                    219: For example:
                    220: 
                    221:     .include <bsd.own.mk>
                    222: 
                    223:     TESTSDIR= ${TESTSBASE}/bin/ls
                    224: 
                    225:     TESTS_C+= fs_test ui_test
                    226: 
                    227:     .include <bsd.test.mk>
                    228: 
                    229: ## Shell test programs
                    230: 
                    231: ### Template
                    232: 
1.2     ! jmmv      233: The following code snippet provides a shell test program with two test cases:
        !           234: 
1.1       jmmv      235:     atf_test_case my_test_case
                    236:     my_test_case_head() {
                    237:         atf_set "descr" "This test case ensures that..."
                    238:     }
                    239:     my_test_case_body() {
                    240:         touch file1 file2
                    241: 
                    242:         cat >expout <<EOF
                    243:     file1
                    244:     file2
                    245:     EOF
                    246:         atf_check -s eq:0 -o file:expout -e empty 'ls'
                    247: 
                    248:         atf_check_equal 4 $((2 + 2))
                    249: 
                    250:         if [ 'a' != 'b' ]; then
                    251:             atf_fail "Condition not met!"  # Explicit failure.
                    252:         fi
                    253:     }
                    254: 
1.2     ! jmmv      255:     atf_test_case another_test_case
        !           256:     another_test_case_body() {
        !           257:         # Do more tests...
        !           258:     }
        !           259: 
1.1       jmmv      260:     atf_init_test_cases() {
                    261:         atf_add_test_case my_test_case
1.2     ! jmmv      262:         atf_add_test_case another_test_case
1.1       jmmv      263:     }
                    264: 
1.2     ! jmmv      265: This program needs to be be executed with the atf-sh(1) interpreter as described
        !           266: below.  The program automatically gains an entry point that provides a
        !           267: consistent user interface to all test programs.  You are simply not inteded to
        !           268: provide your own "main method", nor to deal with the command-line of the
        !           269: invocation.
        !           270: 
1.1       jmmv      271: ### How to build
                    272: 
                    273: To build a shell test program, append the name of the test program (without the
                    274: .sh extension) to the TESTS_SH variable in the Makefile.
                    275: 
                    276: For example:
                    277: 
                    278:     .include <bsd.own.mk>
                    279: 
                    280:     TESTSDIR= ${TESTSBASE}/bin/ls
                    281: 
                    282:     TESTS_SH+= integration_test something_else_test
                    283: 
                    284:     .include <bsd.test.mk>
                    285: 
                    286: If you want to run the test program yourself, you should know that shell-based
                    287: test programs are processed with the atf-sh interpreter.  atf-sh is just a thin
                    288: wrapper over /bin/sh that loads the shared atf code and then delegates execution
                    289: to your source file.
                    290: 
                    291: ## FAQ
                    292: 
                    293: ### How do I atfify a plain test program?
                    294: 
                    295: Let's suppose you have a program to exercise a particular piece of code.
                    296: Conceptually this implements a test but it does not use atf at all.  For
                    297: example:
                    298: 
                    299:     #include <err.h>
                    300:     #include <stdio.h>
                    301:     #include <stdlib.h>
                    302:     #include <string.h>
                    303: 
                    304:     /* This test program exercises the snprintf function. */
                    305: 
                    306:     int main(void)
                    307:     {
                    308:         char buf[1024];
                    309: 
                    310:         printf("Testing integers");
                    311:         snprintf(buf, sizeof(buf), "%d", 3);
                    312:         if (strcmp(buf, "3") != 0)
                    313:             errx(EXIT_FAILURE, "%d failed");
                    314:         snprintf(buf, sizeof(buf), "a %d b", 5);
                    315:         if (strcmp(buf, "a 5 b") != 0)
                    316:             errx(EXIT_FAILURE, "%d failed");
                    317: 
                    318:         printf("Testing strings");
                    319:         snprintf(buf, sizeof(buf), "%s", "foo");
                    320:         if (strcmp(buf, "foo") != 0)
                    321:             errx(EXIT_FAILURE, "%s failed");
                    322:         snprintf(buf, sizeof(buf), "a %s b", "bar");
                    323:         if (strcmp(buf, "a bar b") != 0)
                    324:             errx(EXIT_FAILURE, "%s failed");
                    325: 
                    326:         return EXIT_SUCCESS;
                    327:     }
                    328: 
                    329: To convert this program into an atf test program, use the template above and
                    330: keep this in mind:
                    331: 
                    332: * Split the whole main function into separate test cases.  In this scenario, the
                    333:   calls to printf(3) delimit a good granularity for the test cases: one for the
                    334:   integer formatter, one for the string formatter, etc.
                    335: 
                    336: * Use the ATF_CHECK* and/or atf_tc_fail functions to do the comparisons and
                    337:   report errors.  errx should not be used.
                    338: 
                    339: The result would look like:
                    340: 
                    341:     #include <atf-c.h>
                    342:     #include <stdio.h>
                    343: 
                    344:     ATF_TC(tc, integer_formatter);
                    345:     ATF_TC_HEAD(tc, integer_formatter)
                    346:     {
                    347:         atf_tc_set_md_var(tc, "descr", "Validates the %d formatter");
                    348:     }
                    349:     ATF_TC_BODY(tc, integer_formatter)
                    350:     {
                    351:         char buf[1024];
                    352: 
                    353:         snprintf(buf, sizeof(1024), "%d", 3);
                    354:         ATF_CHECK_STREQ("3", buf);
                    355: 
                    356:         snprintf(buf, sizeof(1024), "a %d b", 5);
                    357:         ATF_CHECK_STREQ("a 5 b", buf);
                    358:     }
                    359: 
                    360:     ATF_TC(tc, string_formatter);
                    361:     ATF_TC_HEAD(tc, string_formatter)
                    362:     {
                    363:         atf_tc_set_md_var(tc, "descr", "Validates the %s formatter");
                    364:     }
                    365:     ATF_TC_BODY(tc, string_formatter)
                    366:     {
                    367:         char buf[1024];
                    368: 
                    369:         snprintf(buf, sizeof(1024), "%s", "foo");
                    370:         ATF_CHECK_STREQ("foo", buf);
                    371: 
                    372:         snprintf(buf, sizeof(1024), "a %s b", "bar");
                    373:         ATF_CHECK_STREQ("a bar b", buf);
                    374:     }
                    375: 
                    376:     ATF_TP_ADD_TCS(tp)
                    377:     {
                    378:         ATF_TP_ADD_TC(tp, integer_formatter);
                    379:         ATF_TP_ADD_TC(tp, string_formatter);
                    380:     }
                    381: 
                    382: Which can later be invoked as any of:
                    383: 
                    384:     $ ./snprintf_test integer_formatter
                    385:     $ ./snprintf_test string_formatter
                    386:     $ atf-run snprintf_test | atf-report
                    387: 
                    388: ### Do I need to remove temporary files?
                    389: 
                    390: No.  atf-run does this automatically for you, because it runs every test program
                    391: in its own temporary subdirectory.

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb