Annotation of wikisrc/tutorials/atf.mdwn, revision 1.1

1.1     ! jmmv        1: # Creating atf-based tests for NetBSD src
        !             2: 
        !             3: This quick tutorial is an attempt to workaround the lack of proper documentation
        !             4: in atf.  The tutorial provides a guideline on how to start creating new test
        !             5: programs and/or test cases, how these tests are tied to the NetBSD source tree
        !             6: and a short reference of the most commonly used functions.
        !             7: 
        !             8: You should start by reading the
        !             9: [tests(7)](http://netbsd.gw.com/cgi-bin/man-cgi?tests++NetBSD-current) manual
        !            10: page, which is probably the only sane document in the whole documentation.  Any
        !            11: other attempts at reading the atf-* manual pages are probably doomed unless you
        !            12: are already familiar with atf itself and its internals.  Still, you may be able
        !            13: to get some useful information out of
        !            14: [atf-run(1)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-run++NetBSD-current),
        !            15: [atf-report(1)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-report++NetBSD-current),
        !            16: [atf-test-program(1)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-test-program++NetBSD-current),
        !            17: [atf-c-api(3)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-c-api++NetBSD-current)
        !            18: and
        !            19: [atf-sh-api(3)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-sh-api++NetBSD-current).
        !            20: 
        !            21: **IMPORTANT: Do not take anything for granted, SPECIALLY if you have previously
        !            22: worked with and/or have seen src/regress/.  Your assumptions are most likely
        !            23: wrong.**
        !            24: 
        !            25: ## Test programs vs. test cases
        !            26: 
        !            27: So, what is what and how do you organize your tests?
        !            28: 
        !            29: A **test case** is a piece of code that exercises a particular functionality of
        !            30: another piece of code.  Commonly, test cases validate the outcome of a
        !            31: particular source function or class method, the validity of the execution of a
        !            32: command with a particular combination of flags/arguments, etc.  Test cases are
        !            33: supposed to be very concise, in the sense that they should just be testing *one
        !            34: behavior*.
        !            35: 
        !            36: A **test program** is a binary that collects and exposes a group of test cases.
        !            37: Typically, these test programs expose conceptually-related tests or all the
        !            38: tests for a particular source file.
        !            39: 
        !            40: In general, having many test programs with **just one test case** in them is
        !            41: **wrong** and smells from the previous layout of src/regress/.  Think about some
        !            42: other organization.  And don't blame atf for this separation: this is extremely
        !            43: common in (almost?) all other test frameworks and, when used wisely, becomes an
        !            44: invaluable classification.
        !            45: 
        !            46: For example, suppose you have the following fictitious source files for the ls
        !            47: tool:
        !            48: 
        !            49: * bin/ls/fs.c: Provides the list_files() and stat_files() functions.
        !            50: 
        !            51: * bin/ls/ui.c: Provides the format_columns() function.
        !            52: 
        !            53: * bin/ls/main.c: The main method for ls.
        !            54: 
        !            55: Then, you could define the following test programs and test cases:
        !            56: 
        !            57: * bin/ls/fs_test.c: Provides test cases for list_files and stat_files.  These
        !            58:   would be named list_files__empty_directory, list_files__one_file,
        !            59:   list_files__multiple_files, stat_files__directory, stat_files__symlink, etc.
        !            60: 
        !            61: * bin/ls/ui_test.c: Provides test cases for the format_columns function.  These
        !            62:   would be named format_columns__no_files, format_columns__multiple_files, etc.
        !            63: 
        !            64: * bin/ls/integration_test.sh: Provides "black box" test cases for the binary
        !            65:   itself.  These would be named lflag, lflag_and_Fflag, no_flags, no_files, etc.
        !            66: 
        !            67: Try to keep your test case names as descriptive as possible.
        !            68: 
        !            69: ## Adding a new test
        !            70: 
        !            71: To add a new *test case* to the source tree, look for any test program in
        !            72: src/tests/ that can assimilate it.  If you find such a program, just add the
        !            73: test case to it; no other changes are required.  Otherwise, you will have to
        !            74: create a new test program.
        !            75: 
        !            76: To add a new *test program* to the source tree:
        !            77: 
        !            78: 1. Locate the appropriate subdirectory in which to put your test program.  It is
        !            79: OK (and **expected**) to have multiple test programs into the same directory.
        !            80: **Restrain yourself from creating one directory per test program.**
        !            81: 
        !            82: If the subdirectory exists:
        !            83: 
        !            84: 1. Choose a sane name for the test program; the name must not be so specific
        !            85:    that it restricts the addition of future test cases into it.
        !            86: 
        !            87: 1. Create the test program source file using one of the templates below.
        !            88:    E.g. src/tests/tutorial/sample_test.c.
        !            89: 
        !            90: 1. Add the new test program to the Makefile.
        !            91: 
        !            92: If the subdirectory does not exist:
        !            93: 
        !            94: 1. Do the same as above.
        !            95: 
        !            96: 1. Create the Makefile for the directory using the templates below.
        !            97: 
        !            98: 1. Edit the parent Makefile to recurse into the new subdirectory.
        !            99: 
        !           100: 1. Edit src/etc/mtree/NetBSD.base.dist to register the new subdirectory.  Your
        !           101:    test will be installed under /usr/tests/.
        !           102: 
        !           103: 1. Edit src/distrib/sets/lists/tests/mi to register the new test program.  Do
        !           104:    not forget to add .debug entries if your test program is a C/C++ binary.
        !           105: 
        !           106: ### Makefile template
        !           107: 
        !           108:     # $NetBSD$
        !           109: 
        !           110:     .include <bsd.own.mk>
        !           111: 
        !           112:     # This must always be defined.
        !           113:     TESTSDIR= ${TESTSBASE}/bin/ls
        !           114: 
        !           115:     # Define only the variables you actually need for the directory.
        !           116:     TESTS_C+= c1_test c2_test  # Correspond to c1_test.c and c2_test.c.
        !           117:     TESTS_SH+= sh1_test sh2_test  # Correspond to sh1_test.c and sh2_test.c
        !           118: 
        !           119:     # Define only if your tests need any data files.
        !           120:     FILESDIR= ${TESTSDIR}
        !           121:     FILES= testdata1.txt testdata2.bin  # Any necessary data files.
        !           122: 
        !           123:     .include <bsd.test.mk>
        !           124: 
        !           125: ## C test programs
        !           126: 
        !           127: ### Template
        !           128: 
        !           129:     #include <atf-c.h>
        !           130: 
        !           131:     ATF_TC(tc, my_test_case);
        !           132:     ATF_TC_HEAD(tc, my_test_case)
        !           133:     {
        !           134:         atf_tc_set_md_var(tc, "descr", "This test case ensures that...");
        !           135:     }
        !           136:     ATF_TC_BODY(tc, my_test_case)
        !           137:     {
        !           138:         ATF_CHECK(returns_a_boolean()); /* Non-fatal test. */
        !           139:         ATF_REQUIRE(returns_a_boolean()); /* Non-fatal test. */
        !           140: 
        !           141:         ATF_CHECK_EQ(4, 2 + 2); /* Non-fatal test. */
        !           142:         ATF_REQUIRE_EQ(4, 2 + 2); /* Fatal test. */
        !           143: 
        !           144:         if (!condition)
        !           145:             atf_tc_fail("Condition not met!"); /* Explicit failure. */
        !           146:     }
        !           147: 
        !           148:     ATF_TP_ADD_TCS(tp)
        !           149:     {
        !           150:         ATF_TP_ADD_TC(tp, my_test_case);
        !           151:     }
        !           152: 
        !           153: ### How to build
        !           154: 
        !           155: To build a C test program, append the name of the test program (without the .c
        !           156: extension) to the TESTS_C variable in the Makefile.
        !           157: 
        !           158: For example:
        !           159: 
        !           160:     .include <bsd.own.mk>
        !           161: 
        !           162:     TESTSDIR= ${TESTSBASE}/bin/ls
        !           163: 
        !           164:     TESTS_C+= fs_test ui_test
        !           165: 
        !           166:     .include <bsd.test.mk>
        !           167: 
        !           168: ## Shell test programs
        !           169: 
        !           170: ### Template
        !           171: 
        !           172:     atf_test_case my_test_case
        !           173:     my_test_case_head() {
        !           174:         atf_set "descr" "This test case ensures that..."
        !           175:     }
        !           176:     my_test_case_body() {
        !           177:         touch file1 file2
        !           178: 
        !           179:         cat >expout <<EOF
        !           180:     file1
        !           181:     file2
        !           182:     EOF
        !           183:         atf_check -s eq:0 -o file:expout -e empty 'ls'
        !           184: 
        !           185:         atf_check_equal 4 $((2 + 2))
        !           186: 
        !           187:         if [ 'a' != 'b' ]; then
        !           188:             atf_fail "Condition not met!"  # Explicit failure.
        !           189:         fi
        !           190:     }
        !           191: 
        !           192:     atf_init_test_cases() {
        !           193:         atf_add_test_case my_test_case
        !           194:     }
        !           195: 
        !           196: ### How to build
        !           197: 
        !           198: To build a shell test program, append the name of the test program (without the
        !           199: .sh extension) to the TESTS_SH variable in the Makefile.
        !           200: 
        !           201: For example:
        !           202: 
        !           203:     .include <bsd.own.mk>
        !           204: 
        !           205:     TESTSDIR= ${TESTSBASE}/bin/ls
        !           206: 
        !           207:     TESTS_SH+= integration_test something_else_test
        !           208: 
        !           209:     .include <bsd.test.mk>
        !           210: 
        !           211: If you want to run the test program yourself, you should know that shell-based
        !           212: test programs are processed with the atf-sh interpreter.  atf-sh is just a thin
        !           213: wrapper over /bin/sh that loads the shared atf code and then delegates execution
        !           214: to your source file.
        !           215: 
        !           216: ## FAQ
        !           217: 
        !           218: ### How do I atfify a plain test program?
        !           219: 
        !           220: Let's suppose you have a program to exercise a particular piece of code.
        !           221: Conceptually this implements a test but it does not use atf at all.  For
        !           222: example:
        !           223: 
        !           224:     #include <err.h>
        !           225:     #include <stdio.h>
        !           226:     #include <stdlib.h>
        !           227:     #include <string.h>
        !           228: 
        !           229:     /* This test program exercises the snprintf function. */
        !           230: 
        !           231:     int main(void)
        !           232:     {
        !           233:         char buf[1024];
        !           234: 
        !           235:         printf("Testing integers");
        !           236:         snprintf(buf, sizeof(buf), "%d", 3);
        !           237:         if (strcmp(buf, "3") != 0)
        !           238:             errx(EXIT_FAILURE, "%d failed");
        !           239:         snprintf(buf, sizeof(buf), "a %d b", 5);
        !           240:         if (strcmp(buf, "a 5 b") != 0)
        !           241:             errx(EXIT_FAILURE, "%d failed");
        !           242: 
        !           243:         printf("Testing strings");
        !           244:         snprintf(buf, sizeof(buf), "%s", "foo");
        !           245:         if (strcmp(buf, "foo") != 0)
        !           246:             errx(EXIT_FAILURE, "%s failed");
        !           247:         snprintf(buf, sizeof(buf), "a %s b", "bar");
        !           248:         if (strcmp(buf, "a bar b") != 0)
        !           249:             errx(EXIT_FAILURE, "%s failed");
        !           250: 
        !           251:         return EXIT_SUCCESS;
        !           252:     }
        !           253: 
        !           254: To convert this program into an atf test program, use the template above and
        !           255: keep this in mind:
        !           256: 
        !           257: * Split the whole main function into separate test cases.  In this scenario, the
        !           258:   calls to printf(3) delimit a good granularity for the test cases: one for the
        !           259:   integer formatter, one for the string formatter, etc.
        !           260: 
        !           261: * Use the ATF_CHECK* and/or atf_tc_fail functions to do the comparisons and
        !           262:   report errors.  errx should not be used.
        !           263: 
        !           264: The result would look like:
        !           265: 
        !           266:     #include <atf-c.h>
        !           267:     #include <stdio.h>
        !           268: 
        !           269:     ATF_TC(tc, integer_formatter);
        !           270:     ATF_TC_HEAD(tc, integer_formatter)
        !           271:     {
        !           272:         atf_tc_set_md_var(tc, "descr", "Validates the %d formatter");
        !           273:     }
        !           274:     ATF_TC_BODY(tc, integer_formatter)
        !           275:     {
        !           276:         char buf[1024];
        !           277: 
        !           278:         snprintf(buf, sizeof(1024), "%d", 3);
        !           279:         ATF_CHECK_STREQ("3", buf);
        !           280: 
        !           281:         snprintf(buf, sizeof(1024), "a %d b", 5);
        !           282:         ATF_CHECK_STREQ("a 5 b", buf);
        !           283:     }
        !           284: 
        !           285:     ATF_TC(tc, string_formatter);
        !           286:     ATF_TC_HEAD(tc, string_formatter)
        !           287:     {
        !           288:         atf_tc_set_md_var(tc, "descr", "Validates the %s formatter");
        !           289:     }
        !           290:     ATF_TC_BODY(tc, string_formatter)
        !           291:     {
        !           292:         char buf[1024];
        !           293: 
        !           294:         snprintf(buf, sizeof(1024), "%s", "foo");
        !           295:         ATF_CHECK_STREQ("foo", buf);
        !           296: 
        !           297:         snprintf(buf, sizeof(1024), "a %s b", "bar");
        !           298:         ATF_CHECK_STREQ("a bar b", buf);
        !           299:     }
        !           300: 
        !           301:     ATF_TP_ADD_TCS(tp)
        !           302:     {
        !           303:         ATF_TP_ADD_TC(tp, integer_formatter);
        !           304:         ATF_TP_ADD_TC(tp, string_formatter);
        !           305:     }
        !           306: 
        !           307: Which can later be invoked as any of:
        !           308: 
        !           309:     $ ./snprintf_test integer_formatter
        !           310:     $ ./snprintf_test string_formatter
        !           311:     $ atf-run snprintf_test | atf-report
        !           312: 
        !           313: ### Do I need to remove temporary files?
        !           314: 
        !           315: No.  atf-run does this automatically for you, because it runs every test program
        !           316: in its own temporary subdirectory.

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb