1*c85f09ccSJohn LevonTest suite 2*c85f09ccSJohn Levon########## 3*c85f09ccSJohn Levon 4*c85f09ccSJohn LevonSparse has a number of test cases in its validation directory. The test-suite 5*c85f09ccSJohn Levonscript aims at making automated checking of these tests possible. It works by 6*c85f09ccSJohn Levonembedding tags in C comments in the test cases. 7*c85f09ccSJohn Levon 8*c85f09ccSJohn LevonTag's syntax 9*c85f09ccSJohn Levon============ 10*c85f09ccSJohn Levon 11*c85f09ccSJohn Levon``check-name:`` *name* 12*c85f09ccSJohn Levon 13*c85f09ccSJohn Levon Name of the test. This is the only mandatory tag. 14*c85f09ccSJohn Levon 15*c85f09ccSJohn Levon``check-description:`` *description ...* 16*c85f09ccSJohn Levon 17*c85f09ccSJohn Levon A description of what the test checks. 18*c85f09ccSJohn Levon 19*c85f09ccSJohn Levon``check-command:`` *command arg ...* 20*c85f09ccSJohn Levon 21*c85f09ccSJohn Levon There are different kinds of tests. Some can validate the sparse 22*c85f09ccSJohn Levon preprocessor, while others will use sparse, cgcc, or even other backends 23*c85f09ccSJohn Levon of the library. check-command allows you to give a custom command to 24*c85f09ccSJohn Levon run the test-case. 25*c85f09ccSJohn Levon The ``$file`` string is special. It will be expanded to the file name at 26*c85f09ccSJohn Levon run time. 27*c85f09ccSJohn Levon It defaults to ``sparse $file``. 28*c85f09ccSJohn Levon 29*c85f09ccSJohn Levon``check-arch-ignore:`` *arch[|...]* 30*c85f09ccSJohn Levon 31*c85f09ccSJohn Levon``check-arch-only:`` *arch[|...]* 32*c85f09ccSJohn Levon 33*c85f09ccSJohn Levon Ignore the test if the current architecture (as returned by ``uname -m``) 34*c85f09ccSJohn Levon matches or not one of the archs given in the pattern. 35*c85f09ccSJohn Levon 36*c85f09ccSJohn Levon``check-assert:`` *condition* 37*c85f09ccSJohn Levon 38*c85f09ccSJohn Levon Ignore the test if the given condition is false when evaluated as a 39*c85f09ccSJohn Levon static assertion (``_Static_assert``). 40*c85f09ccSJohn Levon 41*c85f09ccSJohn Levon``check-cpp-if:`` *condition* 42*c85f09ccSJohn Levon 43*c85f09ccSJohn Levon Ignore the test if the given condition is false when evaluated 44*c85f09ccSJohn Levon by sparse's pre-processor. 45*c85f09ccSJohn Levon 46*c85f09ccSJohn Levon``check-exit-value:`` *value* 47*c85f09ccSJohn Levon 48*c85f09ccSJohn Levon The expected exit value of check-command. It defaults to 0. 49*c85f09ccSJohn Levon 50*c85f09ccSJohn Levon``check-timeout:`` *timeout* 51*c85f09ccSJohn Levon 52*c85f09ccSJohn Levon The maximum expected duration of check-command, in seconds. 53*c85f09ccSJohn Levon It defaults to 1. 54*c85f09ccSJohn Levon 55*c85f09ccSJohn Levon``check-output-start`` / ``check-output-end`` 56*c85f09ccSJohn Levon 57*c85f09ccSJohn Levon The expected output (stdout and stderr) of check-command lies between 58*c85f09ccSJohn Levon those two tags. It defaults to no output. 59*c85f09ccSJohn Levon 60*c85f09ccSJohn Levon``check-output-ignore`` / ``check-error-ignore`` 61*c85f09ccSJohn Levon 62*c85f09ccSJohn Levon Don't check the expected output (stdout or stderr) of check-command 63*c85f09ccSJohn Levon (useful when this output is not comparable or if you're only interested 64*c85f09ccSJohn Levon in the exit value). By default this check is done. 65*c85f09ccSJohn Levon 66*c85f09ccSJohn Levon``check-known-to-fail`` 67*c85f09ccSJohn Levon 68*c85f09ccSJohn Levon Mark the test as being known to fail. 69*c85f09ccSJohn Levon 70*c85f09ccSJohn Levon``check-output-contains:`` *pattern* 71*c85f09ccSJohn Levon 72*c85f09ccSJohn Levon Check that the output (stdout) contains the given pattern. 73*c85f09ccSJohn Levon Several such tags can be given, in which case the output 74*c85f09ccSJohn Levon must contains all the patterns. 75*c85f09ccSJohn Levon 76*c85f09ccSJohn Levon``check-output-excludes:`` *pattern* 77*c85f09ccSJohn Levon 78*c85f09ccSJohn Levon Similar than the above one, but with opposite logic. 79*c85f09ccSJohn Levon Check that the output (stdout) doesn't contain the given pattern. 80*c85f09ccSJohn Levon Several such tags can be given, in which case the output 81*c85f09ccSJohn Levon must contains none of the patterns. 82*c85f09ccSJohn Levon 83*c85f09ccSJohn Levon``check-output-pattern(``\ *nbr*\ ``):`` *pattern* 84*c85f09ccSJohn Levon 85*c85f09ccSJohn Levon``check-output-pattern(``\ *min*\ ``,``\ *max*\ ``):`` *pattern* 86*c85f09ccSJohn Levon 87*c85f09ccSJohn Levon Similar to the contains/excludes above, but with full control 88*c85f09ccSJohn Levon of the number of times the pattern should occur in the output. 89*c85f09ccSJohn Levon If *min* or *max* is ``-`` the corresponding check is ignored. 90*c85f09ccSJohn Levon 91*c85f09ccSJohn LevonUsing test-suite 92*c85f09ccSJohn Levon================ 93*c85f09ccSJohn Levon 94*c85f09ccSJohn LevonThe test-suite script is called through the check target of the Makefile. It 95*c85f09ccSJohn Levonwill try to check every test case it finds (``find validation -name '*.c'``). 96*c85f09ccSJohn LevonIt can be called to check a single test with:: 97*c85f09ccSJohn Levon 98*c85f09ccSJohn Levon $ cd validation 99*c85f09ccSJohn Levon $ ./test-suite single preprocessor/preprocessor1.c 100*c85f09ccSJohn Levon TEST Preprocessor #1 (preprocessor/preprocessor1.c) 101*c85f09ccSJohn Levon preprocessor/preprocessor1.c passed ! 102*c85f09ccSJohn Levon 103*c85f09ccSJohn Levon 104*c85f09ccSJohn LevonWriting a test 105*c85f09ccSJohn Levon============== 106*c85f09ccSJohn Levon 107*c85f09ccSJohn LevonThe test-suite comes with a format command to make a test easier to write:: 108*c85f09ccSJohn Levon 109*c85f09ccSJohn Levon test-suite format [-a] [-l] [-f] file [name [cmd]] 110*c85f09ccSJohn Levon 111*c85f09ccSJohn Levon`name:` check-name value 112*c85f09ccSJohn Levon If no name is provided, it defaults to the file name. 113*c85f09ccSJohn Levon 114*c85f09ccSJohn Levon`cmd:` check-command value 115*c85f09ccSJohn Levon If no cmd is provided, it defaults to ``sparse $file``. 116*c85f09ccSJohn Levon 117*c85f09ccSJohn LevonThe output of the test-suite format command can be redirected into the 118*c85f09ccSJohn Levontest case to create a test-suite formatted file.:: 119*c85f09ccSJohn Levon 120*c85f09ccSJohn Levon $ ./test-suite format bad-assignment.c Assignment >> bad-assignment.c 121*c85f09ccSJohn Levon $ cat !$ 122*c85f09ccSJohn Levon cat bad-assignment.c 123*c85f09ccSJohn Levon /* 124*c85f09ccSJohn Levon * check-name: bad assignment 125*c85f09ccSJohn Levon * 126*c85f09ccSJohn Levon * check-command: sparse $file 127*c85f09ccSJohn Levon * check-exit-value: 1 128*c85f09ccSJohn Levon * 129*c85f09ccSJohn Levon * check-output-start 130*c85f09ccSJohn Levon bad-assignment.c:3:6: error: Expected ; at end of statement 131*c85f09ccSJohn Levon bad-assignment.c:3:6: error: got \ 132*c85f09ccSJohn Levon * check-output-end 133*c85f09ccSJohn Levon */ 134*c85f09ccSJohn Levon 135*c85f09ccSJohn LevonThe same effect without the redirection can be achieved by using the ``-a`` 136*c85f09ccSJohn Levonoption. 137*c85f09ccSJohn Levon 138*c85f09ccSJohn LevonYou can define the check-command you want to use for the test.:: 139*c85f09ccSJohn Levon 140*c85f09ccSJohn Levon $ ./test-suite format -a validation/preprocessor2.c "Preprocessor #2" \ 141*c85f09ccSJohn Levon "sparse -E \$file" 142*c85f09ccSJohn Levon $ cat !$ 143*c85f09ccSJohn Levon cat validation/preprocessor2.c 144*c85f09ccSJohn Levon /* 145*c85f09ccSJohn Levon * This one we happen to get right. 146*c85f09ccSJohn Levon * 147*c85f09ccSJohn Levon * It should result in a simple 148*c85f09ccSJohn Levon * 149*c85f09ccSJohn Levon * a + b 150*c85f09ccSJohn Levon * 151*c85f09ccSJohn Levon * for a proper preprocessor. 152*c85f09ccSJohn Levon */ 153*c85f09ccSJohn Levon #define TWO a, b 154*c85f09ccSJohn Levon 155*c85f09ccSJohn Levon #define UNARY(x) BINARY(x) 156*c85f09ccSJohn Levon #define BINARY(x, y) x + y 157*c85f09ccSJohn Levon 158*c85f09ccSJohn Levon UNARY(TWO) 159*c85f09ccSJohn Levon /* 160*c85f09ccSJohn Levon * check-name: Preprocessor #2 161*c85f09ccSJohn Levon * 162*c85f09ccSJohn Levon * check-command: sparse -E $file 163*c85f09ccSJohn Levon * check-exit-value: 0 164*c85f09ccSJohn Levon * 165*c85f09ccSJohn Levon * check-output-start 166*c85f09ccSJohn Levon 167*c85f09ccSJohn Levon a + b 168*c85f09ccSJohn Levon * check-output-end 169*c85f09ccSJohn Levon */ 170