1Test suite
2##########
3
4Sparse has a number of test cases in its validation directory. The test-suite
5script aims at making automated checking of these tests possible. It works by
6embedding tags in C comments in the test cases.
7
8Tag's syntax
9============
10
11``check-name:`` *name*
12
13	Name of the test. This is the only mandatory tag.
14
15``check-description:`` *description ...*
16
17	A description of what the test checks.
18
19``check-command:`` *command arg ...*
20
21	There are different kinds of tests. Some can validate the sparse
22	preprocessor, while others will use sparse, cgcc, or even other backends
23	of the library. check-command allows you to give a custom command to
24	run the test-case.
25	The ``$file`` string is special. It will be expanded to the file name at
26	run time.
27	It defaults to ``sparse $file``.
28
29``check-arch-ignore:`` *arch[|...]*
30
31``check-arch-only:`` *arch[|...]*
32
33	Ignore the test if the current architecture (as returned by ``uname -m``)
34	matches or not one of the archs given in the pattern.
35
36``check-assert:`` *condition*
37
38	Ignore the test if the given condition is false when evaluated as a
39	static assertion (``_Static_assert``).
40
41``check-cpp-if:`` *condition*
42
43	Ignore the test if the given condition is false when evaluated
44	by sparse's pre-processor.
45
46``check-exit-value:`` *value*
47
48	The expected exit value of check-command. It defaults to 0.
49
50``check-timeout:`` *timeout*
51
52	The maximum expected duration of check-command, in seconds.
53	It defaults to 1.
54
55``check-output-start`` / ``check-output-end``
56
57	The expected output (stdout and stderr) of check-command lies between
58	those two tags. It defaults to no output.
59
60``check-output-ignore`` / ``check-error-ignore``
61
62	Don't check the expected output (stdout or stderr) of check-command
63	(useful when this output is not comparable or if you're only interested
64	in the exit value).  By default this check is done.
65
66``check-known-to-fail``
67
68	Mark the test as being known to fail.
69
70``check-output-contains:`` *pattern*
71
72	Check that the output (stdout) contains the given pattern.
73	Several such tags can be given, in which case the output
74	must contains all the patterns.
75
76``check-output-excludes:`` *pattern*
77
78	Similar than the above one, but with opposite logic.
79	Check that the output (stdout) doesn't contain the given pattern.
80	Several such tags can be given, in which case the output
81	must contains none of the patterns.
82
83``check-output-pattern(``\ *nbr*\ ``):`` *pattern*
84
85``check-output-pattern(``\ *min*\ ``,``\ *max*\ ``):`` *pattern*
86
87	Similar to the contains/excludes above, but with full control
88	of the number of times the pattern should occur in the output.
89	If *min* or *max* is ``-`` the corresponding check is ignored.
90
91Using test-suite
92================
93
94The test-suite script is called through the check target of the Makefile. It
95will try to check every test case it finds (``find validation -name '*.c'``).
96It can be called to check a single test with::
97
98	$ cd validation
99	$ ./test-suite single preprocessor/preprocessor1.c
100	     TEST     Preprocessor #1 (preprocessor/preprocessor1.c)
101	preprocessor/preprocessor1.c passed !
102
103
104Writing a test
105==============
106
107The test-suite comes with a format command to make a test easier to write::
108
109	test-suite format [-a] [-l] [-f] file [name [cmd]]
110
111`name:`  check-name value
112	If no name is provided, it defaults to the file name.
113
114`cmd:`   check-command value
115	If no cmd is provided, it defaults to ``sparse $file``.
116
117The output of the test-suite format command can be redirected into the
118test case to create a test-suite formatted file.::
119
120	$ ./test-suite format bad-assignment.c Assignment >> bad-assignment.c
121	$ cat !$
122	cat bad-assignment.c
123	/*
124	 * check-name: bad assignment
125	 *
126	 * check-command: sparse $file
127	 * check-exit-value: 1
128	 *
129	 * check-output-start
130	bad-assignment.c:3:6: error: Expected ; at end of statement
131	bad-assignment.c:3:6: error: got \
132	 * check-output-end
133	 */
134
135The same effect without the redirection can be achieved by using the ``-a``
136option.
137
138You can define the check-command you want to use for the test.::
139
140	$ ./test-suite format -a validation/preprocessor2.c "Preprocessor #2" \
141			"sparse -E \$file"
142	$ cat !$
143	cat validation/preprocessor2.c
144	/*
145	 * This one we happen to get right.
146	 *
147	 * It should result in a simple
148	 *
149	 *	a + b
150	 *
151	 * for a proper preprocessor.
152	 */
153	#define TWO a, b
154
155	#define UNARY(x) BINARY(x)
156	#define BINARY(x, y) x + y
157
158	UNARY(TWO)
159	/*
160	 * check-name: Preprocessor #2
161	 *
162	 * check-command: sparse -E $file
163	 * check-exit-value: 0
164	 *
165	 * check-output-start
166
167	a + b
168	 * check-output-end
169	 */
170