1Writing a Test
2==============
3
4Writing a test program is pretty easy.  Basically, a test is configured via
5a monster array in the Run script, which specifics (among other things) the
6program to execute and the parameters to pass it.
7
8The test itself is simply a program which is given the optional parameters
9on the command line, and produces logging data on stdout and its results on
10stderr.
11
12
13============================================================================
14
15Test Configuration
16==================
17
18In Run, all tests are named in the "$testList" array.  This names the
19individual tests, and also sets up aliases for groups of tests, eg. "index".
20
21The test specifications are in the "$testParams" array.  This contains the
22details of each individual test as a hash.  The fields in the hash are:
23
24 * "logmsg": the full name to display for this test.
25 * "cat": the category this test belongs to; must be configured
26   in $testCats.
27 * "prog": the name of the program to execute; defaults to the name of
28   the benchmark.
29 * "repeat": number of passes to run; either 'short' (the default),
30   'long', or 'single'.   For 'short' and 'long', the actual numbers of
31   passes are given by $shortIterCount and $longIterCount, which are
32   configured at the top of the script or by the "-i" flag.  'single'
33   means just run one pass; this should be used for test which do their
34   own multi-pass handling internally.
35 * "stdout": non-0 to add the test's stdout to the log file; defaults to 1.
36   Set to 0 for tests that are too wordy.
37 * "stdin": name of a file to send to the program's stdin; default null.
38 * "options": options to be put on the program's command line; default null.
39
40
41============================================================================
42
43Output Format
44=============
45
46The results on stderr take the form of a line header and fields, separated
47by "|" characters.  A result line can be one of:
48
49    COUNT|score|timebase|label
50    TIME|seconds
51    ERROR|message
52
53Any other text on stderr is treated as if it were:
54
55    ERROR|text
56
57Any output to stdout is placed in a log file, and can be used for debugging.
58
59COUNT
60-----
61
62The COUNT line is the line used to report a test score.
63
64 * "score" is the result, typically the number of loops performed during
65   the run
66 * "timebase" is the time base used for the final report to the user.  A
67   value of 1 reports the score as is; a value of 60, for example, divides
68   the time taken by 60 to get loops per minute.  Atimebase of zero indicates
69   that the score is already a rate, ie. a count of things per second.
70 * "label" is the label to use for the score; like "lps" (loops per
71   second), etc.
72
73TIME
74----
75
76The TIME line is optionally used to report the time taken.  The Run script
77normally measures this, but if your test has signifant overhead outside the
78actual test loop, you should use TIME to report the time taken for the actual
79test.  The argument is the time in seconds in floating-point.
80
81ERROR
82-----
83
84The argument is an error message; this will abort the benchmarking run and
85display the message.
86
87Any output to stderr which is not a formatted line will be treated as an
88error message, so use of ERROR is optional.
89
90
91============================================================================
92
93Test Examples
94=============
95
96Iteration Count
97---------------
98
99The simplest thing is to count the number of loops executed in a given time;
100see eg. arith.c.  The utlilty functions in timeit.c can be used to implement
101the fixed time interval, which is generally passed in on the command line.
102
103The result is reported simply as the number of iterations completed:
104
105        fprintf(stderr,"COUNT|%lu|1|lps\n", iterations);
106
107The bnenchmark framework will measure the time taken itself.  If the test
108code has significant overhead (eg. a "pump-priming" pass), then you should
109explicitly report the time taken for the test by adding a line like this:
110
111        fprintf(stderr, "TIME|%.1f\n", seconds);
112
113If you want results reported as loops per minute, then set timebase to 60:
114
115        fprintf(stderr,"COUNT|%lu|60|lpm\n", iterations);
116
117Note that this only affects the final report; all times passed to or
118from the test are still in seconds.
119
120Rate
121----
122
123The other technique is to calculate the rate (things per second) in the test,
124and report that directly.  To do this, just set timebase to 0:
125
126        fprintf(stderr, "COUNT|%ld|0|KBps\n", kbytes_per_sec);
127
128Again, you can use TIME to explicitly report the time taken:
129
130        fprintf(stderr, "TIME|%.1f\n", end - start);
131
132but this isn't so important since you've already calculated the rate.
133
134