test/py/README.md

1 # U-Boot pytest suite
5 This tool aims to test U-Boot by executing U-Boot shell commands using the
6 console interface. A single top-level script exists to execute or attach to the
7 U-Boot console, run the entire script of tests against it, and summarize the
10 - Testing is performed in the same way a user or script would interact with
11   U-Boot; there can be no disconnect.
12 - There is no need to write or embed test-related code into U-Boot itself.
13   It is asserted that writing test-related code in Python is simpler and more
15 - It is reasonably simple to interact with U-Boot in this way.
19 The test suite is implemented using pytest. Interaction with the U-Boot console
24 On Debian or Debian-like distributions, the following packages are required.
28 | -------------- | ----------------------------- |
29 | python         | 2.7.5-5ubuntu3                |
30 | python-pytest  | 2.5.1-1                       |
34 - Executing a sandbox port of U-Boot on the local machine as a sub-process,
36 - Executing an external "hook" scripts to flash a U-Boot binary onto a
44 can use the Python `virtualenv` script to locally install more up-to-date
49 $ cd /path/to/u-boot
50 $ sudo apt-get install python python-virtualenv
58 To run the testsuite on the sandbox port (U-Boot built as a native user-space
62 ./test/py/test.py --bd sandbox --build
65 The `--bd` option tells the test suite which board type is being tested. This
69 The `--build` option tells U-Boot to compile U-Boot. Alternatively, you may
70 omit this option and build U-Boot yourself, in whatever way you choose, before
73 The test script will attach to U-Boot, execute all valid tests for the board,
75 will be written to `${build_dir}/test-log.html`. This is best viewed in a web
81 If you need to run sandbox under a debugger, you may pass the command-line
82 option `--gdbserver COMM`. This causes two things to happens:
84 - Instead of running U-Boot directly, it will be run under gdbserver, with
88 - All timeouts in tests are disabled, allowing U-Boot an arbitrary amount of
89   time to execute commands. This is useful if U-Boot is stopped at a breakpoint
96 ./test/py/test.py --bd sandbox --gdbserver localhost:1234
101 gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234'
104 Alternatively, you could leave off the `-ex` option and type the command
111 reset command, or sandbox's CTRL-C handling. When this happens, you will need
113 relevant to your debugging session, you can skip them using pytest's -k
114 command-line option; see the next section.
116 ## Command-line options
118 - `--board-type`, `--bd`, `-B` set the type of the board to be tested. For
120 - `--board-identity`, `--id` set the identity of the board to be tested.
124   to the hook scripts described below, and may be used in any site-specific
126 - `--build` indicates that the test script should compile U-Boot itself
130 - `--build-dir` sets the directory containing the compiled U-Boot binaries.
131   If omitted, this is `${source_dir}/build-${board_type}`.
132 - `--result-dir` sets the directory to write results, such as log files,
134 - `--persistent-data-dir` sets the directory used to store persistent test
135   data. This is test data that may be re-used across test runs, such as file-
138 `pytest` also implements a number of its own command-line options. Commonly used
140 details. Execute `py.test --version` for a brief summary. Note that U-Boot's
141 test.py script passes all command-line arguments directly to `pytest` for
144 - `-k` selects which tests to run. The default is to run all known tests. This
147   - `'ums'` runs only tests with "ums" in their name.
148   - ``ut_dm'` runs only tests with "ut_dm" in their name. Note that in this
151   - `'not reset'` runs everything except tests with "reset" in their name.
152   - `'ut or hush'` runs only tests with "ut" or "hush" in their name.
153   - `'not (ut or hush)'` runs everything except tests with "ut" or "hush" in
155 - `-s` prevents pytest from hiding a test's stdout. This allows you to see
156   U-Boot's console log in real time on pytest's stdout.
166 unrelated to U-Boot features.
174 - `UBOOT_BOARD_TYPE` the board type being tested.
175 - `UBOOT_BOARD_IDENTITY` the board identity being tested, or `na` if none was
177 - `UBOOT_SOURCE_DIR` the U-Boot source directory.
178 - `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory.
179 - `UBOOT_BUILD_DIR` the U-Boot build directory.
180 - `UBOOT_RESULT_DIR` the test result directory.
181 - `UBOOT_PERSISTENT_DATA_DIR` the test peristent data directory.
183 #### `u-boot-test-console`
185 This script provides access to the U-Boot console. The script's stdin/stdout
192 If you are able to run U-Boot under a hardware simulator such as qemu, then
194 `u-boot-test-reset` may be called multiple times per test script run, and must
195 cause U-Boot to start execution from scratch each time. Hopefully your
197 simulator from `u-boot-test-reset` instead, while arranging for this console
200 #### `u-boot-test-flash`
203 so that the board executes the particular U-Boot binary to be tested. Often,
204 this involves writing the U-Boot binary to the board's flash ROM. The test
209 cause the newly flashed U-Boot binary to be executed.
214 - Some other process has already written the desired U-Boot binary into the
216 - The board allows U-Boot to be downloaded directly into RAM, and executed
218   may be preferable if available, and if cold boot testing of U-Boot is not
219   required. If this feature is used, the `u-boot-test-reset` script should
226 This script will typically be implemented by calling out to some SoC- or
227 board-specific vendor flashing utility.
229 #### `u-boot-test-reset`
233 first test function. If any test fails, the test infra-structure will execute
234 this script again to restore U-Boot to an operational state before running the
240 The semantics of this script require that when it is executed, U-Boot will
241 start running from scratch. If the U-Boot binary to be tested has been written
244 example, it may call out to some SoC- or board-specific vendor utility in order
245 to download the U-Boot binary directly into RAM and execute it. This would
246 avoid the need for `u-boot-test-flash` to actually write U-Boot to flash, thus
251 https://github.com/swarren/uboot-test-hooks contains some working example hook
253 your platform. These scripts are not considered part of U-Boot itself.
255 ### Board-type-specific configuration
262 to provide board-specific information to the test script. Any global value
264 contained in these scripts must be purely derived from U-Boot source code.
265 Hence, these configuration files are part of the U-Boot source tree too.
270 implemented by a particular board's configuration of U-Boot. For example, a
271 U-Boot configuration may support USB device mode and USB Mass Storage, but this
277 board-specific and board-identity-specific information to the test script. Any
281 U-Boot source tree, and should be installed outside of the source tree. Users
290 - None at present.
292 ### U-Boot `.config` feature usage
294 The test scripts rely on various U-Boot `.config` features, either directly in
296 the running U-Boot instance in order to test other features.
305 - `buildconfig.get(...`
306 - `@pytest.mark.buildconfigspec(...`
314 If U-Boot has already been built:
319     ./test/py/test.py --bd seaboard
322 If you want the test script to compile U-Boot for you too, then you likely
327 CROSS_COMPILE=arm-none-eabi- \
330     ./test/py/test.py --bd seaboard --build
336 Details specific to the U-Boot test suite are described below.
339 provides the means to interact with the U-Boot console, and retrieve board and
343 U-Boot console, and returns all output from that command. This allows
345 that certain strings are not seen on the U-Boot console. These include shell
346 error messages and the U-Boot sign-on message (in order to detect unexpected
352 Board- and board-environment configuration values may be accessed as sub-fields