test/py/README.md

5 This tool aims to test U-Boot by executing U-Boot shell commands using the
12 - There is no need to write or embed test-related code into U-Boot itself.
13   It is asserted that writing test-related code in Python is simpler and more
19 The test suite is implemented using pytest. Interaction with the U-Boot console
21 need to implement various "hook" scripts that are called by the test suite at
32 The test script supports either:
43 packages, or may provide versions that are too old to run the test suite. One
62 ./test/py/test.py --bd sandbox --build
65 The `--bd` option tells the test suite which board type is being tested. This
66 lets the test suite know which features the board has, and hence exactly what
71 running the test script.
73 The test script will attach to U-Boot, execute all valid tests for the board,
74 then print a summary of the test process. A complete log of the test session
75 will be written to `${build_dir}/test-log.html`. This is best viewed in a web
96 ./test/py/test.py --bd sandbox --gdbserver localhost:1234
110 Some tests deliberately cause the sandbox process to exit, e.g. to test the
123   not interpreted by the test script in any way, but rather is simply passed
126 - `--build` indicates that the test script should compile U-Boot itself
134 - `--persistent-data-dir` sets the directory used to store persistent test
135   data. This is test data that may be re-used across test runs, such as file-
140 details. Execute `py.test --version` for a brief summary. Note that U-Boot's
141 test.py script passes all command-line arguments directly to `pytest` for
145   option takes a single argument which is used to filter test names. Simple
149     case, "ut_dm" is a parameter to a test rather than the test name. The full
150     test name is e.g. "test_ut[ut_dm_leak]".
155 - `-s` prevents pytest from hiding a test's stdout. This allows you to see
162 For this reason, the test suite does not attempt to directly interact with real
164 via `$PATH`. These scripts implement certain actions on behalf of the test
165 suite. This keeps the test suite simple and isolated from system variances
178 - `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory.
180 - `UBOOT_RESULT_DIR` the test result directory.
181 - `UBOOT_PERSISTENT_DATA_DIR` the test peristent data directory.
183 #### `u-boot-test-console`
187 indefinitely, until killed. The test suite will run this script in parallel
194 `u-boot-test-reset` may be called multiple times per test script run, and must
197 simulator from `u-boot-test-reset` instead, while arranging for this console
200 #### `u-boot-test-flash`
202 Prior to running the test suite against a board, some arrangement must be made
204 this involves writing the U-Boot binary to the board's flash ROM. The test
215   board's flash prior to running the test suite.
219   required. If this feature is used, the `u-boot-test-reset` script should
221   times in a single test run.
229 #### `u-boot-test-reset`
231 Whenever the test suite needs to reset the target board, this script is
233 first test function. If any test fails, the test infra-structure will execute
235 next test function.
246 avoid the need for `u-boot-test-flash` to actually write U-Boot to flash, thus
251 https://github.com/swarren/uboot-test-hooks contains some working example hook
262 to provide board-specific information to the test script. Any global value
263 defined in these modules is available for use by any test function. The data
273 machine running the test script.
277 board-specific and board-identity-specific information to the test script. Any
278 global value defined in these modules is available for use by any test
282 should set `$PYTHONPATH` prior to running the test script to allow these
287 The test scripts rely on the following variables being defined by the board
294 The test scripts rely on various U-Boot `.config` features, either directly in
295 order to test those features, or indirectly in order to query information from
296 the running U-Boot instance in order to test other features.
299 address to use for the test. This data is parsed from the output of the
302 For a complete list of dependencies, please search the test scripts for
312 then you would likely invoke the test script as follows:
319     ./test/py/test.py --bd seaboard
322 If you want the test script to compile U-Boot for you too, then you likely
323 need to set `$CROSS_COMPILE` to allow this, and invoke the test script as
330     ./test/py/test.py --bd seaboard --build
336 Details specific to the U-Boot test suite are described below.
338 A test fixture named `u_boot_console` should be used by each test function. This
348 "bad" strings. Some test scenarios are expected to trigger these strings. Use