test/py/README.md

5 This tool aims to test U-Boot by executing U-Boot shell commands using the
6 console interface. A single top-level script exists to execute or attach to the
12 - There is no need to write or embed test-related code into U-Boot itself.
15 - It is reasonably simple to interact with U-Boot in this way.
21 need to implement various "hook" scripts that are called by the test suite at
36 - Executing an external "hook" scripts to flash a U-Boot binary onto a
37   physical board, attach to the board's console stream, and reset the board.
40 ### Using `virtualenv` to provide requirements
43 packages, or may provide versions that are too old to run the test suite. One
44 can use the Python `virtualenv` script to locally install more up-to-date
49 $ cd /path/to/u-boot
69 The `--build` option tells U-Boot to compile U-Boot. Alternatively, you may
73 The test script will attach to U-Boot, execute all valid tests for the board,
75 will be written to `${build_dir}/test-log.html`. This is best viewed in a web
81 If you need to run sandbox under a debugger, you may pass the command-line
82 option `--gdbserver COMM`. This causes two things to happens:
85   debug communication via the channel `COMM`. You can attach a debugger to the
86   sandbox process in order to debug it. See `man gdbserver` and the example
89   time to execute commands. This is useful if U-Boot is stopped at a breakpoint
110 Some tests deliberately cause the sandbox process to exit, e.g. to test the
112 to attach the debugger to the new sandbox instance. If these tests are not
113 relevant to your debugging session, you can skip them using pytest's -k
118 - `--board-type`, `--bd`, `-B` set the type of the board to be tested. For
120 - `--board-identity`, `--id` set the identity of the board to be tested.
122   physical board that are attached to the same host machine. This parameter is
124   to the hook scripts described below, and may be used in any site-specific
132 - `--result-dir` sets the directory to write results, such as log files,
134 - `--persistent-data-dir` sets the directory used to store persistent test
141 test.py script passes all command-line arguments directly to `pytest` for
144 - `-k` selects which tests to run. The default is to run all known tests. This
145   option takes a single argument which is used to filter test names. Simple
149     case, "ut_dm" is a parameter to a test rather than the test name. The full
155 - `-s` prevents pytest from hiding a test's stdout. This allows you to see
160 The tools and techniques used to interact with real hardware will vary
162 For this reason, the test suite does not attempt to directly interact with real
166 unrelated to U-Boot features.
178 - `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory.
185 This script provides access to the U-Boot console. The script's stdin/stdout
186 should be connected to the board's console. This process should continue to run
192 If you are able to run U-Boot under a hardware simulator such as qemu, then
195 cause U-Boot to start execution from scratch each time. Hopefully your
198 process to always communicate with the current simulator instance.
202 Prior to running the test suite against a board, some arrangement must be made
203 so that the board executes the particular U-Boot binary to be tested. Often,
204 this involves writing the U-Boot binary to the board's flash ROM. The test
209 cause the newly flashed U-Boot binary to be executed.
215   board's flash prior to running the test suite.
216 - The board allows U-Boot to be downloaded directly into RAM, and executed
223 It is up to the user to determine if those situations exist, and to code this
226 This script will typically be implemented by calling out to some SoC- or
231 Whenever the test suite needs to reset the target board, this script is
232 executed. This is guaranteed to happen at least once, prior to executing the
234 this script again to restore U-Boot to an operational state before running the
238 relay or electronic switch attached to the board's reset signal.
241 start running from scratch. If the U-Boot binary to be tested has been written
242 to flash, pulsing the board's reset signal is likely all this script need do.
244 example, it may call out to some SoC- or board-specific vendor utility in order
245 to download the U-Boot binary directly into RAM and execute it. This would
246 avoid the need for `u-boot-test-flash` to actually write U-Boot to flash, thus
262 to provide board-specific information to the test script. Any global value
276 `u_boot_boardenv_${board_type}_${board_identity}` may exist to provide
277 board-specific and board-identity-specific information to the test script. Any
279 function. The data contained in these is specific to a particular user's
282 should set `$PYTHONPATH` prior to running the test script to allow these
283 modules to be loaded.
295 order to test those features, or indirectly in order to query information from
296 the running U-Boot instance in order to test other features.
299 address to use for the test. This data is parsed from the output of the
322 If you want the test script to compile U-Boot for you too, then you likely
323 need to set `$CROSS_COMPILE` to allow this, and invoke the test script as
335 Please refer to the pytest documentation for details of writing pytest tests.
336 Details specific to the U-Boot test suite are described below.
339 provides the means to interact with the U-Boot console, and retrieve board and
346 error messages and the U-Boot sign-on message (in order to detect unexpected
348 "bad" strings. Some test scenarios are expected to trigger these strings. Use
349 `u_boot_console.disable_check()` to temporarily disable checking for specific
357 `u_boot_console.config.buildconfig`, with keys equal to the Kconfig variable