UPSTREAM: test/py: add timestamps to log

It can be useful to record how long tests take; this can help debug slow
running test systems or track changes in performance over time. Enhance
the test sys

UPSTREAM: test/py: add timestamps to log

It can be useful to record how long tests take; this can help debug slow
running test systems or track changes in performance over time. Enhance
the test system to record timestamps while running test:
- Whenever a new log file section is started.
- After U-Boot is started and communication has been established.
- After each host or U-Boot command is executed.

Change-Id: I1db02bfc1d69c8d6a4c390a778ae433bbd858a9d
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Signed-off-by: Kever Yang <kever.yang@rock-chips.com>
(cherry picked from commit 9679d339ad2d4c495d734bad3a0fb7be6c4215eb)

show more ...

UPSTREAM: test/py: u_boot_console_base.py: fix typo

run_command does not have a parameter wait_for_each.

Change-Id: I5b6bb3ed1b732c3ebc307f0618282caea9190b83
Signed-off-by: Heinrich Schuchardt <xyp

UPSTREAM: test/py: u_boot_console_base.py: fix typo

run_command does not have a parameter wait_for_each.

Change-Id: I5b6bb3ed1b732c3ebc307f0618282caea9190b83
Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
Signed-off-by: Kever Yang <kever.yang@rock-chips.com>
(cherry picked from commit cd3e8a72a6fb1fe758c5f73b394527609afe26cc)

show more ...

test/py: match prompt only at line boundaries

This prevents capture of command output from terminating early on boards
that use a simple prompt (e.g. "=> ") that appears in the middle of
command out

test/py: match prompt only at line boundaries

This prevents capture of command output from terminating early on boards
that use a simple prompt (e.g. "=> ") that appears in the middle of
command output (e.g. crc32's "... ==> 2fa737e0").

Reported-by: Tom Rini <trini@konsulko.com>
Signed-off-by: Stephen Warren <swarren@wwwdotorg.org>
Tested-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Heiko Schocher <hs@denx.de>

show more ...

test: Adjust run_command_list() to return a list of strings

Return one string for each command that was executed. This seems cleaner.

Suggested-by: Teddy Reed <teddy.reed@gmail.com>
Signed-off-by:

test: Adjust run_command_list() to return a list of strings

Return one string for each command that was executed. This seems cleaner.

Suggested-by: Teddy Reed <teddy.reed@gmail.com>
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Stephen Warren <swarren@nvidia.com>

show more ...

test: Add a function to restart U-Boot

Add a proper function for this rather than using internal functions. Use it
in the single call site.

Also, do a restart at the end of the vboot test to reset

test: Add a function to restart U-Boot

Add a proper function for this rather than using internal functions. Use it
in the single call site.

Also, do a restart at the end of the vboot test to reset to the normal
device tree.

Signed-off-by: Simon Glass <sjg@chromium.org>
Suggested-by: Stephen Warren <swarren@nvidia.com>

show more ...

test: Fix typos in comments

Fix some typos in various files introduced with the vboot test conversion.

Reported-by: Teddy Reed <teddy.reed@gmail.com>

Signed-off-by: Simon Glass <sjg@chromium.org>

Merge git://git.denx.de/u-boot-dm

test/py: Provide a way to get early console output

Some tests want to check the console output from SPL or U-Boot proper.
Provide a means to do this.

Signed-off-by: Simon Glass <sjg@chromium.org>

test/py: Note which console produced unexpected output

At present the SPL and U-Boot consoles both present the same error message
when the expected console output does not appear. Add "SPL" to the S

test/py: Note which console produced unexpected output

At present the SPL and U-Boot consoles both present the same error message
when the expected console output does not appear. Add "SPL" to the SPL error
message to resolve this ambiguity.

Signed-off-by: Simon Glass <sjg@chromium.org>

show more ...

test/py: Add a helper to run a list of U-Boot commands

Some tests want to execute a sequence of commands. Add a helper for this.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Teddy Ree

test/py: Add a helper to run a list of U-Boot commands

Some tests want to execute a sequence of commands. Add a helper for this.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Teddy Reed <teddy.reed@gmail.com>

show more ...

Merge branch 'master' of git://git.denx.de/u-boot

Signed-off-by: Stefano Babic <sbabic@denx.de>

test/py: Support setting up specific timeout

Large file transfers, flash erasing and more complicated tests
requires more time to finish. Provide a way to setup specific
timeout directly in test.

F

test/py: Support setting up specific timeout

Large file transfers, flash erasing and more complicated tests
requires more time to finish. Provide a way to setup specific
timeout directly in test.

For example description for 50s test:
timeout = 50000
with u_boot_console.temporary_timeout(timeout):
u_boot_console.run_command(...)

Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Reviewed-by: Stephen Warren <swarren@nvidia.com>

show more ...

test/py: use space to interrupt autoboot

Sending CTRL-C to QEMU's stdin aborts the process, even if stdin is being
used as a serial port (at least in the raspi2 machine with "qemu -serial
stdin"). A

test/py: use space to interrupt autoboot

Sending CTRL-C to QEMU's stdin aborts the process, even if stdin is being
used as a serial port (at least in the raspi2 machine with "qemu -serial
stdin"). Avoid sending CTRL-C to U-Boot to prevent it exiting.

I'd originally used CTRL-C to make sure that if the character used to
abort autoboot ended up being treated as part of a command as well, it'd
abort command entry and return the prompt to a known state. However, this
is not needed, since aborting the autoboot eats the character used to do
that.

Signed-off-by: Stephen Warren <swarren@wwwdotorg.org>
Acked-by: Simon Glass <sjg@chromium.org>

show more ...

test/py: Add option to skip SPL signature checking

Provide user option to skip SPL signature verification for cases where
u-boot is build with SPL support but full U-Boot is also verified
without SP

test/py: Add option to skip SPL signature checking

Provide user option to skip SPL signature verification for cases where
u-boot is build with SPL support but full U-Boot is also verified
without SPL.

If you want to support this feature please add env__spl_skipped = True
to your boardenv configuration file.

For example Xilinx Zynq is using this feature where the same U-Boot
binary is checked with SPL and without SPL(with FSBL).

Signed-off-by: Michal Simek <michal.simek@xilinx.com>

show more ...

test/py: only check for SPL signature if SPL uses serial output

check for U-Boot SPL signature only if SPL really has a serial output.
So check if CONFIG_SPL_SERIAL_SUPPORT is active in board config

test/py: only check for SPL signature if SPL uses serial output

check for U-Boot SPL signature only if SPL really has a serial output.
So check if CONFIG_SPL_SERIAL_SUPPORT is active in board config.

Signed-off-by: Heiko Schocher <hs@denx.de>
Tested-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Stephen Warren <swarren@nvidia.com>

show more ...

test/py: put "Starting U-Boot" into separate log section

The initial boot of U-Boot happens within the context of the first test
that needs to access the U-Boot console when there is no existing
con

test/py: put "Starting U-Boot" into separate log section

The initial boot of U-Boot happens within the context of the first test
that needs to access the U-Boot console when there is no existing
connection. This keeps all activity nestled within test execution, which
fits well into the pytest model. However, this mingles the U-Boot startup
logs with the execution of some test(s), which hides find the boundary
between the two.

To solve this, wrap the "Starting U-Boot" logic into a separate log
section. If the user wishes, they can simply collapse this log section
when viewing the HTML log, to concentrate purely on the test's own
interaction.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>

show more ...

test/py: fix CONFIG_SPL test

The Python ini file parser that's used to parse .config converts all keys
to lower-case. Hence, all queries against the results must use lower-case.
Fix u_boot_console.e

test/py: fix CONFIG_SPL test

The Python ini file parser that's used to parse .config converts all keys
to lower-case. Hence, all queries against the results must use lower-case.
Fix u_boot_console.ensure_spawned() to test CONFIG_SPL correctly, or the
connection will fail for boards that have SPL.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>

show more ...

test/py: capture the entire U-Boot version at boot

The existing regex simply ensures that the captured version string doesn't
go past the end of a line. We really want to grab as much as possible. D

test/py: capture the entire U-Boot version at boot

The existing regex simply ensures that the captured version string doesn't
go past the end of a line. We really want to grab as much as possible. Do
this by explicitly including a ) character at the end of the regex to
match the last character of the version test.

Signed-off-by: Stephen Warren <swarren@nvidia.com>

show more ...

test/py: support running sandbox under gdbserver

Implement command--line option --gdbserver COMM, which does two things:

a) Run the sandbox process under gdbserver, using COMM as gdbserver's
com

test/py: support running sandbox under gdbserver

Implement command--line option --gdbserver COMM, which does two things:

a) Run the sandbox process under gdbserver, using COMM as gdbserver's
communication channel.

b) Disables all timeouts, so that if U-Boot is halted under the debugger,
tests don't fail. If the user gives up in the middle of a debugging
session, they can simply CTRL-C the test script to abort it.

This allows easy debugging of test failures without having to manually
re-create the failure conditions. Usage is:

Window 1:
./test/py/test.py --bd sandbox --gdbserver localhost:1234

Window 2:
gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234'

When using this option, it likely makes sense to use pytest's -k option
to limit the set of tests that are executed.

Simply running U-Boot directly under gdb (rather than gdbserver) was
also considered. However, this was rejected because:

a) gdb's output would then be processed by the test script, and likely
confuse it causing false failures.

b) pytest by default hides stdout from tests, which would prevent the
user from interacting with gdb.

While gdb can be told to redirect the debugee's stdio to a separate
PTY, this would appear to leave gdb's stdio directed at the test
scripts and the debugee's stdio directed elsewhere, which is the
opposite of the desired effect. Perhaps some complicated PTY muxing
and process hierarchy could invert this. However, the current scheme
is simple to implement and use, so it doesn't seem worth complicating
matters.

c) Using gdbserver allows arbitrary debuggers to be used, even those with
a GUI. If the test scripts invoked the debugger themselves, they'd have
to know how to execute arbitary applications. While the user could hide
this all in a wrapper script, this feels like extra complication.

An interesting future idea might be a --gdb-screen option, which could
spawn both U-Boot and gdb separately, and spawn the screen into a newly
created window under screen. Similar options could be envisaged for
creating a new xterm/... too.

--gdbserver currently only supports sandbox, and not real hardware.
That's primarily because the test hooks are responsible for all aspects of
hardware control, so there's nothing for the test scripts themselves can
do to enable gdbserver on real hardware. We might consider introducing a
separate --disable-timeouts option to support use of debuggers on real
hardware, and having --gdbserver imply that option.

Signed-off-by: Stephen Warren <swarren@nvidia.com>

show more ...

Merge git://git.denx.de/u-boot-dm

test/py: detect another "bad pattern" in console output

Many error situations in U-Boot print the message:
### ERROR ### Please RESET the board ###

Add this to the list of bad patterns the test

test/py: detect another "bad pattern" in console output

Many error situations in U-Boot print the message:
### ERROR ### Please RESET the board ###

Add this to the list of bad patterns the test system detects. One
practical advantage of this change is to detect the case where sandbox
is told to use a particular DTB file, and the file cannot be opened.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>

show more ...

test/py: check for bad patterns everywhere we wait

Currently, bad patterns are only honored when executing a shell command.
Other cases, such as the initial boot-up of U-Boot or when interacting
wit

test/py: check for bad patterns everywhere we wait

Currently, bad patterns are only honored when executing a shell command.
Other cases, such as the initial boot-up of U-Boot or when interacting
with command output rather than gathering all output prior to the shell
prompt, do not currently look for bad patterns in console output. This
patch makes sure that bad patterns are honored everywhere.

One benefit of this change is that if U-Boot sandbox fails to start up,
the error message it emits can be caught immediately, rather than relying
on a (long) timeout when waiting for the expected signon message and/or
command prompt.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>

show more ...

test.py: calculate bad patterns on change only

A future patch will use the bad_patterns array in multiple places. Rather
than duplicating the code to calculate it, or even sharing it in a
function a

test.py: calculate bad patterns on change only

A future patch will use the bad_patterns array in multiple places. Rather
than duplicating the code to calculate it, or even sharing it in a
function and simply calling it redundantly when nothing has changed, only
re-calculate the list when some change is made to it. This reduces work.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>

show more ...

test/py: use " for docstrings

Python's coding style docs indicate to use " not ' for docstrings.

test/py has other violations of the coding style docs, since the docs
specify a stranger style than

test/py: use " for docstrings

Python's coding style docs indicate to use " not ' for docstrings.

test/py has other violations of the coding style docs, since the docs
specify a stranger style than I would expect, but nobody has complained
about those yet:-)

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

show more ...

test/py: make crash detection more robust

test/py contains logic to detect the target crashing and rebooting by
searching the console output for a U-Boot signon message, which will
presumably be emi

test/py: make crash detection more robust

test/py contains logic to detect the target crashing and rebooting by
searching the console output for a U-Boot signon message, which will
presumably be emitted when the system boots after the crash/reset.

Currently, this logic only searches for the exact signon message that
was printed by the U-Boot version under test, upon the assumption that
binary is written into flash, and hence will be the version booted after
any reset. However, this is not a valid assumption; some test setups
download the U-Boot-under-test into RAM and boot it from there, and in
such a scenario an arbitrary U-Boot version may be located in flash and
hence run after any reset.

Fix the reset detection logic to match any U-Boot signon message. This
prevents false negatives.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>

show more ...