UPSTREAM: test/py: add timestamps to log

It can be useful to record how long tests take; this can help debug slow
running test systems or track changes in performance over time. Enhance
the test sys

UPSTREAM: test/py: add timestamps to log

It can be useful to record how long tests take; this can help debug slow
running test systems or track changes in performance over time. Enhance
the test system to record timestamps while running test:
- Whenever a new log file section is started.
- After U-Boot is started and communication has been established.
- After each host or U-Boot command is executed.

Change-Id: I1db02bfc1d69c8d6a4c390a778ae433bbd858a9d
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Signed-off-by: Kever Yang <kever.yang@rock-chips.com>
(cherry picked from commit 9679d339ad2d4c495d734bad3a0fb7be6c4215eb)

show more ...

test: Check exit status in run_and_log_expect_exception()

This check was missed. Add it and make the message more verbose.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reported-by: Tom Rini <trini

test: Check exit status in run_and_log_expect_exception()

This check was missed. Add it and make the message more verbose.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reported-by: Tom Rini <trini@konsulko.com>
Fixes: 9e17b034 (test/py: Provide a way to check that a command fails)

show more ...

test/py: Provide output from exceptions with RunAndLog()

Tests may want to look at the output from running a command, even if it
fails (e.g. with a non-zero return code). Provide a means to obtain t

test/py: Provide output from exceptions with RunAndLog()

Tests may want to look at the output from running a command, even if it
fails (e.g. with a non-zero return code). Provide a means to obtain this.

Another approach would be to return a class object containing both the
output and the exception, but I'm not sure if that would result in a lot
of refactoring.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Teddy Reed <teddy.reed@gmail.com>

show more ...

test/py: Allow RunAndLog() to return the output

Tests may want to look at the output from running a command. Return it so
that this is possible.

Signed-off-by: Simon Glass <sjg@chromium.org>
Review

test/py: Allow RunAndLog() to return the output

Tests may want to look at the output from running a command. Return it so
that this is possible.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Teddy Reed <teddy.reed@gmail.com>

show more ...

test/py: HTML awesome!

Implement three improvements to the HTML log file:
- Ability to expand/contract sections. All passing sections are contracted
at file load time so the user can concentrate o

test/py: HTML awesome!

Implement three improvements to the HTML log file:
- Ability to expand/contract sections. All passing sections are contracted
at file load time so the user can concentrate on issues requiring
action.
- The overall status report is copied to the top of the log for easy
access.
- Add links from the status report to the test logs, for easy navigation.

This all relies on Javascript and the jquery library. If the user doesn't
have Javascript enabled, or jquery can't be downloaded, the log should
look and behave identically to how it did before this patch.

A few notes on the diff:

- A few more 'with log.section("xxx")' were added, so that all stream
blocks are kept within a section block for consistent HTML entity
nesting structure. This changed indentation in a few places, making
the diff look slightly larger.
- HTML entity IDs are cleaned up. We assign simple incrementing integer
IDs now, rather than using mangled test names which were possibly
invalid.
- Sections and streams now use common CSS class names (in addition to the
current separate class names) to more easily share the new behaviour.
This also reduces the CSS file size since rules don't need to be
duplicated.
- An "OK" status is logged after some external command executions so that
make and flash steps are auto-contracted at log file load time, assuming
they passed.

Signed-off-by: Stephen Warren <swarren@nvidia.com>

show more ...

Merge git://git.denx.de/u-boot-dm

test/py: correctly log xfail/xpass tests

Tests can complete in passed, skipped, xpass, xfailed, or failed, states.
Currently the U-Boot log generation code doesn't handle the xfailed or
xpass states

test/py: correctly log xfail/xpass tests

Tests can complete in passed, skipped, xpass, xfailed, or failed, states.
Currently the U-Boot log generation code doesn't handle the xfailed or
xpass states since they aren't used. Add support for the remaining states.
Without this, tests that xfail end up being reported as skipped.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>

show more ...

test/py: Quote consistency

When converting test/py from " to ', I missed a few places (or added a
few inconsistencies later). Fix these.

Note that only quotes in code are converted; double-quotes i

test/py: Quote consistency

When converting test/py from " to ', I missed a few places (or added a
few inconsistencies later). Fix these.

Note that only quotes in code are converted; double-quotes in comments
and HTML are left as-is, since English and HTML use " not '.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

show more ...

test/py: use " for docstrings

Python's coding style docs indicate to use " not ' for docstrings.

test/py has other violations of the coding style docs, since the docs
specify a stranger style than

test/py: use " for docstrings

Python's coding style docs indicate to use " not ' for docstrings.

test/py has other violations of the coding style docs, since the docs
specify a stranger style than I would expect, but nobody has complained
about those yet:-)

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

show more ...

test/py: optionally ignore errors from shell commands

Sometimes it's useful to run shell commands and ignore any errors. One
example might be cleanup logic; if a test-case experiences an error, the

test/py: optionally ignore errors from shell commands

Sometimes it's useful to run shell commands and ignore any errors. One
example might be cleanup logic; if a test-case experiences an error, the
cleanup logic might experience an error too, and we don't want that error
to mask the original error, so we want to ignore the subsequent error.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>

show more ...

Merge git://git.denx.de/u-boot-dm

test/py: Implement pytest infrastructure

This tool aims to test U-Boot by executing U-Boot shell commands using the
console interface. A single top-level script exists to execute or attach
to the U-

test/py: Implement pytest infrastructure

This tool aims to test U-Boot by executing U-Boot shell commands using the
console interface. A single top-level script exists to execute or attach
to the U-Boot console, run the entire script of tests against it, and
summarize the results. Advantages of this approach are:

- Testing is performed in the same way a user or script would interact
with U-Boot; there can be no disconnect.
- There is no need to write or embed test-related code into U-Boot itself.
It is asserted that writing test-related code in Python is simpler and
more flexible that writing it all in C.
- It is reasonably simple to interact with U-Boot in this way.

A few simple tests are provided as examples. Soon, we should convert as
many as possible of the other tests in test/* and test/cmd_ut.c too.

The hook scripts, relay control utilities, and udev rules I use for my
own HW setup are published at https://github.com/swarren/uboot-test-hooks.

See README.md for more details!

Signed-off-by: Stephen Warren <swarren@wwwdotorg.org>
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Tested-by: Michal Simek <michal.simek@xilinx.com>
Tested-by: Simon Glass <sjg@chromium.org>
Acked-by: Simon Glass <sjg@chromium.org> #v3

show more ...