Re: [PATCH v2] test/py: Add a report show test durations

[Tom Rini]

On Sun, Jan 26, 2025 at 05:39:22PM -0700, Simon Glass wrote:

> Execution time varies widely with the existing tests. Provides a way to
> produce a summary of the time taken for each test, along with a
> histogram.
> 
> This is enabled with the --timing flag.
> 
> Enable it for sandbox in CI.
> 
> Example:
> 
>    Duration : Number of tests
>    ======== : ========================================
>        <1ms :  1
>        <8ms :  1
>       <20ms : # 20
>       <30ms : ######## 127
>       <50ms : ######################################## 582
>       <75ms : ####### 102
>      <100ms : ## 39
>      <200ms : ##### 86
>      <300ms : # 29
>      <500ms : ## 42
>      <750ms : # 16
>       <1.0s : # 15
>       <2.0s : # 23
>       <3.0s :  13
>       <5.0s :  9
>       <7.5s :  1
>      <10.0s :  6
>      <20.0s :  12
> 
> Signed-off-by: Simon Glass <s...@chromium.org>
> ---
> 
> Changes in v2:
> - Only show the report if the --timing flag is provided
> - Use --timing flag to show the report in CI for just the sandbox tests
> 
>  .azure-pipelines.yml       |  3 +-
>  .gitlab-ci.yml             |  3 +-
>  doc/develop/py_testing.rst | 33 +++++++++++++++
>  test/py/conftest.py        | 87 ++++++++++++++++++++++++++++++++++++++
>  4 files changed, 124 insertions(+), 2 deletions(-)
> 
> diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml
> index df3f82074af..1a5d15a8cdc 100644
> --- a/.azure-pipelines.yml
> +++ b/.azure-pipelines.yml
> @@ -295,7 +295,7 @@ stages:
>            export PATH=/opt/qemu/bin:/tmp/uboot-test-hooks/bin:\${PATH}
>            export PYTHONPATH=/tmp/uboot-test-hooks/py/travis-ci
>            # "\${var:+"-k \$var"}" expands to "" if \$var is empty, "-k 
> \$var" if not
> -          ./test/py/test.py -ra -o 
> cache_dir="\$UBOOT_TRAVIS_BUILD_DIR"/.pytest_cache --bd \${TEST_PY_BD} 
> \${TEST_PY_ID} \${TEST_PY_TEST_SPEC:+"-k \${TEST_PY_TEST_SPEC}"} --build-dir 
> "\$UBOOT_TRAVIS_BUILD_DIR" --report-dir "\$UBOOT_TRAVIS_BUILD_DIR" 
> --junitxml=\$(System.DefaultWorkingDirectory)/results.xml
> +          ./test/py/test.py -ra -o 
> cache_dir="\$UBOOT_TRAVIS_BUILD_DIR"/.pytest_cache --bd \${TEST_PY_BD} 
> \${TEST_PY_ID} \${TEST_TIMING} \${TEST_PY_TEST_SPEC:+"-k 
> \${TEST_PY_TEST_SPEC}"} --build-dir "\$UBOOT_TRAVIS_BUILD_DIR" --report-dir 
> "\$UBOOT_TRAVIS_BUILD_DIR" 
> --junitxml=\$(System.DefaultWorkingDirectory)/results.xml

Lets add a variable to pass further arguments to pytest directly and
then use that to pass --timing. That would let us do --capture=tee-sys
when needed to really debug a pytest failure in CI, in CI. Just thought
of that, sorry.

-- 
Tom

signature.asc
Description: PGP signature