DavidSpickett wrote: I tried a few more things, so I will summarise everything up until this point. There are two layers here: 1. Making test reports. 2. Producing the XML test results.
Using the buildkite plugin is not an option (https://github.com/llvm/llvm-project/pull/113290) because we'd need docker or a fork of the plugin. Our own test reporting script is the only way to generate the reports. This decides `#1`, unless we want to leave it as is until the move to GitHub actions. For `#2` the following ideas don't work: * `tail ---follow --retry results_file > combined_results_file`, then splitting it up in the reporting script * Does not work because tail does not print the whole file on truncation. We would miss data. * `mkfifo` and a listener script to write it out to unique files (https://github.com/llvm/llvm-project/pull/113703) * Works great on Linux but - * `mkfifo` fails silently in bash for Windows. There are no plans to fix that any time soon. * I could try https://learn.microsoft.com/en-us/windows/win32/ipc/named-pipes, but I suspect this would be even more code, and folks are already wary of adding too much to this system as it is. * Using `ninja check-all` * Does not allow us to disable testing of a project, but still build it. * Does not help with runtimes tests that are separate builds. These could work but have tradeoffs: * Using the lit feature `--unique-output...` (this PR) * Built into lit (for now). * Minimal changes to scripts and pipeline. * New arguments for LLVM_LIT_ARGS. * Run the report generator on exit. * Risk of exposing an option publicly that we no longer need when the pipeline is re-written. * I could RFC that option and drum up interest in it from other users and external projects, if that would help. I have not done so up to now because it might have looked like I was trying to bias this discussion. * Changing the build scripts to `ninja target` `mv result_file unique_name` in a loop (https://github.com/llvm/llvm-project/pull/113660) * The most script changes of any solution. * No changes to report generator. * Small bonus - result files are named after their check target (though I wonder if some check targets run lit multiple times, in which case this is actually a bug not a bonus). * Wrapping lit in a Python script that will catch the output file name argument, run lit, then move the file to a unique name (https://github.com/llvm/llvm-project/pull/113896) * It works, albeit in a fiddly way because of differences in `.py` or not between Windows and Linux. * We have to re-wrap lit every time cmake is run (what I have implemented) or - * We configure in another directory, patch that lit, and configure with `-DLLVM_EXTERNAL_LIT=<path to the other lit>` (which I have yet to test) * We're not using the actual lit, though the chances of a regression because of that are pretty low. `ninja check-lit` does still work, so though we aren't testing what we ship, it's pretty close. Are any of these solutions acceptable? Is there anything else I could do to address your concerns with any of these? If folks think it's just too close to a move to GitHub Actions and full pipeline rewrite (which I fully support doing), then I will shelve all this. However, personally I'd like better test reporting even if it is only for 2/3 months until the new system appears. https://github.com/llvm/llvm-project/pull/113447 _______________________________________________ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
