[
https://issues.apache.org/jira/browse/SOLR-18642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18143305#comment-18143305
]
Jan Høydahl commented on SOLR-18642:
------------------------------------
h3. Status
* All 29 real BATS files are ported, to 30 JUnit classes.
* Green on macOS, including the rolling upgrade test, which became
Testcontainers rather than {{docker}} command lines.
* The full suite now runs on Windows against {{bin/solr.cmd}} as well; see
below.
* Several ports are *stronger* than their originals: BATS ignored the exit
status of every {{{}run{}}}, so assertions that would have passed against a
command that did nothing now also assert the status.
h3. Benchmarks
Three runs taken back to back on an otherwise quiet 14-core M-series Mac,
picocli CLI throughout. The rolling upgrade test is excluded from both sides.
|| ||BATS||CATS||change||
|Wall clock, full suite|18m27s (1107s)|6m40s (400s)|2.8x faster|
|Wall clock, single worker|sequential by design|15m35s (935s)|1.18x faster|
|Sum of per-test times|991s|891s|10% less|
|Parallelism|sequential, one file at a time|4 forks|—|
|Platforms covered|Linux, macOS|Linux, macOS, Windows|one more|
|Tests|129 (2 skipped)|133 (3 skipped)|+4|
CATS is faster both sequentially and in parallel, so the win is not only
concurrency. Could have
run even faster if Gradle scheduled slow tests earlier.
h3. Windows
The full suite has run against {{bin/solr.cmd}} on a Windows 11 ARM64 VM, 4
forks: *135 tests, 8 failures, 13 skipped, 23m51s*. The 13 skips are the tests
marked bash-only, mostly where cmd.exe cannot express the construct at all.
Every one of the 8 failures is a real divergence between the two scripts -
which is the whole point of the exercise, since nothing in the
project tested that script before.
h3. Findings
18 bugs and shortcomings, none of them regressions, all pre-existing and all to
be filed separately:
||where||count||notes||
|{{bin/solr.cmd}}|8|the largest group, and unsurprising: this script has never
had a test|
|Java (CLI tools, modules)|8|two of these may share one root cause|
|{{bin/solr}}|2|both are divergences where bash looks like the odd one out|
🤖 Summary written by AI, checked by me
> Replace BATS with CATS
> ----------------------
>
> Key: SOLR-18642
> URL: https://issues.apache.org/jira/browse/SOLR-18642
> Project: Solr
> Issue Type: Improvement
> Components: cli, test-framework
> Reporter: Jan Høydahl
> Assignee: Jan Høydahl
> Priority: Major
> Labels: test-framework, windows
>
> Today we have BATS for testing {{bin/solr}} and realistic CLI type tests in
> bash. We have nothing similar for Windows. SOLR-17508 explored Pester, but it
> went nowhere.
> This issue proposes *CATS* — a Cross-platform Automated Testing System — a
> Java-native harness for shell testing, so such tests run under JUnit like any
> other test, in parallel, on every platform. The name follows BATS (Bash
> Automated Testing System) deliberately: same idea, one letter and one
> platform restriction fewer.
> h3. The DSL
> Pure Java with convenience methods, so a ported BATS line reads almost
> unchanged:
> {code:java}
> @Test
> public void aNodeWeStartIsTheOnlyOneListed() {
> env("SOLR_HEAP", "512m");
> run("solr start");
> run("solr create -c " + collName + " --shards 2")
> .assertContains("2 shard(s)");
> run("solr status --port ${SOLR_PORT}")
> .assertContains("running on port");
> run("solr stop --all");
> run("solr status").assertContains("No Solr nodes are running");
> }
> {code}
> Variable references in a command line are resolved by the harness, not by the
> shell — which is what makes one line work under both, since bash expands
> {{$NAME}} and cmd.exe does not, while cmd.exe wants {{%NAME%}} and bash does
> not.
> {{run()}} asserts that the command succeeded, so the happy path carries no
> status boilerplate; a command expected to fail uses {{{}runFail(){}}}, which
> asserts a non-zero status and hands back the result to assert on what it
> printed.
> h3. One source, both shells
> Each test runs under bash on Linux and macOS, and under cmd.exe on Windows. A
> platform-specific construct — a Unix-only tool, a pipe, command substitution
> — is rejected at the call site with the fluent method that replaces it named
> in the message, rather than silently passing on one platform. A test that
> genuinely is one shell's carries {{{}@EnabledOnShell{}}}.
> Suites run in parallel forks: each test class gets its own {{{}SOLR_HOME{}}},
> its own {{SOLR_PID_DIR}} and its own block of ports.
> A failure prints the command transcript — every command, its exit code and
> its output — and one {{FAILED:}} line.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
