[
https://issues.apache.org/jira/browse/SOLR-18642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18143310#comment-18143310
]
David Smiley commented on SOLR-18642:
-------------------------------------
Can the description link to CATS or add as a Jira link?
I particularly like the idea of running the tests via JUnit (and thus I assume
Gradle as well), as we'd then get test history tracking on Develocity.
> Replace BATS with CATS
> ----------------------
>
> Key: SOLR-18642
> URL: https://issues.apache.org/jira/browse/SOLR-18642
> Project: Solr
> Issue Type: Improvement
> Components: cli, test-framework
> Reporter: Jan Høydahl
> Assignee: Jan Høydahl
> Priority: Major
> Labels: pull-request-available, test-framework, windows
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Today we have BATS for testing {{bin/solr}} and realistic CLI type tests in
> bash. We have nothing similar for Windows. SOLR-17508 explored Pester, but it
> went nowhere.
> This issue proposes *CATS* — a Cross-platform Automated Testing System — a
> Java-native harness for shell testing, so such tests run under JUnit like any
> other test, in parallel, on every platform. The name follows BATS (Bash
> Automated Testing System) deliberately: same idea, one letter and one
> platform restriction fewer.
> h3. The DSL
> Pure Java with convenience methods, so a ported BATS line reads almost
> unchanged:
> {code:java}
> @Test
> public void aNodeWeStartIsTheOnlyOneListed() {
> env("SOLR_HEAP", "512m");
> run("solr start");
> run("solr create -c " + collName + " --shards 2")
> .assertContains("2 shard(s)");
> run("solr status --port ${SOLR_PORT}")
> .assertContains("running on port");
> run("solr stop --all");
> run("solr status").assertContains("No Solr nodes are running");
> }
> {code}
> Variable references in a command line are resolved by the harness, not by the
> shell — which is what makes one line work under both, since bash expands
> {{$NAME}} and cmd.exe does not, while cmd.exe wants {{%NAME%}} and bash does
> not.
> {{run()}} asserts that the command succeeded, so the happy path carries no
> status boilerplate; a command expected to fail uses {{{}runFail(){}}}, which
> asserts a non-zero status and hands back the result to assert on what it
> printed.
> h3. One source, both shells
> Each test runs under bash on Linux and macOS, and under cmd.exe on Windows. A
> platform-specific construct — a Unix-only tool, a pipe, command substitution
> — is rejected at the call site with the fluent method that replaces it named
> in the message, rather than silently passing on one platform. A test that
> genuinely is one shell's carries {{{}@EnabledOnShell{}}}.
> Suites run in parallel forks: each test class gets its own {{{}SOLR_HOME{}}},
> its own {{SOLR_PID_DIR}} and its own block of ports.
> A failure prints the command transcript — every command, its exit code and
> its output — and one {{FAILED:}} line.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
