janhoy commented on code in PR #5323:
URL: https://github.com/apache/solr/pull/5323#discussion_r4567748124
##########
dev-docs/writing-tests.adoc:
##########
@@ -0,0 +1,173 @@
+= Writing Tests for Solr
+:toc: left
+:toclevels: 2
+
+Which kind of test to write, and the lifecycle conventions they all share.
Building and running the suite is covered in xref:solr-source-code.adoc[Working
With Solr Source Code]; `./gradlew helpTests` covers running them.
+
+The rule: *write the cheapest kind of test that can actually catch the bug.*
+
+== Test types
+
+[cols="1,2,1,1", options="header"]
+|===
+| Base class | Covers | Cost | Status
+
+| `SolrTestCase`
+| Plain unit tests. Add `@ClassRule SolrJettyTestRule` (or
`EmbeddedSolrServerTestRule`) when you need a live node.
+| ms to seconds
+| Current. The default — extend this unless you need more.
+
+| `SolrCloudTestCase`
+| In-JVM `MiniSolrCloudCluster`: N Jetty nodes plus a `ZkTestServer`. Routing,
replication, recovery, Collections API, cluster state.
+| seconds to minutes
+| Current, but not a default.
+
+| `SolrShellTestCase`
+| CATS — Cross-platform Automated Testing System: `bin/solr` and
`bin\solr.cmd` as real subprocesses through the platform's own shell. Harness
in `solr/test-framework/src/java/org/apache/solr/shell/`, suites in
`solr/packaging/src/test/`.
+| seconds to minutes
+| Current. New CLI coverage goes here.
+
+| `AdminUiTestBase`
+| AngularJS Admin UI in headless Chrome via Selenium, against a 2-node cluster
whose Jetty nodes also serve the UI. In
`solr/webapp/src/test/org/apache/solr/webapp/`.
+| cluster + a browser
+| Current, opt-in with `-Ptests.selenium=true`. See
link:admin-ui-tests.md[admin-ui-tests.md].
+
+| A Testcontainers `@ClassRule`
+| An external service the test needs a real instance of — Tika Server, Kafka,
an S3 mock. Any base class; the container comes from a rule, not from
inheritance.
+| cluster + a container pull
+| Current. Skips itself when Docker is absent.
+
+| `SolrTestCaseJ4`
+| Adds the single-core `TestHarness` to `SolrTestCase`: `initCore`, `assertQ`,
`assertU`, SSL randomization. Directly extended by ~600 test classes and
transitively the base of ~1250 — four in five Solr tests, mostly via
`SolrCloudTestCase`.
Review Comment:
🤖 Dropped, along with every other count in the file.
##########
dev-docs/writing-tests.adoc:
##########
@@ -0,0 +1,173 @@
+= Writing Tests for Solr
+:toc: left
+:toclevels: 2
+
+Which kind of test to write, and the lifecycle conventions they all share.
Building and running the suite is covered in xref:solr-source-code.adoc[Working
With Solr Source Code]; `./gradlew helpTests` covers running them.
+
+The rule: *write the cheapest kind of test that can actually catch the bug.*
+
+== Test types
+
+[cols="1,2,1,1", options="header"]
+|===
+| Base class | Covers | Cost | Status
+
+| `SolrTestCase`
+| Plain unit tests. Add `@ClassRule SolrJettyTestRule` (or
`EmbeddedSolrServerTestRule`) when you need a live node.
+| ms to seconds
+| Current. The default — extend this unless you need more.
+
+| `SolrCloudTestCase`
+| In-JVM `MiniSolrCloudCluster`: N Jetty nodes plus a `ZkTestServer`. Routing,
replication, recovery, Collections API, cluster state.
+| seconds to minutes
+| Current, but not a default.
+
+| `SolrShellTestCase`
+| CATS — Cross-platform Automated Testing System: `bin/solr` and
`bin\solr.cmd` as real subprocesses through the platform's own shell. Harness
in `solr/test-framework/src/java/org/apache/solr/shell/`, suites in
`solr/packaging/src/test/`.
+| seconds to minutes
+| Current. New CLI coverage goes here.
+
+| `AdminUiTestBase`
+| AngularJS Admin UI in headless Chrome via Selenium, against a 2-node cluster
whose Jetty nodes also serve the UI. In
`solr/webapp/src/test/org/apache/solr/webapp/`.
+| cluster + a browser
+| Current, opt-in with `-Ptests.selenium=true`. See
link:admin-ui-tests.md[admin-ui-tests.md].
+
+| A Testcontainers `@ClassRule`
+| An external service the test needs a real instance of — Tika Server, Kafka,
an S3 mock. Any base class; the container comes from a rule, not from
inheritance.
+| cluster + a container pull
+| Current. Skips itself when Docker is absent.
+
+| `SolrTestCaseJ4`
+| Adds the single-core `TestHarness` to `SolrTestCase`: `initCore`, `assertQ`,
`assertU`, SSL randomization. Directly extended by ~600 test classes and
transitively the base of ~1250 — four in five Solr tests, mostly via
`SolrCloudTestCase`.
+| as `SolrTestCase`
+| Deprecated. Not in new tests.
+
+| BATS, `solr/packaging/test/*.bats`
+| The original bash `bin/solr` tests. `./gradlew integrationTests`.
+| seconds to minutes, sequential
+| Deprecated in favour of CATS, which runs the same coverage on both shells.
Bash-only; never ran on Windows.
+|===
+
+`configureCluster(n)` starts `n` Jetty servers and an embedded ZooKeeper in
the test JVM before your first assertion, and every collection then goes
through the real Collections API with real leader election — which is why
`SolrCloudTestCase` is not the default, and why most cluster-test flakiness is
a missing `waitForState`.
+
+`SolrTestCaseJ4` is not annotated `@Deprecated` — too much extends it for that
to be useful — but it is on the way out. Keep editing existing J4 tests in
place; do not convert them as a side effect of an unrelated fix. New classes
extend `SolrTestCase` and build documents and params from plain SolrJ types.
+
+== Choosing a test type
+
+* Plain class, parser, config object, response writer, utility —
`SolrTestCase`, no server.
+* Needs a live Solr, one node — `SolrTestCase` + `SolrJettyTestRule`, or
`EmbeddedSolrServerTestRule` if the HTTP layer is not part of the test.
+* Only reproduces across nodes, or involves ZooKeeper, routing, replication,
recovery or the Collections API — `SolrCloudTestCase`, with the lowest node
count that still shows the bug.
+* In `bin/solr`, `bin/solr.cmd`, `solr.in.sh`/`solr.in.cmd`, CLI parsing,
output or exit codes — `SolrShellTestCase`. The only harness that catches a
Windows-only script bug.
+* Needs a real external service, not a mock — a Testcontainers `@ClassRule`;
see below.
Review Comment:
🤖 Agreed — and the row was arguing against itself, since its own cell said
the container comes from a rule, not from inheritance, in a table whose first
column is "Base class". Dropped the row; the bullet now reads as something you
add to a choice rather than a choice of its own.
##########
dev-docs/writing-tests.adoc:
##########
@@ -0,0 +1,173 @@
+= Writing Tests for Solr
+:toc: left
+:toclevels: 2
+
+Which kind of test to write, and the lifecycle conventions they all share.
Building and running the suite is covered in xref:solr-source-code.adoc[Working
With Solr Source Code]; `./gradlew helpTests` covers running them.
+
+The rule: *write the cheapest kind of test that can actually catch the bug.*
+
+== Test types
+
+[cols="1,2,1,1", options="header"]
+|===
+| Base class | Covers | Cost | Status
+
+| `SolrTestCase`
+| Plain unit tests. Add `@ClassRule SolrJettyTestRule` (or
`EmbeddedSolrServerTestRule`) when you need a live node.
+| ms to seconds
+| Current. The default — extend this unless you need more.
+
+| `SolrCloudTestCase`
+| In-JVM `MiniSolrCloudCluster`: N Jetty nodes plus a `ZkTestServer`. Routing,
replication, recovery, Collections API, cluster state.
+| seconds to minutes
+| Current, but not a default.
+
+| `SolrShellTestCase`
+| CATS — Cross-platform Automated Testing System: `bin/solr` and
`bin\solr.cmd` as real subprocesses through the platform's own shell. Harness
in `solr/test-framework/src/java/org/apache/solr/shell/`, suites in
`solr/packaging/src/test/`.
+| seconds to minutes
+| Current. New CLI coverage goes here.
+
+| `AdminUiTestBase`
+| AngularJS Admin UI in headless Chrome via Selenium, against a 2-node cluster
whose Jetty nodes also serve the UI. In
`solr/webapp/src/test/org/apache/solr/webapp/`.
+| cluster + a browser
+| Current, opt-in with `-Ptests.selenium=true`. See
link:admin-ui-tests.md[admin-ui-tests.md].
+
+| A Testcontainers `@ClassRule`
+| An external service the test needs a real instance of — Tika Server, Kafka,
an S3 mock. Any base class; the container comes from a rule, not from
inheritance.
+| cluster + a container pull
+| Current. Skips itself when Docker is absent.
+
+| `SolrTestCaseJ4`
+| Adds the single-core `TestHarness` to `SolrTestCase`: `initCore`, `assertQ`,
`assertU`, SSL randomization. Directly extended by ~600 test classes and
transitively the base of ~1250 — four in five Solr tests, mostly via
`SolrCloudTestCase`.
+| as `SolrTestCase`
+| Deprecated. Not in new tests.
+
+| BATS, `solr/packaging/test/*.bats`
+| The original bash `bin/solr` tests. `./gradlew integrationTests`.
+| seconds to minutes, sequential
+| Deprecated in favour of CATS, which runs the same coverage on both shells.
Bash-only; never ran on Windows.
+|===
+
+`configureCluster(n)` starts `n` Jetty servers and an embedded ZooKeeper in
the test JVM before your first assertion, and every collection then goes
through the real Collections API with real leader election — which is why
`SolrCloudTestCase` is not the default, and why most cluster-test flakiness is
a missing `waitForState`.
+
+`SolrTestCaseJ4` is not annotated `@Deprecated` — too much extends it for that
to be useful — but it is on the way out. Keep editing existing J4 tests in
place; do not convert them as a side effect of an unrelated fix. New classes
extend `SolrTestCase` and build documents and params from plain SolrJ types.
+
+== Choosing a test type
+
+* Plain class, parser, config object, response writer, utility —
`SolrTestCase`, no server.
+* Needs a live Solr, one node — `SolrTestCase` + `SolrJettyTestRule`, or
`EmbeddedSolrServerTestRule` if the HTTP layer is not part of the test.
+* Only reproduces across nodes, or involves ZooKeeper, routing, replication,
recovery or the Collections API — `SolrCloudTestCase`, with the lowest node
count that still shows the bug.
+* In `bin/solr`, `bin/solr.cmd`, `solr.in.sh`/`solr.in.cmd`, CLI parsing,
output or exit codes — `SolrShellTestCase`. The only harness that catches a
Windows-only script bug.
Review Comment:
🤖 Agreed, and undersold — I had it framed purely as "the CLI harness", which
sends someone chasing a classpath or Jetty-config bug elsewhere. Both the table
row and the choosing bullet now lead with the real-installation property.
##########
dev-docs/writing-tests.adoc:
##########
@@ -0,0 +1,173 @@
+= Writing Tests for Solr
+:toc: left
+:toclevels: 2
+
+Which kind of test to write, and the lifecycle conventions they all share.
Building and running the suite is covered in xref:solr-source-code.adoc[Working
With Solr Source Code]; `./gradlew helpTests` covers running them.
+
+The rule: *write the cheapest kind of test that can actually catch the bug.*
+
+== Test types
+
+[cols="1,2,1,1", options="header"]
+|===
+| Base class | Covers | Cost | Status
+
+| `SolrTestCase`
+| Plain unit tests. Add `@ClassRule SolrJettyTestRule` (or
`EmbeddedSolrServerTestRule`) when you need a live node.
+| ms to seconds
+| Current. The default — extend this unless you need more.
+
+| `SolrCloudTestCase`
+| In-JVM `MiniSolrCloudCluster`: N Jetty nodes plus a `ZkTestServer`. Routing,
replication, recovery, Collections API, cluster state.
+| seconds to minutes
+| Current, but not a default.
+
+| `SolrShellTestCase`
+| CATS — Cross-platform Automated Testing System: `bin/solr` and
`bin\solr.cmd` as real subprocesses through the platform's own shell. Harness
in `solr/test-framework/src/java/org/apache/solr/shell/`, suites in
`solr/packaging/src/test/`.
+| seconds to minutes
+| Current. New CLI coverage goes here.
+
+| `AdminUiTestBase`
+| AngularJS Admin UI in headless Chrome via Selenium, against a 2-node cluster
whose Jetty nodes also serve the UI. In
`solr/webapp/src/test/org/apache/solr/webapp/`.
+| cluster + a browser
+| Current, opt-in with `-Ptests.selenium=true`. See
link:admin-ui-tests.md[admin-ui-tests.md].
+
+| A Testcontainers `@ClassRule`
+| An external service the test needs a real instance of — Tika Server, Kafka,
an S3 mock. Any base class; the container comes from a rule, not from
inheritance.
+| cluster + a container pull
+| Current. Skips itself when Docker is absent.
+
+| `SolrTestCaseJ4`
+| Adds the single-core `TestHarness` to `SolrTestCase`: `initCore`, `assertQ`,
`assertU`, SSL randomization. Directly extended by ~600 test classes and
transitively the base of ~1250 — four in five Solr tests, mostly via
`SolrCloudTestCase`.
+| as `SolrTestCase`
+| Deprecated. Not in new tests.
+
+| BATS, `solr/packaging/test/*.bats`
+| The original bash `bin/solr` tests. `./gradlew integrationTests`.
+| seconds to minutes, sequential
+| Deprecated in favour of CATS, which runs the same coverage on both shells.
Bash-only; never ran on Windows.
+|===
+
+`configureCluster(n)` starts `n` Jetty servers and an embedded ZooKeeper in
the test JVM before your first assertion, and every collection then goes
through the real Collections API with real leader election — which is why
`SolrCloudTestCase` is not the default, and why most cluster-test flakiness is
a missing `waitForState`.
+
+`SolrTestCaseJ4` is not annotated `@Deprecated` — too much extends it for that
to be useful — but it is on the way out. Keep editing existing J4 tests in
place; do not convert them as a side effect of an unrelated fix. New classes
extend `SolrTestCase` and build documents and params from plain SolrJ types.
+
+== Choosing a test type
+
+* Plain class, parser, config object, response writer, utility —
`SolrTestCase`, no server.
+* Needs a live Solr, one node — `SolrTestCase` + `SolrJettyTestRule`, or
`EmbeddedSolrServerTestRule` if the HTTP layer is not part of the test.
+* Only reproduces across nodes, or involves ZooKeeper, routing, replication,
recovery or the Collections API — `SolrCloudTestCase`, with the lowest node
count that still shows the bug.
+* In `bin/solr`, `bin/solr.cmd`, `solr.in.sh`/`solr.in.cmd`, CLI parsing,
output or exit codes — `SolrShellTestCase`. The only harness that catches a
Windows-only script bug.
+* Needs a real external service, not a mock — a Testcontainers `@ClassRule`;
see below.
+* In the Admin UI's JavaScript or templates — `AdminUiTestBase`. Test the APIs
it calls separately; keep the browser test to what the browser renders.
+* Tempted to add a `.bats` file — don't. Write it as a `SolrShellTestCase`.
Touch the `.bats` files only to keep them working until they are ported.
+
+== Randomized testing
+
+Every test that extends `SolrTestCase` runs under randomizedtesting's
`RandomizedRunner`, inherited from `LuceneTestCase`. One master seed per run
drives every `random()` call, so a failure is reproducible from the seed the
runner prints.
+
+The framework already varies Lucene codec and postings format, directory
implementation, merge policy, locale, time zone, and the order of test methods
— so no test may depend on method order, on the default locale, or on the
default time zone. SSL is randomized too, but only for `SolrTestCaseJ4`
subclasses, which is where `@RandomizeSSL` sits; opt out with `@SuppressSSL`.
+
+What to use:
+
+* `random()` — the per-test `Random`. Never `new Random()` or `Math.random()`:
an unseeded source makes the failure unreproducible.
+* `atLeast(n)` — `n * RANDOM_MULTIPLIER`, plus up to 50% random fudge. The
normal way to size a loop.
+* `rarely()` / `usually()` — take an unusual branch about 1% of the time (5%
in nightly runs).
+* `TestUtil.randomSimpleString(random())` and its neighbours for field values.
+
+`RANDOM_MULTIPLIER` is 2 in a nightly run and 1 otherwise, so `atLeast` and
`rarely` scale themselves; read `TEST_NIGHTLY` directly only to skip or resize
work those two cannot express.
+
+Never commit `@Seed`: it pins the master seed and throws away the coverage you
just added. `@Repeat` is for chasing a rare seed while you work, not for
keeping.
+
+Randomize when the input space is large and uniform, correctness does not
depend on the particular value — field values, document and shard counts, the
order of unrelated operations — and you can state the property without
reimplementing the logic: a round trip, an invariant, or a comparison against a
simpler reference or a control core.
+
+== @Rule and lifecycle
+
+`SolrTestCase` already installs, for every subclass:
`SystemPropertiesRestoreRule` at both class and method scope,
`LogLevelTestRule` (honours `@LogLevel`), `RevertDefaultThreadHandlerRule`,
`QueryLimitsTestInjectionRule`, an `ObjectReleaseTracker` check,
`@ThreadLeakFilters` with `SolrIgnoredThreadsFilter` +
`QuickPatchThreadsFilter`, `@ThreadLeakLingering(linger = 1000)`, and
`@SuppressSysoutChecks` (which is why Solr tests may log freely to the
console). Do not re-add any of it, and do not restore system properties by hand.
Review Comment:
🤖 Now says "many" and points at `solrClassRules`/`solrTestRules` for the
list, keeping the two consequences a test author acts on: system properties
restored at both scopes, so a test never has to clear what it set; and
`@LogLevel("...=DEBUG")` on a class or method for turning logging up on one
test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]