epugh commented on code in PR #5338:
URL: https://github.com/apache/solr/pull/5338#discussion_r4578745577


##########
dev-docs/writing-tests.adoc:
##########
@@ -0,0 +1,161 @@
+= Writing Tests for Solr
+:toc: left
+:toclevels: 2
+
+Which kind of test to write, and the lifecycle conventions they all share. 
Building and running the suite is covered in xref:solr-source-code.adoc[Working 
With Solr Source Code]; `./gradlew helpTests` covers running them.
+
+The rule: *write the cheapest kind of test that can actually catch the bug.*
+
+== Test types
+
+[cols="1,2,1,1", options="header"]
+|===
+| Base class | Covers | Cost | Status
+
+| `SolrTestCase`
+| Plain unit tests. Add `@ClassRule SolrJettyTestRule` (or 
`EmbeddedSolrServerTestRule`) when you need a live node.
+| ms to seconds
+| Current. The default — extend this unless you need more.
+
+| `SolrCloudTestCase`
+| In-JVM `MiniSolrCloudCluster`: N Jetty nodes plus a `ZkTestServer`. Routing, 
replication, recovery, Collections API, cluster state.
+| seconds to minutes
+| Current, but not a default.
+
+| `AdminUiTestBase`
+| AngularJS Admin UI in headless Chrome via Selenium, against a 2-node cluster 
whose Jetty nodes also serve the UI. In 
`solr/webapp/src/test/org/apache/solr/webapp/`.
+| cluster + a browser
+| Current, opt-in with `-Ptests.selenium=true`. See 
link:admin-ui-tests.md[admin-ui-tests.md].
+
+| `SolrTestCaseJ4`
+| Adds the single-core `TestHarness` to `SolrTestCase`: `initCore`, `assertQ`, 
`assertU`, SSL randomization. The base of most Solr tests, directly or via 
`SolrCloudTestCase`.
+| as `SolrTestCase`
+| Deprecated. Not in new tests.
+
+| BATS, `solr/packaging/test/*.bats`
+| Bash tests that drive `bin/solr` against a real, unpacked Solr distribution. 
The only harness that runs an installation the way a user gets one, so a 
classpath, Jetty configuration or file layout problem reproduces here and 
nowhere else. `./gradlew integrationTests`.
+| seconds to minutes, sequential
+| Current, and bash-only: `bin\solr.cmd` has no automated coverage.
+
+| `solr/docker/tests/cases/*`
+| Bash cases that `docker run` the Solr image built from this tree: 
entrypoint, `docker-entrypoint-initdb.d` scripts, `/var/solr` volume handling 
and permissions, core precreation, non-root and random UIDs, log4j config. 
`./gradlew -p solr/docker testDocker`, helpers in `tests/shared.sh`.
+| image build + containers
+| Current, bash-only. For the image's own contract, not for Solr behaviour.
+|===
+
+`configureCluster(n)` starts `n` Jetty servers and an embedded ZooKeeper in 
the test JVM before your first assertion, and every collection then goes 
through the real Collections API with real leader election — which is why 
`SolrCloudTestCase` is not the default, and why most cluster-test flakiness is 
a missing `waitForState`.
+
+`SolrTestCaseJ4` is not annotated `@Deprecated` — too much extends it for that 
to be useful — but it is on the way out. Keep editing existing J4 tests in 
place; do not convert them as a side effect of an unrelated fix. New classes 
extend `SolrTestCase` and build documents and params from plain SolrJ types.
+
+== Choosing a test type
+
+* Plain class, parser, config object, response writer, utility — 
`SolrTestCase`, no server.
+* Needs a live Solr, one node — `SolrTestCase` + `SolrJettyTestRule`, or 
`EmbeddedSolrServerTestRule` if the HTTP layer is not part of the test.
+* Only reproduces across nodes, or involves ZooKeeper, routing, replication, 
recovery or the Collections API — `SolrCloudTestCase`, with the lowest node 
count that still shows the bug.
+* In `bin/solr`, `solr.in.sh`, CLI parsing, output or exit codes — a `.bats` 
file in `solr/packaging/test/`. Also the only place a packaging problem 
reproduces: a jar missing from the classpath, a Jetty setting, or a file in the 
wrong place. The `SolrTestCase` hierarchy builds no installation, so none of 
those are visible to it.
+* Needs a real external service, not a mock — pick from the above as usual, 
then add a Testcontainers `@ClassRule`; see below.
+* In the Admin UI's JavaScript or templates — `AdminUiTestBase`. Test the APIs 
it calls separately; keep the browser test to what the browser renders.
+* In the Docker image — its entrypoint, init scripts, volume or permission 
handling, the `SOLR_*` variables it honours — a case under 
`solr/docker/tests/cases/`, not a BATS suite.
+
+== Randomized testing
+
+Every test that extends `SolrTestCase` runs under randomizedtesting's 
`RandomizedRunner`, inherited from `LuceneTestCase`. One master seed per run 
drives every `random()` call, so a failure is reproducible from the seed the 
runner prints.
+
+The framework already varies Lucene codec and postings format, directory 
implementation, merge policy, locale, time zone, and the order of test methods 
— so no test may depend on method order, on the default locale, or on the 
default time zone. SSL is randomized too, but only for `SolrTestCaseJ4` 
subclasses, which is where `@RandomizeSSL` sits; opt out with `@SuppressSSL`.
+
+What to use:
+
+* `random()` — the per-test `Random`. Never `new Random()` or `Math.random()`: 
an unseeded source makes the failure unreproducible.
+* `atLeast(n)` — `n * RANDOM_MULTIPLIER`, plus up to 50% random fudge. The 
normal way to size a loop.
+* `rarely()` / `usually()` — take an unusual branch about 1% of the time (5% 
in nightly runs).

Review Comment:
   is this 1% or 99% of the time?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to