epugh commented on code in PR #5338:
URL: https://github.com/apache/solr/pull/5338#discussion_r4578751656


##########
dev-docs/writing-tests.adoc:
##########
@@ -0,0 +1,161 @@
+= Writing Tests for Solr
+:toc: left
+:toclevels: 2
+
+Which kind of test to write, and the lifecycle conventions they all share. 
Building and running the suite is covered in xref:solr-source-code.adoc[Working 
With Solr Source Code]; `./gradlew helpTests` covers running them.
+
+The rule: *write the cheapest kind of test that can actually catch the bug.*
+
+== Test types
+
+[cols="1,2,1,1", options="header"]
+|===
+| Base class | Covers | Cost | Status
+
+| `SolrTestCase`
+| Plain unit tests. Add `@ClassRule SolrJettyTestRule` (or 
`EmbeddedSolrServerTestRule`) when you need a live node.
+| ms to seconds
+| Current. The default — extend this unless you need more.
+
+| `SolrCloudTestCase`
+| In-JVM `MiniSolrCloudCluster`: N Jetty nodes plus a `ZkTestServer`. Routing, 
replication, recovery, Collections API, cluster state.
+| seconds to minutes
+| Current, but not a default.
+
+| `AdminUiTestBase`
+| AngularJS Admin UI in headless Chrome via Selenium, against a 2-node cluster 
whose Jetty nodes also serve the UI. In 
`solr/webapp/src/test/org/apache/solr/webapp/`.
+| cluster + a browser
+| Current, opt-in with `-Ptests.selenium=true`. See 
link:admin-ui-tests.md[admin-ui-tests.md].
+
+| `SolrTestCaseJ4`
+| Adds the single-core `TestHarness` to `SolrTestCase`: `initCore`, `assertQ`, 
`assertU`, SSL randomization. The base of most Solr tests, directly or via 
`SolrCloudTestCase`.
+| as `SolrTestCase`
+| Deprecated. Not in new tests.
+
+| BATS, `solr/packaging/test/*.bats`
+| Bash tests that drive `bin/solr` against a real, unpacked Solr distribution. 
The only harness that runs an installation the way a user gets one, so a 
classpath, Jetty configuration or file layout problem reproduces here and 
nowhere else. `./gradlew integrationTests`.
+| seconds to minutes, sequential
+| Current, and bash-only: `bin\solr.cmd` has no automated coverage.
+
+| `solr/docker/tests/cases/*`
+| Bash cases that `docker run` the Solr image built from this tree: 
entrypoint, `docker-entrypoint-initdb.d` scripts, `/var/solr` volume handling 
and permissions, core precreation, non-root and random UIDs, log4j config. 
`./gradlew -p solr/docker testDocker`, helpers in `tests/shared.sh`.
+| image build + containers
+| Current, bash-only. For the image's own contract, not for Solr behaviour.
+|===
+
+`configureCluster(n)` starts `n` Jetty servers and an embedded ZooKeeper in 
the test JVM before your first assertion, and every collection then goes 
through the real Collections API with real leader election — which is why 
`SolrCloudTestCase` is not the default, and why most cluster-test flakiness is 
a missing `waitForState`.
+
+`SolrTestCaseJ4` is not annotated `@Deprecated` — too much extends it for that 
to be useful — but it is on the way out. Keep editing existing J4 tests in 
place; do not convert them as a side effect of an unrelated fix. New classes 
extend `SolrTestCase` and build documents and params from plain SolrJ types.
+
+== Choosing a test type
+
+* Plain class, parser, config object, response writer, utility — 
`SolrTestCase`, no server.
+* Needs a live Solr, one node — `SolrTestCase` + `SolrJettyTestRule`, or 
`EmbeddedSolrServerTestRule` if the HTTP layer is not part of the test.
+* Only reproduces across nodes, or involves ZooKeeper, routing, replication, 
recovery or the Collections API — `SolrCloudTestCase`, with the lowest node 
count that still shows the bug.
+* In `bin/solr`, `solr.in.sh`, CLI parsing, output or exit codes — a `.bats` 
file in `solr/packaging/test/`. Also the only place a packaging problem 
reproduces: a jar missing from the classpath, a Jetty setting, or a file in the 
wrong place. The `SolrTestCase` hierarchy builds no installation, so none of 
those are visible to it.
+* Needs a real external service, not a mock — pick from the above as usual, 
then add a Testcontainers `@ClassRule`; see below.
+* In the Admin UI's JavaScript or templates — `AdminUiTestBase`. Test the APIs 
it calls separately; keep the browser test to what the browser renders.
+* In the Docker image — its entrypoint, init scripts, volume or permission 
handling, the `SOLR_*` variables it honours — a case under 
`solr/docker/tests/cases/`, not a BATS suite.
+
+== Randomized testing
+
+Every test that extends `SolrTestCase` runs under randomizedtesting's 
`RandomizedRunner`, inherited from `LuceneTestCase`. One master seed per run 
drives every `random()` call, so a failure is reproducible from the seed the 
runner prints.
+
+The framework already varies Lucene codec and postings format, directory 
implementation, merge policy, locale, time zone, and the order of test methods 
— so no test may depend on method order, on the default locale, or on the 
default time zone. SSL is randomized too, but only for `SolrTestCaseJ4` 
subclasses, which is where `@RandomizeSSL` sits; opt out with `@SuppressSSL`.
+
+What to use:
+
+* `random()` — the per-test `Random`. Never `new Random()` or `Math.random()`: 
an unseeded source makes the failure unreproducible.
+* `atLeast(n)` — `n * RANDOM_MULTIPLIER`, plus up to 50% random fudge. The 
normal way to size a loop.
+* `rarely()` / `usually()` — take an unusual branch about 1% of the time (5% 
in nightly runs).
+* `TestUtil.randomSimpleString(random())` and its neighbours for field values.
+
+`RANDOM_MULTIPLIER` is 2 in a nightly run and 1 otherwise, so `atLeast` and 
`rarely` scale themselves; read `TEST_NIGHTLY` directly only to skip or resize 
work those two cannot express.
+
+Never commit `@Seed`: it pins the master seed and throws away the coverage you 
just added. `@Repeat` is for chasing a rare seed while you work, not for 
keeping.
+
+Randomize when the input space is large and uniform, correctness does not 
depend on the particular value — field values, document and shard counts, the 
order of unrelated operations — and you can state the property without 
reimplementing the logic: a round trip, an invariant, or a comparison against a 
simpler reference or a control core.
+
+== @Rule and lifecycle
+
+`SolrTestCase` installs many rules for every subclass; `solrClassRules` and 
`solrTestRules` in the class itself are the list. Two consequences worth 
knowing: system properties are restored at both class and method scope, so a 
test never has to clear what it set; and 
`@LogLevel("org.apache.solr.cloud=DEBUG")` on a class or a method works out of 
the box, which is how you turn logging up for one test without touching a 
logger.
+
+Do:
+
+* Reach for an existing `@ClassRule` before writing a 
`@BeforeClass`/`@AfterClass` pair: `SolrJettyTestRule` and 
`EmbeddedSolrServerTestRule` already start and close a node for you.
+* In `SolrCloudTestCase`, configure the cluster in `@BeforeClass`; the base 
class's `@AfterClass` shuts it down. For per-test isolation use `@After 
cluster.deleteAllCollections()` (which removes aliases too), not a cluster per 
method.
+* Call `super.setUp()`/`super.tearDown()` if you override them.
+* Use `createTempDir()` for scratch space.
+* Mute expected ERRORs: `try (ErrorLogMuter errors = 
ErrorLogMuter.substring("...")) { ... }`.
+
+Don't:
+
+* Don't set up in a `static` initializer — runners may load all test classes 
long before a suite runs.
+* Don't assume `@AfterClass` sees a complete fixture: JUnit 4 runs it even 
when `@BeforeClass` threw or raised an assumption failure.
+* Don't add a thread to `SolrIgnoredThreadsFilter` instead of fixing the leak; 
anything listed there must be completely stateless. `@ThreadLeakLingering` is 
resolved the same way, so a subclass declaration replaces the inherited 1000 
rather than raising it — and is for a slow-but-correct shutdown only.
+* Don't redeclare `@ThreadLeakFilters` without repeating the inherited filters 
— the annotation is not additive. `AdminUiTestBase` shows the correct form.
+* Don't leak tracked objects. `ObjectReleaseTracker` is asserted empty only 
when the suite passed, so `ObjectTracker found N object(s) that were not 
released` means your test left a `SolrCore`, `SolrIndexSearcher`, `Directory` 
or similar open.
+
+Annotations:
+
+These are Lucene annotations wired up as randomizedtesting *test groups*, 
disabled by default, so annotating a test with one stops it running in ordinary 
builds and in PR CI:
+
+* `@Nightly` — correct, but too slow to run every time.
+* `@AwaitsFix(bugUrl = "...")` — known-broken, and the required bug URL is 
what keeps it discoverable. Prefer it to `@Ignore`, which is plain JUnit and 
invisible to group filters; use `@Ignore` only where nothing is broken, such as 
an abstract base the runner would otherwise pick up.
+* `@LuceneTestCase.Monster(value)` — needs unusual disk or RAM.
+
+There is no `@Slow`, and `@BadApple` is gone: slow is `@Nightly`, broken is 
`@AwaitsFix`. Read solr-source-code.adoc's guidance on frequently-failing tests 
before disabling anything.
+
+== Containers for external services
+
+When a test needs a real instance of something Solr talks to, start it with 
Testcontainers from a JUnit rule rather than by hand. Examples in tree, all an 
`ExternalResource` declared as `@ClassRule`: `TikaServerContainerRule` 
(extraction), `KafkaContainerRule` (cross-dc-manager), `S3MockContainerRule` 
(s3-repository). `TikaServerExtractionBackendTest` is the one to copy.
+
+* `@ClassRule`, not `@Rule` — a container pull and start is far too expensive 
per method.
+* `Assume.assumeTrue(.., DockerClientFactory.instance().isDockerAvailable())` 
in `before()`, so the test skips where Docker is absent instead of failing.
+* `.waitingFor(Wait.forListeningPort())`, so there is no polling loop and no 
sleep.
+* `getMappedPort(n)` for the address. Never publish a fixed host port: it 
collides with whatever else the machine is running, and with another test class 
in a parallel build.
+
+== Waiting, not sleeping
+
+A fixed `Thread.sleep` is paid on every run and still flakes under load, 
because the thing you are waiting for is slower on a busy machine, not faster. 
Wait for the condition instead:

Review Comment:
   we should think about an effort to eradicate `Thread.sleep` from our tests.. 
it's a good use of AI....



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to