Github user tzulitai commented on a diff in the pull request:
https://github.com/apache/flink/pull/6172#discussion_r195992920
--- Diff: flink-end-to-end-tests/README.md ---
@@ -33,6 +33,53 @@ $ FLINK_DIR=<flink dir>
flink-end-to-end-tests/test-scripts/test_batch_wordcount
## Writing Tests
-Have a look at test_batch_wordcount.sh for a very basic test and
-test_streaming_kafka010.sh for a more involved example. Whenever possible,
try
-to put new functionality in common.sh so that it can be reused by other
tests.
+### Examples
+Have a look at `test_batch_wordcount.sh` for a very basic test and
+`test_streaming_kafka010.sh` for a more involved example. Whenever
possible, try
+to put new functionality in `common.sh` so that it can be reused by other
tests.
+
+### Adding a test case
+In order to add a new test case you need a new line to either
`test-scripts/run-nightly-tests.sh` and / or
`test-scripts/run-pre-commit-tests.sh`
+
+Adding a new test case generally follows the following pattern
+
+```sh
+run_test "simple end-to-end test" "$END_TO_END_DIR/test-scripts/simple.sh
arg1 arg2"
+```
+
+_Note: If you want to parameterize your tests please do so by adding
multiple test cases with parameters as arguments to the nightly / pre-commit
test suites. This allows the test runner to do a cleanup in between each
individual test and also to fail those tests individually._
+
+### Passing your test
+A test is considered to have passed if it:
+- has exit code 0
+- there are no non-empty .out files (nothing was written to stdout /
stderr by your flink program)
+- there are no exceptions in the log files
+- there are no errors in the log files
+
+### Failing your test
+A test is considered to have failed if it:
+- exited with non-zero exit code
+- has non-empty *.out files (something was written to stdout / stderr by
your flink program)
+- there are errors in the log files
+- there are exceptions in the log files
+
+_There is a whitelist for exceptions and errors that do not lead to
failure, they can be found in the `check_logs_for_errors` and
`check_logs_for_exceptions` in `test-scripts/common.sh`_
+
+Please note that a previously supported pattern where you could assign a
value the global variable `PASS` to have your tests fail **is not supported
anymore**
+
+### Cleanup
+The test runner performs a cleanup after each test case, which includes:
+- Stopping the cluster
+- Killing all task and job managers
+- Reverting config to default (if changed before)
+- Cleaning up log and temp directories
+
+If your test case should do some *additional* cleanup, a common pattern is
to trap a `test_cleanup` function to `EXIT` in your test case like so:
--- End diff --
I would explicitly state that this typically consists of shutting down
external systems used by the e2e test, e.g. Kafka / Elasticsearch.
---
