I'm copying this back to the mailing list because I thought it might be
more generally useful.
> a. what kind of application it is ?.. requirement?
It's our external mail system. Broadly, this consists of Sendmail plus a
whole host of locally developed or open source software working in tandem
to route and perform content filtering on e-mail.
> b. What kinds of testings are being executed.
All sorts. The tests get run after we make changes, but before we go into
in to the test lab. They act as an 'early warning' for a range of different
errors that we can catch before we go in to the trouble of running up a new
configuration in the test lab.
Ideally, if a new configuration passes all these tests then we should find
no problems in the test lab. If we do find problems in the lab we try and
write tests that expose them so we can catch them in the future.
For example, we have tests that:
* Ensure that, after a change to sendmail.cf, sendmail still starts.
* Verifies the syntax of all the Perl and /bin/sh scripts that make up
the configuration.
* Verifies the syntax of any configuration files used.
* Verifies that certain functionality is missing or enabled. For example,
we don't want Sendmail to support the EXPN or VRFY SMTP verbs. So we
have a test that runs Sendmail with each new configuration file, and
connects to it (speaking SMTP over stdin/stdout). Then the test tries
to run the EXPN and VRFY commands, and makes sure that Sendmail returns
the correct error responses.
* Where possible, we perform black and white box testing of individual Perl
functions that perform content filtering. For example, we have code that
performs SPF checks. We can verify that that code is working correctly.
* We have a library of messages that are known to be good or bad in some way
(e.g., they're malformed (bad), or they have an uncommon, but legit,
encoding mechanism (good)). We run all these messages through the
filtering
system, and verify that they pass, or fail, as expected. When someone
requires that we block messages with a certain characteristic (say, because
they contain a particular attachment) or report that messages with a
particular characteristic are being blocked when they shouldn't be, we ask
them to provide samples that we can add to the library. Then we implement
the change, and can verify that we've provided the functionality they want
without affecting how existing types of messages are handled.
> c. How perl is used to automate those tests.
All the tests are written in Perl, using Test::More and other modules from the
Test::* namespace as appropriate. For example, when checking that VRFY fails,
the snippet of code is (once we've got the result string from Sendmail):
line($result, qr/^252 2.5.2 Cannot/, 'ehlo/vrfy has "Cannot" in result
string')
or diag "Result: $result";
All of this is wrapped in a Makefile that has a test target implemented as:
DEFAULT_TESTS= /path/to/a/collection/of/shared/tests/*.t
DEFAULT_TESTS+= t/*.t
DEFAULT_TESTS+= t/*/*.t
TEST_VERBOSE?= 0
TESTS?= ${DEFAULT_TESTS}
test:
${PERL} -e 'use Test::Harness qw(runtests $$verbose); \
$$verbose = ${TEST_VERBOSE}; runtests @ARGV;' ${TESTS}
which lets the developer run things like:
make test
make TEST_VERBOSE=1 test
make test TESTS=/path/to/a/specific/test/to/run
(it's not quite that simple -- we set various environment variables prior to
testing that point to things like the version of Sendmail that we're testing
against, but I've removed those for clarity).
N
