GitHub user duderino opened a pull request:
https://github.com/apache/trafficserver/pull/141
Add initial python-based functional test framework with example test
# Apache Traffic Server Functional Test Framework
The goal of the functional test framework is to allow traffic server
contributors to develop and exercise a reasonable functional test framework
from the comfort of their development environments. This encourages traffic
server developers to write their own functional test cases (it's not the
province of a detached QA team). This also gives traffic server developers
critical test feedback as early in the development process as possible.
The framework consists of a 'process manager' responsible for configuring
and spawning 1+ traffic server processes, a configuration-driven reusable
origin process so you don't have to write your own for every test case (though
sometimes you will), and a small number of utilities for the test cases
themselves. In this context the test cases are traffic server HTTP/SPDY
clients. That is:
* test cases/clients --> traffic server --> (configurable/reusable
origin | ad hoc origin)
## Framework Prerequisites
* Python 2.6+ (the default python on RHEL, CentOS, etc. 6.5+)
* [Twisted](https://twistedmatrix.com/trac/) 14.0.2+
1. [Install pip](https://pip.pypa.io/en/latest/installing.html)
2. Install twisted and its dependencies using pip:
```
$ sudo pip install --upgrade pyOpenSSL zope.interface twisted
```
Individual test cases may add additional dependencies.
## The ProcessManager class:
The ProcessManager class can be found in tests/framework/atstf.py. It is
responsible for configuring, starting, and stopping the processes used by a
test suite.
### ProcessManager Configuration
The ProcessManager is driven by a JSON config file that describes all
managed processes. Individual test cases can also read this config file to
discover server ports, hostnames, etc. As long as test cases do not hardcode
these values they can later be turned against an external functional test
environment.
#### config.json path
By default the ProcessManager looks for a 'config.json' file in the test
suite's current working directory. An alternate location can be passed to its
constructor:
```python
tm = atstf.ProcessManager(config_path="/tmp/config.json")
```
#### trafficserver and other paths
Test suites inside the trafficserver source repo only have to tell the
ProcessManager the path to the source repo root and it will infer the rest:
```python
tm = atstf.ProcessManager(root_path="../..")
```
Test suites outside the trafficserver source repo have to explicitly set
each of the following paths:
```python
tm = atstf.ProcessManager(ats_path="/usr/local/bin/trafficserver",
default_config_path="~/trafficserver/proxy/config",
origin_path="~/trafficserver/tests/framework/origin.py")
```
The last two paths (default_config_path and origin_path) are paths into a
local trafficserver source repo, but we can potentially incorporate those into
a 'devel' package in the future.
#### logging:
You can tell the ProcessManager to log stuff including the stdout and
stderr of the processes it manages. The ProcessManager uses python's [logging
module](https://docs.python.org/2/library/logging.html). If you configure
python's logging module outside of ProcessManager, the ProcessManager will just
use that config. If you don't want to configure the logging module you can
tell the ProcessManager to do it for you by passing it a log_level (e.g.,
logging.INFO):
```python
import logging
tm = atstf.ProcessManager(log_level=logging.DEBUG)
```
#### max start time
By default the ProcessManager will wait for up to 5 seconds for the
processes it manages to start. If any managed process does not start in time,
the ProcessManager will throw an exception which should fail the test.
The ProcessManager considers a managed process to have started when it can
successfully connect to all ports listed in the process's 'interfaces' section.
More on this later.
To change the max start time:
```python
tm = atstf.ProcessManager(max_start_sec=3)
```
#### The config.json
The JSON configuration file used by the ProcessManager has the following
common format:
<pre>
{
"processes": {
<process name>: {
"type": <"ats"|"origin"|"python"|"other">,
"spawn": <true|false>
"interfaces": {
<interface name>: {
"hostname": <hostname|ipv4 addr|ipv6 addr>,
"port": <port>
}
},
<other stuff specific to the process type>
}
},
<optional: other stuff specific to the test suite. Ignored by
ProcessManager>
}
</pre>
Because this must be valid JSON, process names must be unique. They will
be incorporated into log messages and into exceptions thrown by the
ProcessManager.
The process types supported by the ProcessManager are described below.
Different process types may support additional configuration options.
The 'spawn' attribute defaults to true. If set to false the
ProcessManager will not manage the process. This is intended to support
turning a functional test case against an externally managed set of processes.
The test case code can still discover hostnames and ports from the config.json
without a code change.
The optional 'interfaces' attribute details which ports will be opened by a
managed process. If specified the ProcessManager will block caller until
either all ports are up or the 'max start time' is exceeded.
#### The config.json Process Types
The ProcessManager currently supports the following process types:
##### Process Type 'ats'
'ats' processes support additional config.json options.
Example:
```json
{
"servers": {
"ats1": {
"type": "ats",
"root": "ats1",
"interfaces": {
"http": {
"type": "http",
"hostname": "localhost",
"port": 8080
}
},
"config": {
"records.config": [
{
"match": "^CONFIG proxy.config.diags.debug.enabled",
"substitute": "CONFIG
proxy.config.diags.debug.enabled INT 1"
}
],
"remap.config": [
{
"append": "map /foo/bar
http://localhost:8082/foo/bar";
}
]
}
}
}
}
```
LIMITATION: Only the 'http' interface type is presently supported. Need
to add HTTPS and SPDY ASAP.
The ProcessManager will run the trafficserver in the 'root' dir. This is
the same as the trafficserver 'TS_ROOT' environment variable. If you
configure multiple 'ats' processes, be sure to assign each of them a distinct
root dir. The root dir should be relative to the test case dir and MUST NOT
CONTAIN ANYTHING IMPORTANT! If ats crashes and the system is configured to
capture cores, they should appear in the root dir.
Before trafficserver is started:
* The ProcessManager will rm -rf the 'root' dir if it already exists,
then recreate it with a minimal directory structure (a few var dirs and an etc
dir)
* The ProcessManager will copy over all the default config files and
body factory files from proxy/config
* If the config.json has a 'config' section, the ProcessManager will
transform the copied default config files accordingly. On a per config file
basis:
* match/substitue directives will replace any line that matches a
regex (match) with a substitution line
* append directives will simply add lines to the end of the config
file.
After trafficserver is started:
* trafficserver will create transient files in the root/var dirs,
including log files
##### Process Type 'origin'
A configurable origin server. Multiple types of 'origins' can be
specified on a per-path basis, but currently only a 'chunkedresponse' type has
been developed. The 'chunkedresponse' origin type supports the following
configuration:
1. response status code
1. response headers
1. number of chunks to send in the response
1. size of each chunk
1. byte value of each byte in chunk
1. seconds to wait before sending the first chunk
1. seconds to wait before sending all subsequent chunks
Example:
```json
{
"servers": {
"origin1": {
"type": "origin",
"interfaces": {
"http": {
"type": "http",
"hostname": "localhost",
"port": 8082
}
},
"actions": {
"GET": {
"/foo/bar": {
"type": "chunkedresponse",
"status_code": 200,
"headers": {
"content-type": "text/plain"
},
"delay_first_chunk_sec": 1,
"chunk_size_bytes": 1024,
"num_chunks": 10,
"delay_between_chunk_sec": 0,
"chunk_byte_value": 99
}
}
}
}
}
}
```
##### Process Type 'python'
The ProcessManager can manage ad hoc origins written in python providing
they exit cleanly when sent a SIGTERM. If they specify an 'interfaces'
section then the ProcessManager will also wait for them to start.
Example:
```json
{
"servers": {
"python1": {
"type": "python",
"script": "ad_hoc_origin.py",
"args": [
"python1",
"config.json"
],
"interfaces": {
"http": {
"type": "http",
"hostname": "localhost",
"port": 8082
}
}
}
}
}
```
##### Process Type 'other'
Finally the ProcessManager can manage any executable, again providing it
exits cleanly when sent a SIGTERM. If it specifies an 'interface' secion then
the ProcessManager will also wait for it to start.
Example:
```json
{
"servers": {
"other1": {
"type": "other",
"spawn": false,
"executable": "/usr/local/apache2/bin/httpd",
"args": [
"-f",
"/usr/local/apache2/conf/httpd.conf",
"-k",
"start",
"-x"
],
"interfaces": {
"http": {
"type": "http",
"hostname": "localhost",
"port": 8082
}
}
}
}
}
```
## Writing Test Cases
### Using the 'unittest' and 'twisted' modules
Test cases should use [the Python Standard Library's unittest
module](https://docs.python.org/2/library/unittest.html). Test cases also need
to send traffic to trafficserver. This particular example uses the [Python
twisted framework](https://twistedmatrix.com/trac/), but other approaches are
possible.
The main advantage of twisted is that it can be used for lightweight load
testing. The main disadvantage is that Python's unittest module implements
assertions on top of exceptions. Failed assertions in twisted callbacks will
be eaten by the twisted framework and will not be incorporated into the test
results.
This example works around this issue by saving the state of its interaction
with trafficserver, then performing all assertions in the main thread against
the saved state at the end of the test case:
```python
import sys
import unittest
from twisted.internet import reactor
from twisted.internet.defer import Deferred
from twisted.internet.protocol import Protocol
from twisted.web.client import Agent
from twisted.web.http_headers import Headers
sys.path = ['../framework'] + sys.path
import atstf
class ExampleTwistedTest(unittest.TestCase):
def test_ats1(self):
#
# Read test case configuration from the same config file used to
start ATS and origin processes.
#
conf = atstf.parse_config()
ats1_conf = conf['processes']['ats1']
hostname = ats1_conf['interfaces']['http']['hostname']
port = ats1_conf['interfaces']['http']['port']
origin_conf = conf['processes']['origin1']
expected_status_code =
origin_conf['actions']['GET']['/foo/bar']['status_code']
chunk_size_bytes =
origin_conf['actions']['GET']['/foo/bar']['chunk_size_bytes']
num_chunks = origin_conf['actions']['GET']['/foo/bar']['num_chunks']
expected_bytes_received = chunk_size_bytes * num_chunks
#
# Create a twisted HTTP client that will interact with ATS and save
all details of the interaction. We'll
# later assert that the saved details are correct. Ideally we'd
just throw assertions into the client itself,
# but sadly assertions are implemented with exceptions and twisted
eats any exception raised by our callbacks.
#
# See
https://twistedmatrix.com/documents/current/web/howto/client.html for details
on writing HTTP clients
#
agent = Agent(reactor)
request = agent.request('GET', "http://%s:%d/foo/bar"; %
(hostname.encode("utf-8"), port),
Headers({'User-Agent':
['ExampleTwistedTest']}), None)
response_handler = ResponseHandler(reactor)
request.addCallback(response_handler.handle_response)
request.addCallback(response_handler.handle_completion)
request.addErrback(response_handler.handle_error)
reactor.run()
#
# Now assert
#
self.assertEqual(expected_status_code,
response_handler.get_status_code())
self.assertEqual(expected_bytes_received,
response_handler.get_body_handler().get_bytes_received())
self.assertEqual("Response body fully received", \
response_handler.get_body_handler().get_reason().getErrorMessage())
class ResponseBodyHandler(Protocol):
def __init__(self, deferred):
self.__deferred = deferred
self.__bytes_received = 0
self.__reason = None
def dataReceived(self, data):
self.__bytes_received += len(data)
def connectionLost(self, reason):
#
# This function name is highly misleading. According to the web
client docs:
#
# When the body has been completely delivered, the protocol's
connectionLost method is called. It is important
# to inspect the Failure passed to connectionLost . If the response
body has been completely received, the
# failure will wrap a twisted.web.client.ResponseDone exception.
This indicates that it is known that all data
# has been received. It is also possible for the failure to wrap a
twisted.web.http.PotentialDataLoss exception:
# this indicates that the server framed the response such that
there is no way to know when the entire response
# body has been received. Only HTTP/1.0 servers should behave this
way. Finally, it is possible for the
# exception to be of another type, indicating guaranteed data loss
for some reason (a lost connection, a memory
# error, etc).
#
self.__reason = reason
self.__deferred.callback(None)
def get_bytes_received(self):
return self.__bytes_received
def get_reason(self):
return self.__reason
class ResponseHandler:
def __init__(self, reactor):
self.__reactor = reactor
self.__deferred = Deferred()
self.__body_handler = ResponseBodyHandler(self.__deferred)
self.__error = None
self.__status_code = None
def handle_response(self, response):
self.__status_code = response.code
response.deliverBody(self.__body_handler)
return self.__deferred
def handle_completion(self):
self.__reactor.stop()
def handle_error(self, error):
self.__error = error
self.__reactor.stop()
def get_body_handler(self):
return self.__body_handler
def get_error(self):
return self.__error
def get_status_code(self):
return self.__status_code
```
### Pulling it all together with a Test Suite
Multiple test cases should be run together in a test suite and the results
of the entire suite should be summarized in a JUnit report suitable for
Jenkins. This is also a good place to instantiate and start the ProcessManager
(start it before any test case is invoked, stop it after all test cases
finish).
Example:
```python
#!/bin/env python
import sys
import logging
sys.path = ['../framework'] + sys.path
import atstf
from twisted_test import ExampleTwistedTest
test_cases = [ExampleTwistedTest]
if __name__ == '__main__':
# Spawn ATS and origin processes
tm = atstf.ProcessManager(root_path="../..", config_path="config.json",
log_level=logging.DEBUG, max_start_sec=5)
try:
tm.start()
# Run the test suite and generate a JUnit XML report
if atstf.run_tests(test_cases=test_cases, name='example',
report='report.xml'):
sys.exit(0)
else:
sys.exit(1)
finally:
tm.stop()
```
Here the important bits are:
1. Initializing the ProcessManager and starting it before the running
the test suite
1. Passing all the test case classes to run_tests(). This will auto
discover all test cases in the test classes, run them, and spit the results out
in a report.xml file in the cwd.
1. Exiting zero if the run_tests() method returns True, non-zero
otherwise
Not so important is explicitly stopping the ProcessManager. It's good
practice, but the ProcessManager will stop automatically on process exit.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yahoo/trafficserver
blattj_functional_test_framework_redux
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/trafficserver/pull/141.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #141
----
commit 077d1891f39f2978e9b5936e4a5dbaa106bc4804
Author: Joshua Blatt <[email protected]>
Date: 2014-11-11T20:30:55Z
Add initial python-based functional test framework with example test case
illustrating use of twisted.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
