Anton, Nikolay,
I want to share some more findings about ducktests I've stubmled upon
during porting them to Tiden.
First problem was that GridGain Tiden-based tests by default use real
production-like configuration for Ignite nodes, notably:
- persitence enabled
- ~120 caches in ~40 groups
- data set size around 1M keys per each cache
- primitive and PoJo cache values
- extensive use of query entities (indices)
When I've tried to run 4 nodes with such configuration in docker, my
notebook nearly burns. Nevertheless, grid was starting and working OK,
but for one little 'but': each successive version under test was
starting slower and slower.
The 2.7.6 was the fastest, 2.8.0 and 2.8.1 a little bit slower, and your
fork (2.9.0-SNAPSHOT) failed to start 4 persistence-enabled nodes within
default 120 seconds timeout. In order to mimick behavior of your tests I
had to turn off persistence and use only 1 cache too.
It's a pity that you completely ignore persistence and indices in your
ducktests, otherwise you would quickly stuck into same limitation.
I hope in the nearest time I would adopt Tiden docker PoC to our
TeamCity and we'll try to git-bisect in order to find where this
slowdown comes from. After that I'll file a bug to IGNITE Jira.
Another problem with your rebalance benchmark is it's low accuracy due
to granularity of measurements.
You don't actually measure rebalance time, you measure time that takes
you to find a specific string in logs, that's confusing.
The scenario of your test is as follows:
1. start 3 server nodes
2. start 1 data loading client, preload a data, stop client
3. start 1 more server node
4. wait till server joins topology
5. wait till this server node completes exchange and write
'rebalanced=true, wasRebalanced=false' message to log
6. report time was taken by step 5 as 'Rebalance time'
Confusing thing here is that 'wait till' implementation - you actually
continuously re-scan logs sleeping each second and wait till message
appear. So, that means that rebalance time is at least of second
granularity or even higher, though it is reported with nanosecond
precision.
But for such lightweight configuration (single in-memory cache) and such
small set of data (1M keys only), rebalancing is very fast, and usually
performs under 1 second or just slightly slower.
Before waiting for rebalance message you first wait for topology message
and that wait also takes time to execute.
So, at the time Python part of the test performs first scan of the logs,
rebalancing is in most cases already done and time you report as
'0.0760810375213623' is actually the time to execute logs scanning code.
However, if rebalancing perform just a little bit slower after topology
update, then first scan of logs is failed, you sleep for whole one
second and rescan logs and there you got your message and report it as
'1.02205491065979'.
Under different conditions, dockerized application may run a little
slower or a little faster, that depends on overall system load, free
memory, etc. I've tried to increase load on my laptop by running browser
or maven build, and time to scan logs may fluctuate from 0.02 to 0.09 or
even 1.02 seconds. Note, that in CI environment, high system load from
tenants is a quite ordinary situation.
Suppose we adopted rebalance improvements and all versions after 2.9.0
would perform within 1 second just as 2.9.0 itself. Then your benchmark
would either report false negative (e.g. 0.02 for master and 0.03 for
PR), while actually on next re-run it would pass (e.g. 0.07 for master
and 0.03 for PR). That's not quite the 'stable and non-flacky' test
Ignite community wants.
What suggestions do you have to improve benchmark measurement accuracy?
A third question is about PME free switch benchmark. Under some
conditions, LongTxStreamerApplication actually hangs up PME. It need to
be investigated further, but either this was due to persistence enabled
or due to missing -DIGNITE_ALLOW_ATOMIC_OPS_IN_TX=false
Can you share some details about IGNITE_ALLOW_ATOMIC_OPS_IN_TX option?
Also, have you had performed a test of PME free switch with
persistence-enabled caches?
On 09.07.2020 10:11, Max Shonichev wrote:
Anton,
well, strange thing, but clean up and rerun helped.
Ubuntu 18.04
====================================================================================================
SESSION REPORT (ALL TESTS)
ducktape version: 0.7.7
session_id: 2020-07-06--003
run time: 4 minutes 44.835 seconds
tests run: 5
passed: 5
failed: 0
ignored: 0
====================================================================================================
test_id:
ignitetest.tests.benchmarks.add_node_rebalance_test.AddNodeRebalanceTest.test_add_node.version=2.8.1
status: PASS
run time: 41.927 seconds
{"Rebalanced in (sec)": 1.02205491065979}
----------------------------------------------------------------------------------------------------
test_id:
ignitetest.tests.benchmarks.add_node_rebalance_test.AddNodeRebalanceTest.test_add_node.version=dev
status: PASS
run time: 51.985 seconds
{"Rebalanced in (sec)": 0.0760810375213623}
----------------------------------------------------------------------------------------------------
test_id:
ignitetest.tests.benchmarks.pme_free_switch_test.PmeFreeSwitchTest.test.version=2.7.6
status: PASS
run time: 1 minute 4.283 seconds
{"Streamed txs": "1900", "Measure duration (ms)": "34818", "Worst
latency (ms)": "31035"}
----------------------------------------------------------------------------------------------------
test_id:
ignitetest.tests.benchmarks.pme_free_switch_test.PmeFreeSwitchTest.test.version=dev
status: PASS
run time: 1 minute 13.089 seconds
{"Streamed txs": "73134", "Measure duration (ms)": "35843", "Worst
latency (ms)": "139"}
----------------------------------------------------------------------------------------------------
test_id:
ignitetest.tests.spark_integration_test.SparkIntegrationTest.test_spark_client
status: PASS
run time: 53.332 seconds
----------------------------------------------------------------------------------------------------
MacBook
================================================================================
SESSION REPORT (ALL TESTS)
ducktape version: 0.7.7
session_id: 2020-07-06--001
run time: 6 minutes 58.612 seconds
tests run: 5
passed: 5
failed: 0
ignored: 0
================================================================================
test_id:
ignitetest.tests.benchmarks.add_node_rebalance_test.AddNodeRebalanceTest.test_add_node.version=2.8.1
status: PASS
run time: 48.724 seconds
{"Rebalanced in (sec)": 3.2574470043182373}
--------------------------------------------------------------------------------
test_id:
ignitetest.tests.benchmarks.add_node_rebalance_test.AddNodeRebalanceTest.test_add_node.version=dev
status: PASS
run time: 1 minute 23.210 seconds
{"Rebalanced in (sec)": 2.165921211242676}
--------------------------------------------------------------------------------
test_id:
ignitetest.tests.benchmarks.pme_free_switch_test.PmeFreeSwitchTest.test.version=2.7.6
status: PASS
run time: 1 minute 12.659 seconds
{"Streamed txs": "642", "Measure duration (ms)": "33177", "Worst latency
(ms)": "31063"}
--------------------------------------------------------------------------------
test_id:
ignitetest.tests.benchmarks.pme_free_switch_test.PmeFreeSwitchTest.test.version=dev
status: PASS
run time: 1 minute 57.257 seconds
{"Streamed txs": "32924", "Measure duration (ms)": "48252", "Worst
latency (ms)": "1010"}
--------------------------------------------------------------------------------
test_id:
ignitetest.tests.spark_integration_test.SparkIntegrationTest.test_spark_client
status: PASS
run time: 1 minute 36.317 seconds
=============
while relative numbers proportion remains the same for different Ignite
versions, absolute number for mac/linux differ more then twice.
I'm finalizing code with 'local Tiden' appliance for your tests. PR
would be ready soon.
Have you had a chance to deploy ducktests in bare metal?
On 06.07.2020 14:27, Anton Vinogradov wrote:
Max,
Thanks for the check!
Is it OK for those tests to fail?
No.
I see really strange things at logs.
Looks like you have concurrent ducktests run started not expected
services,
and this broke the tests.
Could you please clean up the docker (use clean-up script [1]).
Compile sources (use script [2]) and rerun the tests.
[1]
https://github.com/anton-vinogradov/ignite/blob/dc98ee9df90b25eb5d928090b0e78b48cae2392e/modules/ducktests/tests/docker/clean_up.sh
[2]
https://github.com/anton-vinogradov/ignite/blob/3c39983005bd9eaf8cb458950d942fb592fff85c/scripts/build.sh
On Mon, Jul 6, 2020 at 12:03 PM Nikolay Izhikov <nizhi...@apache.org>
wrote:
Hello, Maxim.
Thanks for writing down the minutes.
There is no such thing as «Nikolay team» on the dev-list.
I propose to focus on product requirements and what we want to gain from
the framework instead of taking into account the needs of some team.
Can you, please, write down your version of requirements so we can
reach a
consensus on that and therefore move to the discussion of the
implementation?
6 июля 2020 г., в 11:18, Max Shonichev <mshon...@yandex.ru> написал(а):
Yes, Denis,
common ground seems to be as follows:
Anton Vinogradov and Nikolay Izhikov would try to prepare and run PoC
over physical hosts and share benchmark results. In the meantime,
while I
strongly believe that dockerized approach to benchmarking is a road to
misleading and false positives, I'll prepare a PoC of Tiden in
dockerized
environment to support 'fast development prototyping' usecase Nikolay
team
insist on. It should be a matter of few days.
As a side note, I've run Anton PoC locally and would like to have some
comments about results:
Test system: Ubuntu 18.04, docker 19.03.6
Test commands:
git clone -b ignite-ducktape g...@github.com:anton-vinogradov/ignite.git
cd ignite
mvn clean install -DskipTests -Dmaven.javadoc.skip=true
-Pall-java,licenses,lgpl,examples,!spark-2.4,!spark,!scala
cd modules/ducktests/tests/docker
./run_tests.sh
Test results:
====================================================================================================
SESSION REPORT (ALL TESTS)
ducktape version: 0.7.7
session_id: 2020-07-05--004
run time: 7 minutes 36.360 seconds
tests run: 5
passed: 3
failed: 2
ignored: 0
====================================================================================================
test_id:
ignitetest.tests.benchmarks.add_node_rebalance_test.AddNodeRebalanceTest.test_add_node.version=2.8.1
status: FAIL
run time: 3 minutes 12.232 seconds
----------------------------------------------------------------------------------------------------
test_id:
ignitetest.tests.benchmarks.pme_free_switch_test.PmeFreeSwitchTest.test.version=2.7.6
status: FAIL
run time: 1 minute 33.076 seconds
Is it OK for those tests to fail? Attached is full test report
On 02.07.2020 17:46, Denis Magda wrote:
Folks,
Please share the summary of that Slack conversation here for records
once
you find common ground.
-
Denis
On Thu, Jul 2, 2020 at 3:22 AM Nikolay Izhikov <nizhi...@apache.org>
wrote:
Igniters.
All who are interested in integration testing framework discussion
are
welcome into slack channel -
https://join.slack.com/share/zt-fk2ovehf-TcomEAwiXaPzLyNKZbmfzw?cdn_fallback=2
2 июля 2020 г., в 13:06, Anton Vinogradov <a...@apache.org>
написал(а):
Max,
Thanks for joining us.
1. tiden can deploy artifacts by itself, while ducktape relies on
dependencies being deployed by external scripts.
No. It is important to distinguish development, deploy, and
orchestration.
All-in-one solutions have extremely limited usability.
As to Ducktests:
Docker is responsible for deployments during development.
CI/CD is responsible for deployments during release and nightly
checks.
It's up to the team to chose AWS, VM, BareMetal, and even OS.
Ducktape is responsible for orchestration.
2. tiden can execute actions over remote nodes in real parallel
fashion,
while ducktape internally does all actions sequentially.
No. Ducktape may start any service in parallel. See Pme-free
benchmark
[1] for details.
if we used ducktape solution we would have to instead prepare some
deployment scripts to pre-initialize Sberbank hosts, for example,
with
Ansible or Chef.
Sure, because a way of deploy depends on infrastructure.
How can we be sure that OS we use and the restrictions we have
will be
compatible with Tiden?
You have solved this deficiency with docker by putting all
dependencies
into one uber-image ...
and
I guess we all know about docker hyped ability to run over
distributed
virtual networks.
It is very important not to confuse the test's development (docker
image
you're talking about) and real deployment.
If we had stopped and started 5 nodes one-by-one, as ducktape does
All actions can be performed in parallel.
See how Ducktests [2] starts cluster in parallel for example.
[1]
https://github.com/apache/ignite/pull/7967/files#diff-59adde2a2ab7dc17aea6c65153dfcda7R84
[2]
https://github.com/apache/ignite/pull/7967/files#diff-d6a7b19f30f349d426b8894a40389cf5R79
On Thu, Jul 2, 2020 at 1:00 PM Nikolay Izhikov <nizhi...@apache.org>
wrote:
Hello, Maxim.
1. tiden can deploy artifacts by itself, while ducktape relies on
dependencies being deployed by external scripts
Why do you think that maintaining deploy scripts coupled with the
testing framework is an advantage?
I thought we want to see and maintain deployment scripts separate
from
the testing framework.
2. tiden can execute actions over remote nodes in real parallel
fashion, while ducktape internally does all actions sequentially.
Can you, please, clarify, what actions do you have in mind?
And why we want to execute them concurrently?
Ignite node start, Client application execution can be done
concurrently
with the ducktape approach.
If we used ducktape solution we would have to instead prepare some
deployment scripts to pre-initialize Sberbank hosts, for example,
with
Ansible or Chef
We shouldn’t take some user approach as an argument in this
discussion.
Let’s discuss a general approach for all users of the Ignite. Anyway,
what
is wrong with the external deployment script approach?
We, as a community, should provide several ways to run integration
tests
out-of-the-box AND the ability to customize deployment regarding the
user
landscape.
You have solved this deficiency with docker by putting all
dependencies into one uber-image and that looks like simple and
elegant
solution however, that effectively limits you to single-host testing.
Docker image should be used only by the Ignite developers to test
something locally.
It’s not intended for some real-world testing.
The main issue with the Tiden that I see, it tested and
maintained as
a
closed source solution.
This can lead to the hard to solve problems when we start using and
maintaining it as an open-source solution.
Like, how many developers used Tiden? And how many of developers
were
not authors of the Tiden itself?
2 июля 2020 г., в 12:30, Max Shonichev <mshon...@yandex.ru>
написал(а):
Anton, Nikolay,
Let's agree on what we are arguing about: whether it is about "like
or
don't like" or about technical properties of suggested solutions.
If it is about likes and dislikes, then the whole discussion is
meaningless. However, I hope together we can analyse pros and cons
carefully.
As far as I can understand now, two main differences between
ducktape
and tiden is that:
1. tiden can deploy artifacts by itself, while ducktape relies on
dependencies being deployed by external scripts.
2. tiden can execute actions over remote nodes in real parallel
fashion, while ducktape internally does all actions sequentially.
As for me, these are very important properties for distributed
testing
framework.
First property let us easily reuse tiden in existing
infrastructures,
for example, during Zookeeper IEP testing at Sberbank site we used
the
same
tiden scripts that we use in our lab, the only change was putting a
list of
hosts into config.
If we used ducktape solution we would have to instead prepare some
deployment scripts to pre-initialize Sberbank hosts, for example,
with
Ansible or Chef.
You have solved this deficiency with docker by putting all
dependencies into one uber-image and that looks like simple and
elegant
solution,
however, that effectively limits you to single-host testing.
I guess we all know about docker hyped ability to run over
distributed
virtual networks. We used to go that way, but quickly found that
it is
more
of the hype than real work. In real environments, there are problems
with
routing, DNS, multicast and broadcast traffic, and many others, that
turn
docker-based distributed solution into a fragile hard-to-maintain
monster.
Please, if you believe otherwise, perform a run of your PoC over at
least two physical hosts and share results with us.
If you consider that one physical docker host is enough, please,
don't
overlook that we want to run real scale scenarios, with 50-100 cache
groups, persistence enabled and a millions of keys loaded.
Practical limit for such configurations is 4-6 nodes per single
physical host. Otherwise, tests become flaky due to resource
starvation.
Please, if you believe otherwise, perform at least a 10 of runs of
your PoC with other tests running at TC (we're targeting TeamCity,
right?)
and share results so we could check if the numbers are reproducible.
I stress this once more: functional integration tests are OK to run
in
Docker and CI, but running benchmarks in Docker is a big NO GO.
Second property let us write tests that require real-parallel
actions
over hosts.
For example, agreed scenario for PME benchmarkduring "PME
optimization
stream" was as follows:
- 10 server nodes, preloaded with 1M of keys
- 4 client nodes perform transactional load (client nodes
physically
separated from server nodes)
- during load:
-- 5 server nodes stopped in parallel
-- after 1 minute, all 5 nodes are started in parallel
- load stopped, logs are analysed for exchange times.
If we had stopped and started 5 nodes one-by-one, as ducktape does,
then partition map exchange merge would not happen and we could not
have
measured PME optimizations for that case.
These are limitations of ducktape that we believe as a more
important
argument "against" than you provide "for".
On 30.06.2020 14:58, Anton Vinogradov wrote:
Folks,
First, I've created PR [1] with ducktests improvements
PR contains the following changes
- Pme-free switch proof-benchmark (2.7.6 vs master)
- Ability to check (compare with) previous releases (eg. 2.7.6 &
2.8)
- Global refactoring
-- benchmarks javacode simplification
-- services python and java classes code deduplication
-- fail-fast checks for java and python (eg. application should
explicitly write it finished with success)
-- simple results extraction from tests and benchmarks
-- javacode now configurable from tests/benchmarks
-- proper SIGTERM handling at javacode (eg. it may finish last
operation and log results)
-- docker volume now marked as delegated to increase execution
speed
for mac & win users
-- Ignite cluster now start in parallel (start speed-up)
-- Ignite can be configured at test/benchmark
- full and module assembly scripts added
Great job done! But let me remind one of Apache Ignite principles:
week of thinking save months of development.
Second, I'd like to propose to accept ducktests [2] (ducktape
integration) as a target "PoC check & real topology benchmarking
tool".
Ducktape pros
- Developed for distributed system by distributed system
developers.
So does Tiden
- Developed since 2014, stable.
Tiden is also pretty stable, and development start date is not a
good
argument, for example pytest is since 2004, pytest-xdist (plugin for
distributed testing) is since 2010, but we don't see it as a
alternative at
all.
- Proven usability by usage at Kafka.
Tiden is proven usable by usage at GridGain and Sberbank
deployments.
Core, storage, sql and tx teams use benchmark results provided by
Tiden on a daily basis.
- Dozens of dozens tests and benchmarks at Kafka as a great
example
pack.
We'll donate some of our suites to Ignite as I've mentioned in
previous letter.
- Built-in Docker support for rapid development and checks.
False, there's no specific 'docker support' in ducktape itself, you
just wrap it in docker by yourself, because ducktape is lacking
deployment
abilities.
- Great for CI automation.
False, there's no specific CI-enabled features in ducktape.
Tiden, on
the other hand, provide generic xUnit reporting format, which is
supported
by both TeamCity and Jenkins. Also, instead of using private keys,
Tiden
can use SSH agent, which is also great for CI, because both
TeamCity and Jenkins store keys in secret storage available only
for
ssh-agent and only for the time of the test.
As an additional motivation, at least 3 teams
- IEP-45 team (to check crash-recovery speed-up (discovery and
Zabbix
speed-up))
- Ignite SE Plugins team (to check plugin's features does not
slow-down or broke AI features)
- Ignite SE QA team (to append already developed
smoke/load/failover
tests to AI codebase)
Please, before recommending your tests to other teams, provide
proofs
that your tests are reproducible in real environment.
now, wait for ducktest merge to start checking cases they
working on
in AI way.
Thoughts?
Let us together review both solutions, we'll try to run your
tests in
our lab, and you'll try to at least checkout tiden and see if same
tests
can be implemented with it?
[1] https://github.com/apache/ignite/pull/7967
[2] https://github.com/apache/ignite/tree/ignite-ducktape
On Tue, Jun 16, 2020 at 12:22 PM Nikolay Izhikov <
nizhi...@apache.org
<mailto:nizhi...@apache.org>> wrote:
Hello, Maxim.
Thank you for so detailed explanation.
Can we put the content of this discussion somewhere on the
wiki?
So It doesn’t get lost.
I divide the answer in several parts. From the requirements to
the
implementation.
So, if we agreed on the requirements we can proceed with the
discussion of the implementation.
1. Requirements:
The main goal I want to achieve is *reproducibility* of the
tests.
I’m sick and tired with the zillions of flaky, rarely
failed, and
almost never failed tests in Ignite codebase.
We should start with the simplest scenarios that will be as
reliable
as steel :)
I want to know for sure:
- Is this PR makes rebalance quicker or not?
- Is this PR makes PME quicker or not?
So, your description of the complex test scenario looks as
a next
step to me.
Anyway, It’s cool we already have one.
The second goal is to have a strict test lifecycle as we
have in
JUnit and similar frameworks.
> It covers production-like deployment and running a
scenarios
over
a single database instance.
Do you mean «single cluster» or «single host»?
2. Existing tests:
> A Combinator suite allows to run set of operations
concurrently
over given database instance.
> A Consumption suite allows to run a set production-like
actions
over given set of Ignite/GridGain versions and compare test
metrics
across versions
> A Yardstick suite
> A Stress suite that simulates hardware environment
degradation
> An Ultimate, DR and Compatibility suites that performs
functional
regression testing
> Regression
Great news that we already have so many choices for testing!
Mature test base is a big +1 for Tiden.
3. Comparison:
> Criteria: Test configuration
> Ducktape: single JSON string for all tests
> Tiden: any number of YaML config files, command line option
for
fine-grained test configuration, ability to select/modify
tests
behavior based on Ignite version.
1. Many YAML files can be hard to maintain.
2. In ducktape, you can set parameters via «—parameters»
option.
Please, take a look at the doc [1]
> Criteria: Cluster control
> Tiden: additionally can address cluster as a whole and
execute
remote commands in parallel.
It seems we implement this ability in the PoC, already.
> Criteria: Test assertions
> Tiden: simple asserts, also few customized assertion
helpers.
> Ducktape: simple asserts.
Can you, please, be more specific.
What helpers do you have in mind?
Ducktape has an asserts that waits for logfile messages or
some
process finish.
> Criteria: Test reporting
> Ducktape: limited to its own text/HTML format
Ducktape have
1. Text reporter
2. Customizable HTML reporter
3. JSON reporter.
We can show JSON with the any template or tool.
> Criteria: Provisioning and deployment
> Ducktape: can provision subset of hosts from cluster for
test
needs. However, that means, that test can’t be scaled without
test
code changes. Does not do any deploy, relies on external
means,
e.g.
pre-packaged in docker image, as in PoC.
This is not true.
1. We can set explicit test parameters(node number) via
parameters.
We can increase client count of cluster size without test code
changes.
2. We have many choices for the test environment. These
choices
are
tested and used in other projects:
* docker
* vagrant
* private cloud(ssh access)
* ec2
Please, take a look at Kafka documentation [2]
> I can continue more on this, but it should be enough for
now:
We need to go deeper! :)
[1]
https://ducktape-docs.readthedocs.io/en/latest/run_tests.html#options
[2]
https://github.com/apache/kafka/tree/trunk/tests#ec2-quickstart
> 9 июня 2020 г., в 17:25, Max A. Shonichev
<mshon...@yandex.ru
<mailto:mshon...@yandex.ru>> написал(а):
>
> Greetings, Nikolay,
>
> First of all, thank you for you great effort preparing
PoC of
integration testing to Ignite community.
>
> It’s a shame Ignite did not have at least some such
tests yet,
however, GridGain, as a major contributor to Apache Ignite
had a
profound collection of in-house tools to perform
integration and
performance testing for years already and while we slowly
consider
sharing our expertise with the community, your initiative
makes
us
drive that process a bit faster, thanks a lot!
>
> I reviewed your PoC and want to share a little about
what we
do
on our part, why and how, hope it would help community take
proper
course.
>
> First I’ll do a brief overview of what decisions we made
and
what
we do have in our private code base, next I’ll describe
what we
have
already donated to the public and what we plan public next,
then
I’ll compare both approaches highlighting deficiencies in
order
to
spur public discussion on the matter.
>
> It might seem strange to use Python to run Bash to run Java
applications because that introduces IT industry best of
breed’ –
the Python dependency hell – to the Java application code
base.
The
only strangest decision one can made is to use Maven to run
Docker
to run Bash to run Python to run Bash to run Java, but
desperate
times call for desperate measures I guess.
>
> There are Java-based solutions for integration testing
exists,
e.g. Testcontainers [1], Arquillian [2], etc, and they
might go
well
for Ignite community CI pipelines by them selves. But we also
wanted
to run performance tests and benchmarks, like the dreaded PME
benchmark, and this is solved by totally different set of
tools
in
Java world, e.g. Jmeter [3], OpenJMH [4], Gatling [5], etc.
>
> Speaking specifically about benchmarking, Apache Ignite
community
already has Yardstick [6], and there’s nothing wrong with
writing
PME benchmark using Yardstick, but we also wanted to be
able to
run
scenarios like this:
> - put an X load to a Ignite database;
> - perform an Y set of operations to check how Ignite copes
with
operations under load.
>
> And yes, we also wanted applications under test be deployed
‘like
in a production’, e.g. distributed over a set of hosts. This
arises
questions about provisioning and nodes affinity which I’ll
cover
in
detail later.
>
> So we decided to put a little effort to build a simple
tool to
cover different integration and performance scenarios, and
our QA
lab first attempt was PoC-Tester [7], currently open source
for
all
but for reporting web UI. It’s a quite simple to use 95%
Java-based
tool targeted to be run on a pre-release QA stage.
>
> It covers production-like deployment and running a
scenarios
over
a single database instance. PoC-Tester scenarios consists of a
sequence of tasks running sequentially or in parallel.
After all
tasks complete, or at any time during test, user can run logs
collection task, logs are checked against exceptions and a
summary
of found issues and task ops/latency statistics is
generated at
the
end of scenario. One of the main PoC-Tester features is its
fire-and-forget approach to task managing. That is, you can
deploy
a
grid and left it running for weeks, periodically firing some
tasks
onto it.
>
> During earliest stages of PoC-Tester development it becomes
quite
clear that Java application development is a tedious
process and
architecture decisions you take during development are slow
and
hard
to change.
> For example, scenarios like this
> - deploy two instances of GridGain with master-slave data
replication configured;
> - put a load on master;
> - perform checks on slave,
> or like this:
> - preload a 1Tb of data by using your favorite tool of
choice
to
an Apache Ignite of version X;
> - run a set of functional tests running Apache Ignite
version
Y
over preloaded data,
> do not fit well in the PoC-Tester workflow.
>
> So, this is why we decided to use Python as a generic
scripting
language of choice.
>
> Pros:
> - quicker prototyping and development cycles
> - easier to find DevOps/QA engineer with Python skills than
one
with Java skills
> - used extensively all over the world for DevOps/CI
pipelines
and
thus has rich set of libraries for all possible integration
uses
cases.
>
> Cons:
> - Nightmare with dependencies. Better stick to specific
language/libraries version.
>
> Comparing alternatives for Python-based testing
framework we
have
considered following requirements, somewhat similar to what
you’ve
mentioned for Confluent [8] previously:
> - should be able run locally or distributed (bare metal
or in
the
cloud)
> - should have built-in deployment facilities for
applications
under test
> - should separate test configuration and test code
> -- be able to easily reconfigure tests by simple
configuration
changes
> -- be able to easily scale test environment by simple
configuration changes
> -- be able to perform regression testing by simple
switching
artifacts under test via configuration
> -- be able to run tests with different JDK version by
simple
configuration changes
> - should have human readable reports and/or reporting tools
integration
> - should allow simple test progress monitoring, one does
not
want
to run 6-hours test to find out that application actually
crashed
during first hour.
> - should allow parallel execution of test actions
> - should have clean API for test writers
> -- clean API for distributed remote commands execution
> -- clean API for deployed applications start / stop and
other
operations
> -- clean API for performing check on results
> - should be open source or at least source code should
allow
ease
change or extension
>
> Back at that time we found no better alternative than to
write
our own framework, and here goes Tiden [9] as GridGain
framework
of
choice for functional integration and performance testing.
>
> Pros:
> - solves all the requirements above
> Cons (for Ignite):
> - (currently) closed GridGain source
>
> On top of Tiden we’ve built a set of test suites, some of
which
you might have heard already.
>
> A Combinator suite allows to run set of operations
concurrently
over given database instance. Proven to find at least 30+ race
conditions and NPE issues.
>
> A Consumption suite allows to run a set production-like
actions
over given set of Ignite/GridGain versions and compare test
metrics
across versions, like heap/disk/CPU consumption, time to
perform
actions, like client PME, server PME, rebalancing time, data
replication time, etc.
>
> A Yardstick suite is a thin layer of Python glue code to
run
Apache Ignite pre-release benchmarks set. Yardstick itself
has a
mediocre deployment capabilities, Tiden solves this easily.
>
> A Stress suite that simulates hardware environment
degradation
during testing.
>
> An Ultimate, DR and Compatibility suites that performs
functional
regression testing of GridGain Ultimate Edition features like
snapshots, security, data replication, rolling upgrades, etc.
>
> A Regression and some IEPs testing suites, like IEP-14,
IEP-15,
etc, etc, etc.
>
> Most of the suites above use another in-house developed
Java
tool
– PiClient – to perform actual loading and miscellaneous
operations
with Ignite under test. We use py4j Python-Java gateway
library
to
control PiClient instances from the tests.
>
> When we considered CI, we put TeamCity out of scope,
because
distributed integration and performance tests tend to run for
hours
and TeamCity agents are scarce and costly resource. So,
bundled
with
Tiden there is jenkins-job-builder [10] based CI pipelines and
Jenkins xUnit reporting. Also, rich web UI tool Ward
aggregates
test
run reports across versions and has built in visualization
support
for Combinator suite.
>
> All of the above is currently closed source, but we plan to
make
it public for community, and publishing Tiden core [9] is the
first
step on that way. You can review some examples of using
Tiden for
tests at my repository [11], for start.
>
> Now, let’s compare Ducktape PoC and Tiden.
>
> Criteria: Language
> Tiden: Python, 3.7
> Ducktape: Python, proposes itself as Python 2.7, 3.6, 3.7
compatible, but actually can’t work with Python 3.7 due to
broken
Zmq dependency.
> Comment: Python 3.7 has a much better support for
async-style
code which might be crucial for distributed application
testing.
> Score: Tiden: 1, Ducktape: 0
>
> Criteria: Test writers API
> Supported integration test framework concepts are basically
the
same:
> - a test controller (test runner)
> - a cluster
> - a node
> - an application (a service in Ducktape terms)
> - a test
> Score: Tiden: 5, Ducktape: 5
>
> Criteria: Tests selection and run
> Ducktape: suite-package-class-method level selection,
internal
scheduler allows to run tests in suite in parallel.
> Tiden: also suite-package-class-method level selection,
additionally allows selecting subset of tests by attribute,
parallel
runs not built in, but allows merging test reports after
different
runs.
> Score: Tiden: 2, Ducktape: 2
>
> Criteria: Test configuration
> Ducktape: single JSON string for all tests
> Tiden: any number of YaML config files, command line option
for
fine-grained test configuration, ability to select/modify
tests
behavior based on Ignite version.
> Score: Tiden: 3, Ducktape: 1
>
> Criteria: Cluster control
> Ducktape: allow execute remote commands by node granularity
> Tiden: additionally can address cluster as a whole and
execute
remote commands in parallel.
> Score: Tiden: 2, Ducktape: 1
>
> Criteria: Logs control
> Both frameworks have similar builtin support for remote
logs
collection and grepping. Tiden has built-in plugin that can
zip,
collect arbitrary log files from arbitrary locations at
test/module/suite granularity and unzip if needed, also
application
API to search / wait for messages in logs. Ducktape allows
each
service declare its log files location (seemingly does not
support
logs rollback), and a single entrypoint to collect service
logs.
> Score: Tiden: 1, Ducktape: 1
>
> Criteria: Test assertions
> Tiden: simple asserts, also few customized assertion
helpers.
> Ducktape: simple asserts.
> Score: Tiden: 2, Ducktape: 1
>
> Criteria: Test reporting
> Ducktape: limited to its own text/html format
> Tiden: provides text report, yaml report for reporting
tools
integration, XML xUnit report for integration with
Jenkins/TeamCity.
> Score: Tiden: 3, Ducktape: 1
>
> Criteria: Provisioning and deployment
> Ducktape: can provision subset of hosts from cluster for
test
needs. However, that means, that test can’t be scaled without
test
code changes. Does not do any deploy, relies on external
means,
e.g.
pre-packaged in docker image, as in PoC.
> Tiden: Given a set of hosts, Tiden uses all of them for the
test.
Provisioning should be done by external means. However,
provides
a
conventional automated deployment routines.
> Score: Tiden: 1, Ducktape: 1
>
> Criteria: Documentation and Extensibility
> Tiden: current API documentation is limited, should
change as
we
go open source. Tiden is easily extensible via hooks and
plugins,
see example Maven plugin and Gatling application at [11].
> Ducktape: basic documentation at readthedocs.io
<http://readthedocs.io>. Codebase is rigid, framework core is
tightly coupled and hard to change. The only possible
extension
mechanism is fork-and-rewrite.
> Score: Tiden: 2, Ducktape: 1
>
> I can continue more on this, but it should be enough for
now:
> Overall score: Tiden: 22, Ducktape: 14.
>
> Time for discussion!
>
> ---
> [1] - https://www.testcontainers.org/
> [2] - http://arquillian.org/guides/getting_started/
> [3] - https://jmeter.apache.org/index.html
> [4] - https://openjdk.java.net/projects/code-tools/jmh/
> [5] - https://gatling.io/docs/current/
> [6] - https://github.com/gridgain/yardstick
> [7] - https://github.com/gridgain/poc-tester
> [8] -
https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements
> [9] - https://github.com/gridgain/tiden
> [10] - https://pypi.org/project/jenkins-job-builder/
> [11] - https://github.com/mshonichev/tiden_examples
>
> On 25.05.2020 11:09, Nikolay Izhikov wrote:
>> Hello,
>>
>> Branch with duck tape created -
https://github.com/apache/ignite/tree/ignite-ducktape
>>
>> Any who are willing to contribute to PoC are welcome.
>>
>>
>>> 21 мая 2020 г., в 22:33, Nikolay Izhikov
<nizhikov....@gmail.com <mailto:nizhikov....@gmail.com>>
написал(а):
>>>
>>> Hello, Denis.
>>>
>>> There is no rush with these improvements.
>>> We can wait for Maxim proposal and compare two
solutions :)
>>>
>>>> 21 мая 2020 г., в 22:24, Denis Magda <dma...@apache.org
<mailto:dma...@apache.org>> написал(а):
>>>>
>>>> Hi Nikolay,
>>>>
>>>> Thanks for kicking off this conversation and sharing
your
findings with the
>>>> results. That's the right initiative. I do agree that
Ignite
needs to have
>>>> an integration testing framework with capabilities
listed
by
you.
>>>>
>>>> As we discussed privately, I would only check if
instead of
>>>> Confluent's Ducktape library, we can use an integration
testing framework
>>>> developed by GridGain for testing of Ignite/GridGain
clusters.
That
>>>> framework has been battle-tested and might be more
convenient for
>>>> Ignite-specific workloads. Let's wait for @Maksim
Shonichev
>>>> <mshonic...@gridgain.com
<mailto:mshonic...@gridgain.com>>
who
promised to join this thread once he finishes
>>>> preparing the usage examples of the framework. To my
knowledge, Max has
>>>> already been working on that for several days.
>>>>
>>>> -
>>>> Denis
>>>>
>>>>
>>>> On Thu, May 21, 2020 at 12:27 AM Nikolay Izhikov
<nizhi...@apache.org <mailto:nizhi...@apache.org>>
>>>> wrote:
>>>>
>>>>> Hello, Igniters.
>>>>>
>>>>> I created a PoC [1] for the integration tests of
Ignite.
>>>>>
>>>>> Let me briefly explain the gap I want to cover:
>>>>>
>>>>> 1. For now, we don’t have a solution for automated
testing
of
Ignite on
>>>>> «real cluster».
>>>>> By «real cluster» I mean cluster «like a production»:
>>>>> * client and server nodes deployed on different
hosts.
>>>>> * thin clients perform queries from some other
hosts
>>>>> * etc.
>>>>>
>>>>> 2. We don’t have a solution for automated benchmarks of
some
internal
>>>>> Ignite process
>>>>> * PME
>>>>> * rebalance.
>>>>> This means we don’t know - Do we perform
rebalance(or PME)
in
2.7.0 faster
>>>>> or slower than in 2.8.0 for the same cluster?
>>>>>
>>>>> 3. We don’t have a solution for automated testing of
Ignite
integration in
>>>>> a real-world environment:
>>>>> Ignite-Spark integration can be taken as an example.
>>>>> I think some ML solutions also should be tested in
real-world
deployments.
>>>>>
>>>>> Solution:
>>>>>
>>>>> I propose to use duck tape library from confluent
(apache
2.0
license)
>>>>> I tested it both on the real cluster(Yandex Cloud)
and on
the
local
>>>>> environment(docker) and it works just fine.
>>>>>
>>>>> PoC contains following services:
>>>>>
>>>>> * Simple rebalance test:
>>>>> Start 2 server nodes,
>>>>> Create some data with Ignite client,
>>>>> Start one more server node,
>>>>> Wait for rebalance finish
>>>>> * Simple Ignite-Spark integration test:
>>>>> Start 1 Spark master, start 1 Spark
worker,
>>>>> Start 1 Ignite server node
>>>>> Create some data with Ignite client,
>>>>> Check data in application that queries it
from
Spark.
>>>>>
>>>>> All tests are fully automated.
>>>>> Logs collection works just fine.
>>>>> You can see an example of the tests report - [4].
>>>>>
>>>>> Pros:
>>>>>
>>>>> * Ability to test local changes(no need to public
changes
to
some remote
>>>>> repository or similar).
>>>>> * Ability to parametrize test environment(run the same
tests
on different
>>>>> JDK, JVM params, config, etc.)
>>>>> * Isolation by default so system tests are as
reliable as
possible.
>>>>> * Utilities for pulling up and tearing down services
easily
in clusters in
>>>>> different environments (e.g. local, custom cluster,
Vagrant,
K8s, Mesos,
>>>>> Docker, cloud providers, etc.)
>>>>> * Easy to write unit tests for distributed systems
>>>>> * Adopted and successfully used by other distributed
open
source project -
>>>>> Apache Kafka.
>>>>> * Collect results (e.g. logs, console output)
>>>>> * Report results (e.g. expected conditions met,
performance
results, etc.)
>>>>>
>>>>> WDYT?
>>>>>
>>>>> [1] https://github.com/nizhikov/ignite/pull/15
>>>>> [2] https://github.com/confluentinc/ducktape
>>>>> [3]
https://ducktape-docs.readthedocs.io/en/latest/run_tests.html
>>>>> [4] https://yadi.sk/d/JC8ciJZjrkdndg
<2020-07-05--004.tar.gz>
