[
https://issues.apache.org/jira/browse/METRON-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391819#comment-16391819
]
ASF GitHub Bot commented on METRON-1483:
----------------------------------------
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/958
# Testing Plan:
We presume
* `ZOOKEEPER` is an environment variable set to the zk quorum (e.g.
`node1:2181`)
* `BROKER` is an environment variable set to the broker (e.g. `node1:6667`)
## Unbiased Generation
In order to test this, we will:
1. create a new topic
```
# Create dummy_12_unbiased with 12 partitions
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOOKEEPER
--create --topic dummy_12_unbiased --partitions 12 --replication-factor 1
```
2. Create some dummy data to send into kafka
```
# Generate some templates of dummy data that look like CSV in
~/dummy.templates
# With a number, a string, a GUID and the timestamp
for i in $(seq 1 100);do echo "$i,foo,\$METRON_GUID,\$METRON_TS";done >
~/dummy.templates
```
3. Write into that topic from a set of templates at a specified rate
```
# Write out 1000 messages per second to dummy_12_unbiased
# Monitor dummy_12_unbiased
# Each message is drawn from the set of templates in step 2.
# Stop after 60 seconds and report how much you've written
$METRON_HOME/bin/load_tool.sh -p 5 -ot dummy_12_unbiased -mt
dummy_12_unbiased -t ~/dummy.template -eps 1000 -z $ZOOKEEPER -tl 60000
```
* It should indicate that it generated something like 60k messages
4. Measure if the messages generated roughly match the requested rate.
```
# You will have to ctrl-c this in a few seconds
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh
--bootstrap-server node1:6667 --topic dummy_12_topic --from-beginning | wc -l
```
* It should output something like `Processed a total of 60400 messages`,
which would be about 1000 messages per second.
> Create a tool to monitor performance of the topologies
> ------------------------------------------------------
>
> Key: METRON-1483
> URL: https://issues.apache.org/jira/browse/METRON-1483
> Project: Metron
> Issue Type: New Feature
> Reporter: Casey Stella
> Priority: Major
>
> In performance evaluation, generating synthetic load and monitoring the write
> throughput of our kafka-to-kafka topologies has required a lot of custom
> scripting. We should have a tool that could do the following:
> * Take a file representing a message template and generate synthetic load at
> a given events per second
> * Monitor the kafka offsets of a topic and report throughput numbers in
> events per second
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
