jnioche opened a new pull request, #2213:
URL: https://github.com/apache/stormcrawler/pull/2213
Adds `StatusUpdaterBenchmark`, a small utility in
`org.apache.stormcrawler.persistence` for measuring the throughput of a status
updater bolt, and of the backend behind it, without running a topology.
### How it works
- Instantiates the bolt from its class name and calls `prepare()` with the
given configuration.
- Reads the URLs from a file and passes them straight to the bolt's
`store()` method instead of `execute()`, so that only the storage is measured:
no cache of discovered URLs (it is disabled), no scheduling, no metadata
filtering.
- Waits for all the tuples to be acked or failed, then prints the counts,
the total time and the average OPS. As in the URLFrontier client, the number of
acked URLs and the OPS are also printed every minute.
- Only warnings and errors are logged, so that the bolts' INFO messages
don't drown the output.
### Input
Same format as the URLFrontier client, optionally gzipped:
- a plain URL per line, stored as `DISCOVERED` with the current date as next
fetch date;
- or a JSON `URLItem`: `discovered` items are stored as `DISCOVERED`,
`known` ones as `FETCHED` with their `refetchable_from_date` as next fetch date
(none if 0 or not set). The metadata are stored as they are.
The URLs should be shuffled beforehand. Dumps are usually sorted by host,
and with a backend that routes by host (e.g. OpenSearch with routing enabled),
each bulk request then lands on only one or two shards.
### Usage
It must be run with `storm local`, from the jar of a crawl project whose
pom.xml has the dependency for the backend module (e.g.
`stormcrawler-opensearch`):
```bash
storm local target/crawler-1.0-SNAPSHOT.jar \
org.apache.stormcrawler.persistence.StatusUpdaterBenchmark \
org.apache.stormcrawler.opensearch.persistence.StatusUpdaterBolt \
crawler-conf.yaml opensearch-conf.yaml urls.txt.gz
```
The arguments are positional (bolt class, one or more configuration files,
URL file) because the `storm` command intercepts options such as `-c` and
`--conf`.
### Notes for reviewers
- `store()` is protected but accessible because the class is in the same
package as `AbstractStatusUpdaterBolt`.
- The bolt gets a minimal `TopologyContext` built by the class, and a simple
`IOutputCollector` counts the acks and fails.
- Log levels are set with log4j's `Configurator`, which comes through the
`provided` storm-client dependency and is present at runtime under `storm
local`.
- No unit tests. Tested manually with `StdOutStatusUpdater`,
`MemoryStatusUpdater` (the harness alone reaches ~450k URLs/sec) and the
OpenSearch status updater.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
