jnioche opened a new pull request, #2222:
URL: https://github.com/apache/stormcrawler/pull/2222
Follow-up to #2213.
`StatusUpdaterBenchmark` can now run several instances of the status updater
bolt in parallel, to show what a topology with a higher parallelism for the
status updater would get. The number of instances is an optional argument after
the class name of the bolt; it defaults to 1, so existing commands keep working.
```
storm local target/crawler-1.0-SNAPSHOT.jar \
org.apache.stormcrawler.persistence.StatusUpdaterBenchmark \
org.apache.stormcrawler.opensearch.persistence.StatusUpdaterBolt 4 \
backend.yaml urls.txt.gz
```
- Each instance gets its own task in the `TopologyContext` and its own
thread, as a Storm executor would.
- The URLs are routed to the instances based on their hash, as a fields
grouping on the URL does, so that a given URL always goes to the same instance.
- The input is read and parsed by a separate thread and handed over in
batches of 1,000 URLs, which also takes the parsing off the thread calling
`store()`.
The Javadoc no longer suggests passing `crawler-conf.yaml`: a single
configuration file with the settings of the backend is enough, whatever its
name.
### Notes for reviewers
- Because of the routing by hash, an instance whose in-flight window is full
blocks the reading thread and so the other instances, as a fields grouping
would in a topology.
- The reading thread caps the total throughput; a run with
`MemoryStatusUpdater` on a sample of the input gives an idea of that limit.
- Tested manually with 4 instances of the URLFrontier status updater,
against both the Java and the Rust implementations of URLFrontier.
### For all changes
- [ ] Is there a issue associated with this PR? Is it referenced in the
commit message?
- [ ] Does your PR title start with `#XXXX` where `XXXX` is the issue number
you are trying to resolve?
- [x] Has your PR been rebased against the latest commit within the target
branch (typically main)?
- [x] Is your initial contribution a single, squashed commit?
- [x] Is the code properly formatted with `mvn git-code-format:format-code
-Dgcf.globPattern="**/*" -Dskip.format.code=false`?
### For code changes
- [ ] Have you ensured that the full suite of tests is executed via `mvn
clean verify`?
- [ ] Have you written or updated unit tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)? (no new
dependencies)
- [x] If applicable, have you updated the LICENSE file, including the main
LICENSE file? (not applicable)
- [x] If applicable, have you updated the NOTICE file, including the main
NOTICE file? (not applicable)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
