jnioche opened a new pull request, #2223: URL: https://github.com/apache/stormcrawler/pull/2223
The URLFrontier `StatusUpdaterBolt` keeps track of every URL it sends so that it can ack or fail the corresponding tuples. This makes that bookkeeping cheaper. ### Changes - **One permit release per batch.** When the `BatchAck` for a batch of discovered URLs comes back, the in-flight permits of the whole batch are released in one go instead of once per URL, so that a send waiting for room is woken up once per batch. The release is done in a `finally` so that an exception while acking can't leak permits. - **Eviction listener instead of removal listener on `waitAck`.** `onRemoval` only acted on evictions but, registered as a removal listener, it was also notified asynchronously on the common `ForkJoinPool` of every URL removed on its ack: one task per URL for nothing. Evicted entries are handled as before. - **No more fair lock around `waitAck`.** The `ReentrantLock(true)` taken by the sending thread and by the gRPC callback thread for every URL is replaced by the atomic operations of the cache's map view: `compute()` when adding a tuple, `remove()` on an ack. A tuple added while its URL is being acked ends up either in the list being removed or in a new entry. ### Behaviour change Adding a tuple to an entry which already has one, e.g. the FETCHED status of a URL whose DISCOVERED is still pending, now restarts the expiry of the entry (`urlfrontier.cache.expireafter.sec`), as any write does; it used to keep the deadline of the first send. The check for a discovered URL already being sent remains a plain read, so that a URL discovered over and over can't keep postponing the expiry of an entry whose ack got lost. ### Measurements With `StatusUpdaterBenchmark` (#2213), 4 instances (#2222) and the default batch size, against URLFrontier: 164K → 180K URLs/sec. ### Notes for reviewers - The eviction listener now runs synchronously within the cache operation that evicts the entry. It only releases the permits, logs and fails the tuples; it doesn't touch the cache. - No change to the configuration or to the messages sent to the frontier. ### For all changes - [ ] Is there a issue associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with `#XXXX` where `XXXX` is the issue number you are trying to resolve? - [x] Has your PR been rebased against the latest commit within the target branch (typically main)? - [x] Is your initial contribution a single, squashed commit? - [x] Is the code properly formatted with `mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false`? ### For code changes - [ ] Have you ensured that the full suite of tests is executed via `mvn clean verify`? - [ ] Have you written or updated unit tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? (no new dependencies) - [x] If applicable, have you updated the LICENSE file, including the main LICENSE file? (not applicable) - [x] If applicable, have you updated the NOTICE file, including the main NOTICE file? (not applicable) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
