GGraziadei commented on PR #2261: URL: https://github.com/apache/stormcrawler/pull/2261#issuecomment-6425710756
Thanks a lot for the clarification, @jnioche! Sorry I didn’t check the benchmark myself and instead followed up on a previous message. I agree that dynamic batching might not make much sense in this context. The idea behind it is to address backpressure, but I agree that in a crawling topology, where most of a tuple’s lifetime is spent waiting in the FetcherBolt queues due to politeness constraints, batching is unlikely to have a significant impact. One possible scenario could be a very small cluster, although even there, I’d expect the effect to be limited. In any case, we would need to demonstrate its actual impact with some numbers. Thanks again for the clarification! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
