jnioche commented on code in PR #1373:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1373#discussion_r1816487862
##########
external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/Spout.java:
##########
@@ -131,12 +144,19 @@ protected void populateBuffer() {
maxBucketNum,
maxURLsPerBucket);
- GetParams request =
+ GetParams.Builder builder =
GetParams.newBuilder()
.setMaxUrlsPerQueue(maxURLsPerBucket)
- .setMaxQueues(maxBucketNum)
.setDelayRequestable(delayRequestable)
- .build();
+ .setMaxQueues(maxBucketNum);
+
+ if (URLFRONTIER_ANY_CRAWL_ID.equals(globalCrawlID)) {
+ builder.setAnyCrawlID(anyCrawlID);
Review Comment:
we can set it explicitly but given it is what it does by default we can
probably get away with just adding a comment.
We could then remove the constant `URLFRONTIER_ANY_CRAWL_ID` as well as
`anyCrawlID` and simplify the logic?
Said differently, if a value is set use it, otherwise rely on the default
behaviour?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
