[
https://issues.apache.org/jira/browse/NUTCH-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16416102#comment-16416102
]
ASF GitHub Bot commented on NUTCH-2536:
---------------------------------------
lewismc closed pull request #298: NUTCH-2536 change GeneratorReducer.count
field to non-static variable…
URL: https://github.com/apache/nutch/pull/298
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/src/java/org/apache/nutch/crawl/GeneratorReducer.java
b/src/java/org/apache/nutch/crawl/GeneratorReducer.java
index 852adaf44..e427ecc6e 100644
--- a/src/java/org/apache/nutch/crawl/GeneratorReducer.java
+++ b/src/java/org/apache/nutch/crawl/GeneratorReducer.java
@@ -43,7 +43,7 @@
private long limit;
private long maxCount;
- protected static long count = 0;
+ private long count = 0;
private boolean byDomain = false;
private Map<String, Integer> hostCountMap = new HashMap<>();
private Utf8 batchId;
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> GeneratorReducer.count is a static variable
> -------------------------------------------
>
> Key: NUTCH-2536
> URL: https://issues.apache.org/jira/browse/NUTCH-2536
> Project: Nutch
> Issue Type: Bug
> Components: generator
> Affects Versions: 2.3.1
> Environment: Non-distributed, single node, standalone Nutch jobs run
> in a sinlge JVM with HBase as the data store. 2.3.1
> Reporter: Ben Vachon
> Priority: Minor
> Labels: Generate
> Fix For: 2.4
>
> Original Estimate: 2.4h
> Remaining Estimate: 2.4h
>
> The count field of the GeneratorReducer class is a static field. This means
> that if the GeneratorJob is run multiple times within the same JVM, it will
> count all the webpages generated across all batches.
> The count field is checked against the GeneratorJob's topN configuration
> variable, which is described as:
> "top threshold for maximum number of URLs permitted in a batch"
> I understand this to mean that EACH batch should be capped at the topN value,
> not ALL batches.
> This isn't a problem with the way that Nutch is typically used because the
> script starts a new JVM each time. I'm not using the script, I'm calling the
> java classes directly (using the ToolRunner) within an existing JVM, so I'm
> categorizing this as an SDK issue.
> Changing the field to be non-static will not affect the behavior of the class
> as its run by the script.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
