[
https://issues.apache.org/jira/browse/NUTCH-2509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415774#comment-16415774
]
ASF GitHub Bot commented on NUTCH-2509:
---------------------------------------
sebastian-nagel closed pull request #301: NUTCH-2509 Apply URL
filters/normalizers also to URLs of
URL: https://github.com/apache/nutch/pull/301
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/src/java/org/apache/nutch/util/SitemapProcessor.java
b/src/java/org/apache/nutch/util/SitemapProcessor.java
index ab4a23ef6..380ac0710 100644
--- a/src/java/org/apache/nutch/util/SitemapProcessor.java
+++ b/src/java/org/apache/nutch/util/SitemapProcessor.java
@@ -151,9 +151,16 @@ else if (value instanceof HostDatum) {
if (tryDefaultSitemapXml && sitemaps.size() == 0) {
sitemaps.add(url + "sitemap.xml");
}
- for(String sitemap: sitemaps) {
+ for (String sitemap : sitemaps) {
context.getCounter("Sitemap", "sitemaps_from_hostdb").increment(1);
- generateSitemapUrlDatum(protocolFactory.getProtocol(sitemap),
sitemap, context);
+ sitemap = filterNormalize(sitemap);
+ if (sitemap == null) {
+ context.getCounter("Sitemap", "filtered_sitemaps_from_hostdb")
+ .increment(1);
+ } else {
+ generateSitemapUrlDatum(protocolFactory.getProtocol(sitemap),
+ sitemap, context);
+ }
}
}
else if (value instanceof Text) {
@@ -263,8 +270,11 @@ else if (asm instanceof SiteMapIndex) {
}
LOG.info("Parsing sitemap index file: {}", index.getUrl().toString());
- for(AbstractSiteMap sitemap: sitemapUrls) {
- generateSitemapUrlDatum(protocol, sitemap.getUrl().toString(),
context);
+ for (AbstractSiteMap sitemap : sitemapUrls) {
+ String sitemapUrl = filterNormalize(sitemap.getUrl().toString());
+ if (sitemapUrl != null) {
+ generateSitemapUrlDatum(protocol, sitemapUrl, context);
+ }
}
}
}
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Inconsistent behavior in SitemapProcessor
> -----------------------------------------
>
> Key: NUTCH-2509
> URL: https://issues.apache.org/jira/browse/NUTCH-2509
> Project: Nutch
> Issue Type: Bug
> Components: sitemap
> Affects Versions: 1.14
> Reporter: Yossi Tamari
> Priority: Minor
> Fix For: 1.15
>
> Attachments: SitemapProcessor.patch
>
>
> There are two inconsistent behaviors in SitemapProcessor:
> # There is a member variable maxRedir that is supposed to limit the number
> of redirections on sitemap URLs, and it is initialized from config property
> sitemap.redir.max, but it is ignored in the code because a local variable
> with the same name is defined in the relevant method, and is always set to 3.
> # When a sitemap URL goes through redirect, it is filtered and normalized.
> However, if a sitemap URL comes from a sitemapindex, it is not. This seems
> inconsistent, as in both cases we have a URL from an outside source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
