I am building a news aggregator application. I plan to put in a list of
RSS feeds and I am trying to get nutch to parse rss feed and save news
articles in Solr.
But Nutch does not get links of articles from the RSS feed. It only
indexes RSS file as a single document in Solr index. It does not crawl
any links inside rss feed file. Here is how nutch-site.xml is configured
to use plugins:
<property>
<name>plugin.includes</name>
<value>protocol-http|feed|urlfilter-regex|suffix-urlfilter|parse-html|index-(basic|anchor)|indexer-solr|query-(basic|site|url)|response-(json|xml)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
</property>
Also parse-plugins.xml has follwing configuration:
<mimeType name="application/rss+xml">
<plugin id="feed" />
<plugin id="parse-tika" />
</mimeType>
<mimeType name="text/xml">
<plugin id="feed" />
<plugin id="parse-tika" />
</mimeType>
I am using following command to run nutch:
bin/crawl urls crawldir http://localhost:8082/solr 5