I am building a news aggregator application. I plan to put in a list of RSS feeds and I am trying to get nutch to parse rss feed and save news articles in Solr. But Nutch does not get links of articles from the RSS feed. It only indexes RSS file as a single document in Solr index. It does not crawl any links inside rss feed file. Here is how nutch-site.xml is configured to use plugins:


      <property>
        <name>plugin.includes</name>
<value>protocol-http|feed|urlfilter-regex|suffix-urlfilter|parse-html|index-(basic|anchor)|indexer-solr|query-(basic|site|url)|response-(json|xml)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
        </property>


Also parse-plugins.xml has follwing configuration:

    <mimeType name="application/rss+xml">
            <plugin id="feed" />


             <plugin id="parse-tika" />

        </mimeType>
    <mimeType name="text/xml">
            <plugin id="feed" />
            <plugin id="parse-tika" />

        </mimeType>

I am using following command to run nutch:
bin/crawl urls crawldir http://localhost:8082/solr 5

Reply via email to