Hi,

we've had similar issues with GSA. There are two things that helped here:
Crawlers on that list in sitemap.xmap get special treatment with regards to the 'last modified' date of files (if-modified-since); if gsa isn't on the list then it will re-request the same file over and over and over again.

I just made a pull request for this: https://github.com/DSpace/DSpace/pull/1435

cheers,
Andrea


On 14/06/16 20:55, Louise Gruenberg wrote:
The American Library Association has an archive ( https://alair.ala.org/) powered by DSpace Direct. We are attempting to implement federated search across about 100 web properties that shouldn't have so much content that they exceed our license limit. The ALAIR has about 5,000 digital records but the search appliance crawls about a quarter of a million on the site. The /handler director is responsible for 183,000+. Does anyone know what we would need to restrict to knock the crawl back to a reasonable level? 

If you could also respond to [email protected] (my work email), that would be great.
--
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

-- 
Dr Andrea Schweer
Lead Software Developer, ITS Information Systems
The University of Waikato, Hamilton, New Zealand
+64-7-837 9120


--
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

Reply via email to