Hi Louise,
Actually the Google bot do not use to be the problem, they respect
what you have in the robots.txt(
https://wiki.duraspace.org/display/DSDOC4x/Search+Engine+Optimization) ,
However in the worst scenario you can implement a servlet filter to block
bots.
Best regards
Luiz
On Tue, Jun 14, 2016 at 3:55 PM, Louise Gruenberg <[email protected]>
wrote:
> The American Library Association has an archive ( https://alair.ala.org/)
> powered by DSpace Direct. We are attempting to implement federated search
> across about 100 web properties that shouldn't have so much content that
> they exceed our license limit. The ALAIR has about 5,000 digital records
> but the search appliance crawls about a quarter of a million on the site.
> The /handler director is responsible for 183,000+. Does anyone know what we
> would need to restrict to knock the crawl back to a reasonable level?
>
> If you could also respond to [email protected] (my work email), that
> would be great.
>
> --
> You received this message because you are subscribed to the Google Groups
> "DSpace Technical Support" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at https://groups.google.com/group/dspace-tech.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.