On 1/14/14 10:45 PM, Edward Summers wrote:
Just out of curiosity, does it work for a little bit then stop working? I know
arXiv throttle crawlers, and am not sure if they throttle oai-pmh clients.
Simeon Warner who helps run arXiv has been know to post code4lib, so maybe this
will cross his radar.
The arXiv OAI endpoint at http://export.arxiv.org/oai2 uses 503
responses [1] control request frequency. I think most harvester
libraries support this OK.
I'm sad to say that our export.arxiv.org server is a bit overloaded at
the moment (mainly arXiv API and RSS load) and sometimes this affects
the OAI-PMH performance. We are working on improving performance to
handle the ever increasing load...
Cheers,
Simeon
[1]
http://www.openarchives.org/OAI/2.0/guidelines-repository.htm#FlowControl
In the meantime, could you share your harvesting script on gist.github.com or
somewhere similar for us to take a look?
//Ed
On Jan 14, 2014, at 4:46 PM, Eka Grguric <egrgu...@gmail.com> wrote:
Thanks for responding!
I initialized it as follows (following the code from the synopsis on the site).
my $harvester = Net::OAI::Harvester->new(
baseURL =>
'http://contentpro.lib.bcit.ca/iii/oairep/OAIRepository'
);
