Re: Slowdown when fetching multiple records
On Sun, Feb 19, 2006 at 06:08:40PM -0600, Edward Summers wrote:
> If you find out anything and have the time please post your findings
> here for the rest of us :-)
Thanks for the pointers.
On further investigation I discovered the present() method in
Net::Z3950::ResultSet.
This pre-fetches a given range of results in one shot.
So, rather than a simple loop around the resultset, I now do something
like:
for my $i (1 .. $numrec) {
$rs->present($i, 20) unless $i % 20;
my $zrec = $rs->record($i);
}
The time it takes to fetch each batch varies significantly (sometimes
over 5 seconds), but I haven't yet seen a pattern to it that would imply
throttling. At a guess I would say it's more likely to be a queueing
system at the far end.
I need to do some more analysis to work out what the most effective
quantity is to pre-fetch at a time, but this seems to work nicely for
now.
Thanks,
Tony
Re: Slowdown when fetching multiple records
At 4:14 PM + 2/19/06, Tony Bowden wrote: So, it presumably is an issue with the Library of Congress server. Is there some sort of automatic throttling there? Or is there likely to be some sort of option that I should be setting, but not? I've not used Perl-based Z39.50 searching, but I think LC requires a 6-second pause between searches, to reduce burden on their servers. While searching using MarcEdit 4.6 (or pre-5), we were locked out for the day for not following this new (Oct. 2005?) rule. MarcEdit 5 beta allows a 6-second pause for servers that might require it, like LC's. I don't know how the Perl modules handle this pause for batch searching LC, but it might be why you are experiencing a delay. I hope this helps, Bryan Baldus [EMAIL PROTECTED] [EMAIL PROTECTED] http://home.inwave.com/eija
