Slowdown when fetching multiple records
I'm using code similar to the "get MARC record by ISBN" example in MARC::Doc::Tutorial, and everything is working fine. However, when I change the query to search by Author, rather than by ISBN, (using @attr 1=1003 $name), I've noticed that looping through the results gets dramatically slower as the number of results increases. The first 18 results come back pretty much instantly. But after that each successive record takes about 2 seconds to fetch. For an author with a lot of books, I've seen it take about 4-5 seconds to fetch each of the later records. I've done some searching through various docs, but I'm a little lost. I can't work out if there's some way to fetch everything back in one shot and use MARC::Batch to loop through, or whether there are some options I'm meant to set in Z3950 (there are some reference in the zoom docs to small result sets vs medium result sets etc., but I can't really figure out what to do with these). Any pointers gratefully received. Thanks, Tony
Re: Slowdown when fetching multiple records
On Feb 19, 2006, at 3:57 AM, Tony Bowden wrote: Any pointers gratefully received. I haven't used the Net::Z3950 enough to be able to provide any help with this. However, you might want to post your message to the net- z3950 discussion list [1] which fields questions specific to the Net::Z3950 module. The author is available on there and may be able to provide some guidance. Have you observed similar performance degradation on other z3950 targets? I'm wondering if the slowdown is possibly happening on the server side rather than the client side. //Ed [1] http://lists.indexdata.dk/cgi-bin/mailman/listinfo/net-z3950
Re: Slowdown when fetching multiple records
On Sun, Feb 19, 2006 at 09:53:59AM -0600, Edward Summers wrote: > I haven't used the Net::Z3950 enough to be able to provide any help > with this. However, you might want to post your message to the net- > z3950 discussion list [1] which fields questions specific to the > Net::Z3950 module. The author is available on there and may be able > to provide some guidance. Thanks for the pointer. > Have you observed similar performance degradation on other z3950 > targets? I'm wondering if the slowdown is possibly happening on the > server side rather than the client side. I foolishly hadn't actually tried any other servers, but I've just ran the script against the Oxford University server with no such slowdown. So, it presumably is an issue with the Library of Congress server. Is there some sort of automatic throttling there? Or is there likely to be some sort of option that I should be setting, but not? Thanks for the help, Tony
Re: Slowdown when fetching multiple records
On Feb 19, 2006, at 10:14 AM, Tony Bowden wrote: So, it presumably is an issue with the Library of Congress server. Is there some sort of automatic throttling there? Or is there likely to be some sort of option that I should be setting, but not? Yeah, I could well imagine there to be throttling in place. I believe that Mike Taylor (author of Net::Z3950) may know details of the LoC server, so posting on the net-z3950 might still help. If you are just after bibliographic data you may want to try the SRU server at the LoC which lets you grab various metadata flavors as XML via HTTP. It may have similar throttling though. For an example here is a query for an ISBN 0-486-25778-9: [get DublinCore] http://z3950.loc.gov:7090/voyager? query=bath.isbn=0486257789&startRecord=1&maximumRecords=1&recordSchema=d c&version=1.1&operation=searchRetrieve [get MARCXML] http://z3950.loc.gov:7090/voyager? query=bath.isbn=0486257789&startRecord=1&maximumRecords=1&recordSchema=m arcxml&version=1.1&operation=searchRetrieve [get MODS] http://z3950.loc.gov:7090/voyager? query=bath.isbn=0486257789&startRecord=1&maximumRecords=1&recordSchema=m ods&version=1.1&operation=searchRetrieve If you find out anything and have the time please post your findings here for the rest of us :-) //Ed
