RE: Survey : Max size allowable for slurping files

2004-01-22 Thread Dan Muey
> Dan Anderson wrote: > > > > > Very true. But you also need to look at what you're doing. > A spider > > that indexes or coallates pages across several sites might need to > > slurp up a large number of pages -- which even at a few kilobytes a > > piece would be costly on system resources.

Re: Survey : Max size allowable for slurping files

2004-01-22 Thread Wiggins d'Anconia
Dan Anderson wrote: Very true. But you also need to look at what you're doing. A spider that indexes or coallates pages across several sites might need to slurp up a large number of pages -- which even at a few kilobytes a piece would be costly on system resources. Ironically this is the one tim

Re: Survey : Max size allowable for slurping files

2004-01-22 Thread Dan Anderson
On Thu, 2004-01-22 at 18:21, Dan Anderson wrote: > On Thu, 2004-01-22 at 17:59, James Edward Gray II wrote: > > On Jan 22, 2004, at 4:12 PM, Tim Johnson wrote: > > > > > Here's another argument against slurping: When you slurp a file all at > > > once, even if your program isn't using up much of

Re: Survey : Max size allowable for slurping files

2004-01-22 Thread Dan Anderson
On Thu, 2004-01-22 at 17:59, James Edward Gray II wrote: > On Jan 22, 2004, at 4:12 PM, Tim Johnson wrote: > > > Here's another argument against slurping: When you slurp a file all at > > once, even if your program isn't using up much of the CPU, on many > > machines it will slow down performance

Re: Survey : Max size allowable for slurping files

2004-01-22 Thread James Edward Gray II
On Jan 22, 2004, at 4:12 PM, Tim Johnson wrote: Here's another argument against slurping: When you slurp a file all at once, even if your program isn't using up much of the CPU, on many machines it will slow down performance considerably if you slurp a large file (large, of course, is still some

RE: Survey : Max size allowable for slurping files

2004-01-22 Thread Dan Muey
> Here's another argument against slurping: When you slurp a > file all at once, even if your program isn't using up much of > the CPU, on many machines it will slow down performance > considerably if you slurp a large file (large, of course, is > still sometimes relative). If that is the onl

RE: Survey : Max size allowable for slurping files

2004-01-22 Thread Dan Muey
> > Good comparison, I never see advice to use no warnigns and > no strict > > though :) > > I've actually seen it a few times in code, but it's usually surrounded > by: > > ## > ## > #WARNING!! > ## > # Warnings / Strict

RE: Survey : Max size allowable for slurping files

2004-01-22 Thread Tim Johnson
assume that. -Original Message- From: Dan Muey [mailto:[EMAIL PROTECTED] Sent: Thursday, January 22, 2004 1:49 PM To: Dan Anderson Cc: Perl Beginners Mailing List Subject: RE: Survey : Max size allowable for slurping files Good comparison, I never see advice to use no warnigns and no

RE: Survey : Max size allowable for slurping files

2004-01-22 Thread Dan Anderson
On Thu, 2004-01-22 at 16:49, Dan Muey wrote: > > > But I see a lot of "don't slurp that" and I was hoping for more > > > clear reasons/situatuions to or not to slurp so people > > positn code can have a better idea why a perosn said: > > > "do(n't) slurp your file here" > > > > > > Basically we n

RE: Survey : Max size allowable for slurping files

2004-01-22 Thread Dan Muey
> > But I see a lot of "don't slurp that" and I was hoping for more > > clear reasons/situatuions to or not to slurp so people > positn code can have a better idea why a perosn said: > > "do(n't) slurp your file here" > > > > Basically we need to expalin why more: > > > > - Don't slurp this beca

RE: Survey : Max size allowable for slurping files

2004-01-22 Thread Dan Anderson
On Thu, 2004-01-22 at 16:32, Dan Muey wrote: > > On Thu, 2004-01-22 at 16:16, Dan Muey wrote: > > > > On Thu, 2004-01-22 at 13:18, Dan Muey wrote: > > > > > There are always comments like "you can slurp the file > > as long as > > > > > it's not too big" or "becareful not to slurp a really > > b

RE: Survey : Max size allowable for slurping files

2004-01-22 Thread Dan Muey
> On Thu, 2004-01-22 at 16:16, Dan Muey wrote: > > > On Thu, 2004-01-22 at 13:18, Dan Muey wrote: > > > > There are always comments like "you can slurp the file > as long as > > > > it's not too big" or "becareful not to slurp a really > big file or > > > > you'll be in trouble". > > > > > > I

RE: Survey : Max size allowable for slurping files

2004-01-22 Thread Dan Anderson
On Thu, 2004-01-22 at 16:16, Dan Muey wrote: > > On Thu, 2004-01-22 at 13:18, Dan Muey wrote: > > > There are always comments like "you can slurp the file as > > > long as it's not too big" or "becareful not to slurp a > > > really big file or you'll be in trouble". > > > > I'd like to add that s

RE: Survey : Max size allowable for slurping files

2004-01-22 Thread Dan Muey
> On Thu, 2004-01-22 at 13:18, Dan Muey wrote: > > There are always comments like "you can slurp the file as > > long as it's not too big" or "becareful not to slurp a > > really big file or you'll be in trouble". > > I'd like to add that some of it depends on swap space. I've > slurped well pa

Re: Survey : Max size allowable for slurping files

2004-01-22 Thread Dan Anderson
On Thu, 2004-01-22 at 13:18, Dan Muey wrote: > There are always comments like "you can slurp the file as > long as it's not too big" or "becareful not to slurp a > really big file or you'll be in trouble". I'd like to add that some of it depends on swap space. I've slurped well past physical me

RE: Survey : Max size allowable for slurping files

2004-01-22 Thread Dan Muey
> On Jan 22, 2004, at 12:18 PM, Dan Muey wrote: > > > There are always comments like "you can slurp the file as > long as it's > > not too big" or "becareful not to slurp a really big file > or you'll be > > in trouble". > > > > So what I'd like to survey is what would be what the safest > ma

Re: Survey : Max size allowable for slurping files

2004-01-22 Thread James Edward Gray II
On Jan 22, 2004, at 12:18 PM, Dan Muey wrote: There are always comments like "you can slurp the file as long as it's not too big" or "becareful not to slurp a really big file or you'll be in trouble". So what I'd like to survey is what would be what the safest max size of a file that one should ev