Re: zip() function troubles

On 31 Jul 2007 12:57:13 -0700, Paul Rubin <"http://phr.cx"@nospam.invalid> wrote: > "Chris Mellon" <[EMAIL PROTECTED]> writes: > > Better hand in your computer, then. You're never going to find a > > situation where the environment won't affect the running time of your > > algorithms. > > The envir

Re: zip() function troubles

"Chris Mellon" <[EMAIL PROTECTED]> writes: > Better hand in your computer, then. You're never going to find a > situation where the environment won't affect the running time of your > algorithms. The environment may affect the running time by an additive or linear multiplicative constant but it sh

Re: zip() function troubles

"Istvan Albert" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] | On Jul 27, 2:18 pm, Raymond Hettinger <[EMAIL PROTECTED]> wrote: | | >> What was really surprising is that it works | >> with no issues up until 1 million items" | | later editing made the sentence more difficult to rea

Re: zip() function troubles

On Jul 27, 2:18 pm, Raymond Hettinger <[EMAIL PROTECTED]> wrote: >> What was really surprising is that it works >> with no issues up until 1 million items" later editing made the sentence more difficult to read I should have said: "What was really surprising is that zip works with no issues up un

Re: zip() function troubles

Raymond Hettinger <[EMAIL PROTECTED]> writes: > > What was really surprising is that it works with no issues up > > until 1 million items, but for say 10 million it pretty much goes > > nuts. Does anyone know why? > > There's nothing in izip() that holds memory, tracks indices, or ... The issue w

Re: zip() function troubles

On Jul 26, 4:25 pm, Istvan Albert <[EMAIL PROTECTED]> wrote: > Now I know that zip () wastes lots of memory because it copies the > content of the lists, I had used zip to try to trade memory for speed > (heh!) , and now that everything was replaced with izip it works just > fine. What was really

Re: zip() function troubles

On 26 Jul 2007 23:35:44 -0700, Paul Rubin <"http://phr.cx"@nospam.invalid> wrote: > Peter Otten <[EMAIL PROTECTED]> writes: > > When you are allocating a lot of objects without releasing them the garbage > > collector kicks in to look for cycles. Try switching it off: > > I think that is the answer

Re: zip() function troubles

On 7/27/07, Istvan Albert <[EMAIL PROTECTED]> wrote: > On Jul 27, 2:16 am, "Terry Reedy" <[EMAIL PROTECTED]> wrote: > > > References are not objects. > > yes this a valid objection, > > but in all fairness the example in the original post operates on > comparably sized objects and also exhibited un

Re: zip() function troubles

On Jul 27, 2:16 am, "Terry Reedy" <[EMAIL PROTECTED]> wrote: > References are not objects. yes this a valid objection, but in all fairness the example in the original post operates on comparably sized objects and also exhibited unexpected performance degradation as it turns out the garbage coll

Re: zip() function troubles

On Jul 27, 1:24 am, Peter Otten <[EMAIL PROTECTED]> wrote: > When you are allocating a lot of objects without releasing them the garbage > collector kicks in to look for cycles. Try switching it off: > > import gc > gc.disable() Yes, this solves the problem I was experiencing. Thanks. Istvan --

Re: zip() function troubles

Peter Otten <[EMAIL PROTECTED]> writes: > When you are allocating a lot of objects without releasing them the garbage > collector kicks in to look for cycles. Try switching it off: I think that is the answer. The zip took almost 2 minutes without turning gc off, but takes 1.25 seconds with gc off

Re: zip() function troubles

"Istvan Albert" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] || if you try it yourself you'll see that it is very easy to generate 10 | million tuples, No it is not on most machines. | on my system it takes 3 (!!!) seconds to do the following: | | size = 10**7 | data = [] | for

Re: zip() function troubles

Istvan Albert wrote: > I've been debugging the reason for a major slowdown in a piece of > code ... and it turns out that it was the zip function. In the past > the lists that were zipped were reasonably short, but once the size > exceeded 10 million the zip function slowed to a crawl. Note that >

Re: zip() function troubles

Istvan Albert <[EMAIL PROTECTED]> writes: > Now it takes over two minutes to do this: > > size = 10**7 > a = [ 0 ] * size > b = zip(a,a) OK, I'm getting similar results under 64 bit Pytnon 2.4.4c1 and also under 2.5. About 103 seconds for 10**7 and 26 seconds for 5*10**6. So it looks like zip is

Re: zip() function troubles

On Jul 26, 9:33 pm, Paul Rubin wrote: > Do a top or vmstat while that is happening and see if you are > swapping. You are allocating 10 million ints and 10 million tuple > nodes, = 20 million objects. Although, even at 100 bytes per object > that would be 1GB which wou

Re: zip() function troubles

Istvan Albert <[EMAIL PROTECTED]> writes: > I tested this on a linux server system with 4Gb of RAM > a = [ 0 ] * 10**7 > takes miliseconds, but say the > b = zip(a,a) > will take a very long time to finish: Do a top or vmstat while that is happening and see if you are swapping. You are allocating

Re: zip() function troubles

On Jul 26, 7:44 pm, Paul Rubin wrote: > Istvan Albert <[EMAIL PROTECTED]> writes: > > exceeded 10 million the zip function slowed to a crawl. Note that > > there was memory available to store over 100 million items. > > How many bytes is that? Maybe the items (heap-alloc

Re: zip() function troubles

Istvan Albert <[EMAIL PROTECTED]> writes: > exceeded 10 million the zip function slowed to a crawl. Note that > there was memory available to store over 100 million items. How many bytes is that? Maybe the items (heap-allocated boxed integers in your code example) are bigger than you expect. --

zip() function troubles

Hello all, I've been debugging the reason for a major slowdown in a piece of code ... and it turns out that it was the zip function. In the past the lists that were zipped were reasonably short, but once the size exceeded 10 million the zip function slowed to a crawl. Note that there was memory av