John Nagle <[EMAIL PROTECTED]> writes on Mon, 28 Apr 2008 11:41:41 -0700:
> Dieter Maurer wrote:
> > Christian Heimes <[EMAIL PROTECTED]> writes on Sat, 12 Apr 2008 18:47:32
> > +0200:
> >> [EMAIL PROTECTED] schrieb:
> >>> which made me suggest to use these as defaults, but then
>
> > We observed
> I do not argue that Python's default GC parameters must change -- only
> that applications with lots of objects may want to consider a
> reconfiguration.
I would argue that changing the GC to some sort
of adaptive strategy should at least be investigated.
Having an app which doesn't need gc spe
On Apr 12, 11:11 am, [EMAIL PROTECTED] wrote:
> I should have been more specific about possible fixes.
>
> > > python2.5 -m timeit 'gc.disable();l=[(i,) for i in range(200)]'
>
> > 10 loops, best of 3: 662 msec per loop
>
> > > python2.5 -m timeit 'gc.enable();l=[(i,) for i in range(200)]'
> I do not argue that Python's default GC parameters must change -- only
> that applications with lots of objects may want to consider a
> reconfiguration.
That's exactly what I was trying to say: it's not that the parameters
are useful for *all* applications (that's why they are tunable
parameter
Dieter Maurer wrote:
Christian Heimes <[EMAIL PROTECTED]> writes on Sat, 12 Apr 2008 18:47:32 +0200:
[EMAIL PROTECTED] schrieb:
which made me suggest to use these as defaults, but then
We observed similar very bad behaviour -- in a Web application server.
Apparently, the standard behaviour i
"Martin v. Löwis" wrote at 2008-4-27 19:33 +0200:
>> Martin said it but nevertheless it might not be true.
>>
>> We observed similar very bad behaviour -- in a Web application server.
>> Apparently, the standard behaviour is far from optimal when the
>> system contains a large number of objects an
"Terry Reedy" <[EMAIL PROTECTED]> writes:
> Can this alternative be made easier by adding a context manager to gc
> module to use with 'with' statements? Something like
>
> with gc.delay() as dummy:
>
That sonuds worth adding as a hack, but really I hope there can be
an improved gc someday.
"Dieter Maurer" <[EMAIL PROTECTED]> wrote in message
news:[EMAIL PROTECTED]
| We observed similar very bad behaviour -- in a Web application server.
| Apparently, the standard behaviour is far from optimal when the
| system contains a large number of objects and occationally, large
| numbers of o
> Martin said it but nevertheless it might not be true.
>
> We observed similar very bad behaviour -- in a Web application server.
> Apparently, the standard behaviour is far from optimal when the
> system contains a large number of objects and occationally, large
> numbers of objects are created
Christian Heimes <[EMAIL PROTECTED]> writes on Sat, 12 Apr 2008 18:47:32 +0200:
> [EMAIL PROTECTED] schrieb:
> > which made me suggest to use these as defaults, but then
> > Martin v. Löwis wrote that
> >
> >> No, the defaults are correct for typical applications.
> >
> > At that point I felt los
Aaron Watters <[EMAIL PROTECTED]> writes:
> Even with Btree's if you jump around in the tree the performance can
> be awful.
The Linux file cache really helps. The simplest approach is to just
"cat" the index files to /dev/null a few times an hour. Slightly
faster (what I do with Solr) is mmap
On Apr 14, 11:18 pm, Carl Banks <[EMAIL PROTECTED]> wrote:
> However, that is for the OP to decide. The reason I don't like the
> sort of question I posed is it's presumptuous--maybe the OP already
> considered and rejected this, and has taken steps to ensure the in
> memory data structure won't
On Apr 14, 4:27 pm, Aaron Watters <[EMAIL PROTECTED]> wrote:
> > A question often asked--and I am not a big a fan of these sorts of
> > questions, but it is worth thinking about--of people who are creating
> > very large data structures in Python is "Why are you doing that?"
> > That is, you should
> A question often asked--and I am not a big a fan of these sorts of
> questions, but it is worth thinking about--of people who are creating
> very large data structures in Python is "Why are you doing that?"
> That is, you should consider whether some kind of database solution
> would be better.
On Apr 12, 6:58 pm, Steve Holden <[EMAIL PROTECTED]> wrote:
> Paul Rubin wrote:
> > Steve Holden <[EMAIL PROTECTED]> writes:
> >> I believe you are making surmises outside your range of competence
> >> there. While your faith in the developers is touching, the garbage
> >> collection scheme is some
> I still don't see what is so good about defaults that lead to O(N*N)
> computation for a O(N) problem, and I like Amaury's suggestion a lot,
> so I would like to see comments on its disadvantages. Please don't
> tell me that O(N*N) is good enough. For N>1E7 it isn't.
Please understand that chang
Paul Rubin wrote:
> Steve Holden <[EMAIL PROTECTED]> writes:
>> I believe you are making surmises outside your range of competence
>> there. While your faith in the developers is touching, the garbage
>> collection scheme is something that has received a lot of attention
>> with respect to performa
Steve Holden <[EMAIL PROTECTED]> writes:
> I believe you are making surmises outside your range of competence
> there. While your faith in the developers is touching, the garbage
> collection scheme is something that has received a lot of attention
> with respect to performance under typical worklo
[EMAIL PROTECTED] wrote:
>> Martin said that the default settings for the cyclic gc works for most
>> people.
>
> I agree.
>
>> Your test case has found a pathologic corner case which is *not*
>> typical for common application but typical for an artificial benchmark.
>
> I agree that my "corner"
> Martin said that the default settings for the cyclic gc works for most
> people.
I agree.
> Your test case has found a pathologic corner case which is *not*
> typical for common application but typical for an artificial benchmark.
I agree that my "corner" is not typical, but I strongly disagr
Sorry, I have to correct my last posting again:
> > Disabling the gc may not be a good idea in a real application; I suggest
> > you to play with the gc.set_threshold function and set larger values, at
> > least while building the dictionary. (700, 1000, 10) seems to yield good
> > results.
> > py
[EMAIL PROTECTED] schrieb:
> which made me suggest to use these as defaults, but then
> Martin v. Löwis wrote that
>
>> No, the defaults are correct for typical applications.
>
> At that point I felt lost and as the general wish in that thread was
> to move
> discussion to comp.lang.python, I bro
[EMAIL PROTECTED] wrote:
> In an application dealing with very large text files, I need to create
> dictionaries indexed by tuples of words (bi-, tri-, n-grams) or nested
> dictionaries. The number of different data structures in memory grows
> into orders beyond 1E7.
>
> It turns out that the def
I should have been more specific about possible fixes.
> > python2.5 -m timeit 'gc.disable();l=[(i,) for i in range(200)]'
>
> 10 loops, best of 3: 662 msec per loop
>
> > python2.5 -m timeit 'gc.enable();l=[(i,) for i in range(200)]'
>
> 10 loops, best of 3: 15.2 sec per loop
>
> In the l
On Apr 12, 7:02 am, [EMAIL PROTECTED] wrote:
> I would suggest to configure the default behaviour of the garbage
> collector in such a way that this squared complexity is avoided
> without requiring specific knowledge and intervention by the user. Not
> being an expert in these details I would like
[...]
> I would suggest to configure the default behaviour of the garbage
> collector in such a way that this squared complexity is avoided
> without requiring specific knowledge and intervention by the user. Not
> being an expert in these details I would like to ask the gurus how
> this could be d
26 matches
Mail list logo
