Re: [Python-Dev] Stdlib Logging questions (PEP 337 SoC)

[Fredrik Lundh]

Phillip J. Eby wrote:

> If this *has* to be added to the modules that don't currently do any 
> logging, can we please delay the import until it's actually needed?

now that we've had a needforspeed sprint, it's clear that it's time to 
start working on slowing things down again ;-)

> I think it would be a good idea to revisit the module list.  I can see a 
> reasonable case for the BaseHTTP stuff and asyncore needing a logging 
> framework, if you plan to make them part of some larger framework -- the 
> configurability would be a plus

asyncore's logging is configurable through subclassing, as is everything 
else in asyncore (the components don't do anything if you don't subclass 
them).  mapping to the logger module by default will save 5-10 lines of 
code in an application that wants to use the logger module.  *replacing* 
the existing logging hooks with the logger module will break tons of stuff.

BaseHTTP only does webserver-style logging, as far as I can tell.



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mac/wastemodule build failing

[skip]

Ronald> The failure was the result of the removeal of Mac/Wastemods,
Ronald> these are needed for the waste wrappers. I've just checked in a
Ronald> patch that removes these wrappers (revision 46644), they won't
Ronald> work on Intel macs, aren't used by anything in the current tree
Ronald> and are undocumented.

Worked fine, thanks.

Skip


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] wsgiref documentation

[A.M. Kuchling]

On Fri, Jun 02, 2006 at 03:33:41PM -0400, Doug Fort wrote:
> I'm going over the possible tasks for the Arlington Sprint.
> Documentation for wsgiref looks like somethng I could handle.
> 
> Is anyone already working on this?

I had the start of an outline in sandbox/wsgiref-docs, but am not
working on them at the moment because no one is willing to say if the
list of documented classes is complete (or includes too much).  Please
feel free to start writing actual content.  I can check in whatever
you produce; just e-mail me the new version of the file.

--amk

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch #1454481 vs buildbot

[Andrew MacIntyre]

Tim Peters wrote:

> #if THREAD_STACK_MIN < PTHREAD_STACK_MIN
> 
> assumes that the expansion of PTHREAD_STACK_MIN acts like a
> compile-time constant expression, but there's no such guarantee.
> 
>http://cvs.opensolaris.org/source/xref/on/usr/src/head/limits.h
> 
> shows that, on one version of Solaris, it's actually defined via
> 
> #definePTHREAD_STACK_MIN ((size_t)_sysconf(_SC_THREAD_STACK_MIN))
> 
> That has a runtime value, but not a useful compile-time value.  The
> only useful thing you can do with it in an #if expression is
> defined(PTHREAD_STACK_MIN).

Ok.

>> 2.  I don't know what to make of the failure of test_threading on Linux,
>> as test_thread succeeds as far as I could see.  These tests succeed on my
>> FreeBSD box and also appear to be succeeding on the Windows buildbots.
> 
> Not all pthreads-using builds fail, and not all failing pthreads-using
> builds fail in the same way.  Welcome to pthreads on Linux ;-)
> 
> BTW, this sucks:
> 
> test_thread
> /home/buildbot/Buildbot/trunk.baxter-ubuntu/build/Lib/test/test_thread.py:138:
>  
> 
> RuntimeWarning: thread stack size of 0x1000 bytes not supported
>  thread.stack_size(tss)
> 
> That's from a successful run.  RuntimeWarning really doesn't make
> sense for a failing operation.  This should raise an exception
> (xyzError, not xyzWarning), or a failing stack_size() should return an
> error value after ensuring the original stack size is still in effect.

Fair enough.

{...}

> If PyThread_start_new_thread() fails in any way
> (like,pthread_attr_setstacksize() failing), ""can't start new thread"
> is the error we see.
> 
> The difference between test_thread and test_threading here is that
> only test_threading asks for a 16MB stack; test_thread doesn't ask for
> a stack larger than 4MB.

Thanks for the analysis!

> Until all this gets resolved, I strongly suggest reverting this patch
> (if you don't, someone else will ...) and hammering out the problems
> on a new branch instead.  See python-dev email from yesterday for how
> to force a buildbot slave to build a branch.

I see that you've already reverted this - Thanks & sorry I couldn't get
to it quickly.

Regards,
Andrew.

-
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: [EMAIL PROTECTED]  (pref) | Snail: PO Box 370
[EMAIL PROTECTED] (alt) |Belconnen ACT 2616
Web:http://www.andymac.org/   |Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch #1454481 vs buildbot

[Andrew MacIntyre]

[EMAIL PROTECTED] wrote:

> Andrew, look in your mail for a patch file.

Received, thanks.
Andrew.

-
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: [EMAIL PROTECTED]  (pref) | Snail: PO Box 370
[EMAIL PROTECTED] (alt) |Belconnen ACT 2616
Web:http://www.andymac.org/   |Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Let's stop eating exceptions in dict lookup

[Steve Holden]

Raymond Hettinger wrote:
> Armin Rigo wrote:
[...]
>>At the moment, I'm trying to, but 2.5 HEAD keeps failing mysteriously on
>>the tests I try to time, and even going into an infinite loop consuming
>>all my memory - since the NFS sprint.  Am I allowed to be grumpy here,
>>and repeat that speed should not be used to justify bugs?  I'm proposing
>>a bug fix, I honestly don't care about 0.5% of speed.
>>  
>>
> If it is really 0.5%, then we're fine.  Just remember that PyStone is an 
> amazingly uninformative and crappy benchmark.
> 
Which sadly doesn't distinguish it particularly, since all benchmarks 
tend towards the uninformative and crappy.

> The "justify bugs" terminology is pejorative and inaccurate.  It is 
> clear that the current dict behavior was a concious design decision and 
> documented as such.  Perhaps the decision sucked and should be changed, 
> but it is not a bug.
> 
> 
>> and I consider
>>myself an everyday Python user,
>>  
>>
> 
> Something may have been lost in translation.  Using it every day is not 
> the same as being an everyday user ;-)  There is no doubt that you 
> routinely stress the language in ways ways that are not at all commonplace.
> 
Just the same, I think Armin's point that the change in behavior might 
induce breakage in "working" programs is one we need to consider 
carefully, even though programs relying on the current behaviour might 
reasonably be considered broken (for some value of "broken").

> All I'm asking is that there be a well thought-out assessment of whether 
> the original design decision was struck the appropriate balance between 
> practicality
> 
I think the discussion has touched all the major points, but this won't 
stave off the squeals of anguish form those running programs that break 
under 2.5.

There are some similarities between this change and the (2.0?) change 
that stopped socket.socket() from accepting two arguments. IMHO I think 
we should accept that the behaviour needs to change and be prepared for 
a few anguished squeals. FWIW I suspect they will be even fewer than 
anticipated.

regards
  Steve
-- 
Steve Holden   +44 150 684 7255  +1 800 494 3119
Holden Web LLC/Ltd  http://www.holdenweb.com
Love me, love my blog  http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python Benchmarks

[Steve Holden]

M.-A. Lemburg wrote:
> Fredrik Lundh wrote:
> 
>>M.-A. Lemburg wrote:
>>
>>
>>>Seriously, I've been using and running pybench for years
>>>and even though tweaks to the interpreter do sometimes
>>>result in speedups or slow-downs where you wouldn't expect
>>>them (due to the interpreter using the Python objects),
>>>they are reproducable and often enough have uncovered
>>>that optimizations in one area may well result in slow-downs
>>>in other areas.
>>
>> > Often enough the results are related to low-level features
>> > of the architecture you're using to run the code such as
>> > cache size, cache lines, number of registers in the CPU or
>> > on the FPU stack, etc. etc.
>>
>>and that observation has never made you stop and think about whether 
>>there might be some problem with the benchmarking approach you're using? 
> 
> 
> The approach pybench is using is as follows:
> 
> * Run a calibration step which does the same as the actual
>   test without the operation being tested (ie. call the
>   function running the test, setup the for-loop, constant
>   variables, etc.)
> 
>   The calibration step is run multiple times and is used
>   to calculate an average test overhead time.
> 
I believe my recent changes now take the minimum time rather than 
computing an average, since the minimum seems to be the best reflection 
of achievable speed. I assumed that we wanted to measure achievable 
speed rather than average speed as our benchmark of performance.

> * Run the actual test which runs the operation multiple
>   times.
> 
>   The test is then adjusted to make sure that the
>   test overhead / test run ratio remains within
>   reasonable bounds.
> 
>   If needed, the operation code is repeated verbatim in
>   the for-loop, to decrease the ratio.
> 
> * Repeat the above for each test in the suite
> 
> * Repeat the suite N number of rounds
> 
> * Calculate the average run time of all test runs in all rounds.
> 
Again, we are now using the minimum value. The reasons are similar: if 
extraneous processes interfere with timings then we don't want that to 
be reflected in the given timings. That's why we now report "notional 
minimum round time", since it's highly unlikely that any specific test 
round will give the minimum time for all tests.

Even with these changes we still see some disturbing variations in 
timing both on Windows and on Unix-like platforms.
> 
>>  after all, if a change to e.g. the try/except code slows things down 
>>or speed things up, is it really reasonable to expect that the time it
>>takes to convert Unicode strings to uppercase should suddenly change due 
>>to cache effects or a changing number of registers in the CPU?  real 
>>hardware doesn't work that way...
> 
> 
> Of course, but then changes to try-except logic can interfere
> with the performance of setting up method calls. This is what
> pybench then uncovers.
> 
> The only problem I see in the above approach is the way
> calibration is done. The run-time of the calibration code
> may be to small w/r to the resolution of the used timers.
> 
> Again, please provide the parameters you've used to run the
> test case and the output. Things like warp factor, overhead,
> etc. could hint to the problem you're seeing.
> 
> 
>>is PyBench perhaps using the following approach:
>>
>> T = set of tests
>> for N in range(number of test runs):
>> for t in T:
>> t0 = get_process_time()
>> t()
>> t1 = get_process_time()
>> assign t1 - t0 to test t
>> print assigned time
>>
>>where t1 - t0 is very short?
> 
> 
> See above (or the code in pybench.py). t1-t0 is usually
> around 20-50 seconds:
> 
> """
> The tests must set .rounds to a value high enough to let the
> test run between 20-50 seconds. This is needed because
> clock()-timing only gives rather inaccurate values (on Linux,
> for example, it is accurate to a few hundreths of a
> second). If you don't want to wait that long, use a warp
> factor larger than 1.
> """
> 
First, I'm not sure that this is the case for the default test 
parameters on modern machines. On my current laptop, for example, I see 
a round time of roughly four seconds and a notional minimum round time 
of 3.663 seconds.

Secondly, while this recommendation may be very sensible, with 50 
individual tests a decrease in the warp factor to 1 (the default is 
currently 20) isn't sufficient to increase individual test times to your 
recommended value, and decreasing the warp factor tends also to decrease 
reliability and repeatability.

Thirdly, since each round of the suite at warp factor 1 takes between 80 
and 90 seconds, pybench run this way isn't something one can usefully 
use to quickly evaluate the impact of a single change - particularly 
since even continuing development work on the benchmark machine 
potentially affects the benchmark results in unknown ways.

> 
>>that's not a very good idea, gi

Re: [Python-Dev] [Python-checkins] Python Regression Test Failures refleak (1)

[Tim Peters]

[moving to python-dev]

[Tim, gets different results across whole runs of
python_d  ../Lib/test/regrtest.py -R 2:40: test_filecmp test_exceptions
]
>>> Does that make any sense?  Not to me -- I don't know of a clear reason
>>> other than wild loads/stores for why such runs should ever differ.

[Neal]
>> The problem generally has to do with modules cacheing things.

[Martin]
> Then the numbers should repeat on each run.

Right.  The point wasn't that I saw a variety of different integers in
the "leak vector" in a single run, it was that I got different results
_across_ runs:  no leaks in either test, a leak in one but not the
other, or (very rarely) leaks in both.

> So wild loads/stores are a more compelling explanation. Of course, it
> *should* even be repeatable with wild loads/stores, unless the OS
> shuffles the address space on each run (which at least Linux,
> unfortunately, does).

Well, I just tried this (illegal) C program under VC 7.1:

#include 
#include 

int main() {
int *p;
int i, sum;

p = (int *)malloc(sizeof(int));
printf("%p %d ...", p, sum);
for (sum = 0, i = -12; i < 29; ++i)
sum += p[i];
printf("%d\n", sum);

return 0;
}

Here are several runs.  Note that the address malloc() returns is
always the same, but adding the junk "around" that address often gives
a different result, and the stack trash `sum` contains at first also
varies.  Put those together, and they show that wild loads from stack
trash _and_ heap trash can vary across runs:

C:\Code>cl boogle.c
Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 13.10.3077 for 80x86
Copyright (C) Microsoft Corporation 1984-2002. All rights reserved.

boogle.c
c:\code\boogle.c(9) : warning C4700: local variable 'sum' used without
having been initialized
Microsoft (R) Incremental Linker Version 7.10.3077
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:boogle.exe
boogle.obj

C:\Code>for %i in (1 2 3 4 5 6 7 8 9) do boogle.exe

C:\Code>boogle.exe
00322EA8 315894624 ...75845519

C:\Code>boogle.exe
00322EA8 316050874 ...125913836

C:\Code>boogle.exe
00322EA8 316050874 ...125913836

C:\Code>boogle.exe
00322EA8 316207124 ...8930763

C:\Code>boogle.exe
00322EA8 316207124 ...8930763

C:\Code>boogle.exe
00322EA8 316207124 ...8930763

C:\Code>boogle.exe
00322EA8 316363374 ...42224689

C:\Code>boogle.exe
00322EA8 316363374 ...42224689

C:\Code>boogle.exe
00322EA8 316519624 ...92948238

How did I pick -12 and 29 for i's bounds?  Easy:  I started with much
larger bounds, and reduced them haphazardly until the program stopped
segfaulting :-)

Now I hate to think this is "the cause" for regrtest -R varying across
identical runs, but I don't have many other suspects in mind.  For
example, I tried forcing random.seed() at the start of
regrtest.main(), but that didn't make any visible difference.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] wsgiref documentation

[Phillip J. Eby]

At 08:08 AM 6/5/2006 -0400, A.M. Kuchling wrote:
>I had the start of an outline in sandbox/wsgiref-docs, but am not
>working on them at the moment because no one is willing to say if the
>list of documented classes is complete (or includes too much).

Huh?  This is the first I've heard of it.

I was already working on some documentation in my local tree, though, so 
I've now started merging your work into it and checked in a snapshot at:

 http://svn.eby-sarna.com/wsgiref/docs/

I'll merge more into it later.  If anyone has any part of the remaining 
stuff that they'd like to volunteer to document, please let me know so I 
don't duplicate your work.  Thanks.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] wsgiref documentation

[A.M. Kuchling]

On Mon, Jun 05, 2006 at 06:33:29PM -0400, Phillip J. Eby wrote:
> At 08:08 AM 6/5/2006 -0400, A.M. Kuchling wrote:
> >I had the start of an outline in sandbox/wsgiref-docs, but am not
> >working on them at the moment because no one is willing to say if the
> >list of documented classes is complete (or includes too much).
> 
> Huh?  This is the first I've heard of it.

,
and the checkins to sandbox/wsgiref-docs/ would also have been an
indicator.  But if Doug wants to write the docs, he certainly should;
we can always use more contributors.

--amk
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Stdlib Logging questions (PEP 337 SoC)

[Jim Jewett]

On 6/4/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
> Jim> (2)  Should NAME be module.__name__?

> Seems reasonable.

(The clipped part was that the output will look a bit different when,
say, the module is run as a script and the name is __main__).

But if no one objects, I'll take this as a "good enough", since I
really don't like repeating the module name.

> Jim> (3)  Should she put out a message when a (now logged) module is
> Jim>  loaded?  If so, at what level?

> -1.  I don't think it buys you anything.  If you need it, the -v command
> line flag should be sufficient.

The timestamp could be useful, though it doesn't show up in the
default basicformat.

Realistically, I think the main reason is consistency with JSPs, which
(at high enough logging levels) produce huge amounts of trace
information.  If no one else chimes in, I'll consider the parallel
uncompelling.

(6)  And a new issue -- the PEP says py.modulename -- is it reasonable
to use something more specific?

The use case is again from asyncore; the existing API has separate
methods for "logging" a hit and "logging" instrumentation messages.
Apache would send these to two separate files (access_log and
error_log); it seems reasonable to support at least the possibility of
handlers treating the two types of messages separately.

If no explicit changes are made locally,

py.asyncore.dispatcher.hits
py.asyncore.dispatcher.messages

would both roll up to (PEP337) py.asycnore (and then to the root logger).

-jJ
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Stdlib Logging questions (PEP 337 SoC)

[Jim Jewett]

On 6/4/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote:
> can we please delay the import until it's actually needed?  i.e.,
> until after some logging option is enabled?

I have asked her to make this change.

I don't like the extra conditional dance it causes, but I agree that
not wanting to log is a valid use case.

On the other hand, the one-time import cost is pretty low for a
long-running process, and eventually gets paid if any other module
calls logging.  Would it make more sense to offer a null package that
can be installed earlier in the search path if you want to truly
disable logging?

-jJ
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Stdlib Logging questions (PEP 337 SoC)

[Phillip J. Eby]

At 09:04 PM 6/5/2006 -0400, Jim Jewett wrote:
>On 6/4/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote:
> > can we please delay the import until it's actually needed?  i.e.,
> > until after some logging option is enabled?
>
>I have asked her to make this change.
>
>I don't like the extra conditional dance it causes, but I agree that
>not wanting to log is a valid use case.
>
>On the other hand, the one-time import cost is pretty low for a
>long-running process, and eventually gets paid if any other module
>calls logging.  Would it make more sense to offer a null package that
>can be installed earlier in the search path if you want to truly
>disable logging?

I notice you've completely avoided the question of whether this should be 
being done at all.  It sounds like Fredrik is -1 on this even for the 
modules that I'm not -1 on.

As far as I can tell, this PEP hasn't actually been discussed.  Please 
don't waste time changing modules for which there is no consensus that this 
*should* be done.

The original discussion that occurred prior to PEP 337's creation discussed 
only modules that *already* do some kind of logging.  There was no 
discussion of changing *all* debugging output to use the logging module, 
nor of adding logging to modules that do not even have any debugging output 
(e.g. pkgutil).

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com