Re: [Python-Dev] Using logging in the stdlib and its unit tests

2010-12-11 Thread Nick Coghlan 
On Sat, Dec 11, 2010 at 4:25 PM, Glenn Linderman  wrote:
> On 12/10/2010 9:24 PM, Nick Coghlan wrote:
> This could actually make a reasonably good basic for a "task oriented"
> subsection of the logging documentation. Something like:
>
> Yep, agree.  But sadly, for each point, there may be multiple options (your
> StreamHandler, but I'd want a FileHandler; your separation of messages by
> level, my wanting them combined; etc.)

No, no, no, that's the *whole point* of using logging. The library
writer doesn't really *care* about where the messages end up - that is
entirely in the hands of the application developer when they choose
which handlers to install. The only situation that the library writer
cares about is the one that tripped up concurrent.futures and that has
already been changed for 3.2b2: that warnings and errors written to a
logger were silenced by default if the application never even called
basicConfig(). That has now been fixed so they will be emitted on
sys.stderr instead (including obeying any redirections of stderr over
the lifetime of the program).

Notice that my task list is *entirely* from the point of view of the
person emitting the messages. How those messages are later displayed
is then up to the application writer (or the logging module default
settings, if the application writer doesn't do anything specific).

> Your comment about basicConfig setting the level on the root logger, but not
> on the default handler making it useless is opaque to me,

An early version of my code used basicConfig to create the stderr
StreamHandler, but that turned out to be pointless since I wanted
different levels on the logger and the handler.

> but is there
> perhaps room for another basic setup API that could get the setup code down
> to a line or two in simple cases?

For 3.2? No.

For 3.3? Maybe.

> Would that be a useful set of functionality to bundle?  And could it be
> extended, when the user wants more power, or would it have to be replaced,
> because it gets in the way of the user that wants more power?

Logging already has powerful configuration mechanisms, especially
following the addition of dictConfig
(http://docs.python.org/dev/library/logging#logging-config-dictschema),
so it really doesn't need anything else along those lines.

At the simpler end, basicConfig already covers sending all messages to
a single stream with a single format - it's only split-stream handling
(such as stderr/stdout potentially being different endpoints) that it
can't cope with.

What may make more sense than yet another global config mechanism, is
a module level "addHandler" helper function along the following lines:

from logging import Formatter, FileHandler, StreamHandler, getLogger
def addHandler(*, handler=None, stream=None, filename=None, filemode='a',
  format=None, datefmt=None, style='{',
  level=None, max_level=None, filters=(),
  logger=None):
"""stream, filename, level, format, datefmt, style: as per
logging.basicConfig

handler: use a precreated handler instead of creating a new one
logger: logger to add the handler to (uses root logger if none
specified)
filters: an iterable of filters to add to the handler
max_level: can optionally limit the handler to messages of a
certain level and below
"""
# Create the handler if one hasn't been passed in
if handler is None:
if filename is not None:
handler = FileHandler(filename, filemode)
else:
handler = StreamHandler(stream)
# Set up the formatting of the log messages
# New API, so it can default to str.format instead of %-formatting
formatter = Formatter(format, datefmt, style)
handler.setFormatter(formatter)
# Set up filtering of which messages to handle
if level is not None:
handler.setLevel(level)
if max_level is not None:
def level_ok(record):
return record.levelno <= max_level
handler.addFilter(level_ok)
for filter in filters:
handler.addFilter(filter)
# Add the fully configured handler to the specified logger
if logger is None:
logger = getLogger()
logger.addHandler(handler)
return handler

# Previous set up example is now only three lines long
import sys, logging
# Let root logger handlers see all messages
logging.getLogger().setLevel(logging.NOTSET)
# Send WARNING and above to stderr
addHandler(stream=sys.stderr, level=logging.WARNING)
# Send INFO to stdout
addHandler(stream=sys.stdout, level=logging.INFO, max_level=logging.INFO)

logging.info("Hello world!")
logging.warn("Hello world!")

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail

Re: [Python-Dev] Using logging in the stdlib and its unit tests

2010-12-11 Thread Nick Coghlan 
On Sat, Dec 11, 2010 at 6:00 PM, Nick Coghlan  wrote:
> What may make more sense than yet another global config mechanism, is
> a module level "addHandler" helper function along the following lines:

For a more readable version of that example, take a look at the copy I
put up over on the Python Cookbook site:
http://code.activestate.com/recipes/577496-simple-creation-configuration-and-installation-of-/

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API for the new sysconfig module

2010-12-11 Thread Tarek Ziadé 
Sorry to get late in the discussion I am travelling.

Nick resumes well the motivations behind sysconfig.

I'll emphase that this module could hold more functions in the future that
could be useful to other python implementations to abstract what is a python
installation. E.g. more than paths and variables.

On Dec 11, 2010 3:53 AM, "Nick Coghlan"  wrote:

On Sat, Dec 11, 2010 at 7:59 AM, R. David Murray 
wrote:
> On Thu, 09 Dec 201...

>> ISTM it mostly could have been reduced to single call returning a nested
dictionary.
>
> If what ...
We've reverted from beta to pseudo-alpha after discovering
sufficiently egregious mistakes in the past. (we didn't *actually*
revert the build naming because release IDs that don't increase
monotonically can confuse matters). However, I don't think this is
sufficiently egregious to qualify.

My own reasons for advocating retention of the separate module:

1. Something equivalent to the current sysconfig module is going to be
needed to actually work out the contents of the data structure.
Rewriting all that in C would be silly, so the most that could happen
is to rename "sysconfig" to "_sysconfig".

2. Invoking Python code from sys module functions is possible but
untidy (generally involving implicit module imports). Since we need a
module anyway for implementation purposes, why not make it public?

3. The sysconfig module docs provide a nicer space to talk about the
purpose of the provided information rather than squirreling it away in
a corner of the sys module docs

4. "python -m sysconfig" provides a useful dump of the system status

5. The API design did not get much *new* thought, as the idea was to
allow users to do anything that the old distutils.sysconfig could do,
but without requiring them to depend on distutils to get that
functionality. The easiest way to ensure that is to just copy the
existing API rather than trying to get creative. Don't think of it as
adding a "whole new module", thinking of it as decoupling a relatively
small existing module (i.e. distutils.sysconfig) from a larger package
that isn't installed by default on all systems (i.e. distutils).

A concrete problem with Raymond's idea in particular is that there is
actually a full set of paths defined for *each scheme*. get_path() and
get_paths() let you ignore this most of the time, since there is an
appropriate default scheme defined for each platform, so they just use
that if you don't specify one explicitly. A named tuple could
obviously expose the paths for all of the schemes, with the paths for
the default scheme duplicated at the top level, but the data structure
starts looking very messy at that point. The named tuple idea also
conflates meta-data about the sysconfig data (the list of scheme names
and path names) with the actual data for the current platform
(platform, version, paths, config_vars).

So I think reusing the tried and tested API is easily justified in
this case. What may be desirable in 3.3 (particularly with the
distutils2 reintegration taking place), is to revisit the sysconfig
API to see if there are better ways to expose some of the data (such
as a function returning a named tuple containing the raw data
underlying the output of "python -m sysconfig").


Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
...

Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Uns...
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using logging in the stdlib and its unit tests

2010-12-11 Thread Nick Coghlan 
On Sat, Dec 11, 2010 at 4:18 PM, Vinay Sajip  wrote:
>> formatting (when the documentation for the new styles feature goes in
>> would probably be an appropriate time to fix this). Similarly, the
>
> Sorry, what do you mean by "new styles feature"?

The switching between percent, str.format and string.Template
formatting. It turns out that has been documented, but the table in
question is still written from a percent-style formatting point of
view.

>> Anyway, the shortest way I could find of setting this up (debug
>
> Yes, except that you could have set the root logger level to INFO rather than
> DEBUG.
>
> This raises some more questions. Obviously, we don't want users to have to go
> through these steps themselves. Once these steps have been finalised, the last
> resort handler could be modified to do all this. Should this handler be part 
> of
> the public API, so that it can be inherited from etc?

The "lazy" stream handler might be useful to make public in some way.
For example, rather than hardcoding sys.stderr, it could take a
callable that it uses to retrieve the stream. That kind of change can
wait until 3.3 though - what is there now should be fine to let us get
concurrent.futures and the logging tests to play nicely together.

As far as whether anything else in the logging defaults needs to
change, I don't think so. I'm now happy that if an application needs
to emit text on stdout as part of its normal operation, then that
isn't what the logging module is for - logging messages are intended
to record "this happened, then that happened, then this other thing
happened". They are *not* properly part of an application's console UI
(which is reflected in the whole "silent by default" and "console
output defaults to sys.stderr" philosophy). The change to the default
behaviour was just to make logging.warning() etc a suitable
replacement for writing directly to sys.stderr when exceptions can't
be raised, which is in keeping with the existing spirit of the logging
module.

> I realise we're in beta and hence feature freeze ... just sayin'. We're all
> consenting adults here, after all :-)

I don't think we should be adding more *code* at this stage. But
documentation fixes and adjustments may be a good idea. I've learned
quite a bit myself about the logging module in the course of this
discussion, so it would be nice if some of that could be captured and
used to improve the documentation.

That said, something that might be interesting is to see what my
addHandler() recipe [1] would do for the examples in the logging
documentation. If it makes a big enough difference, then it may be
worth talking to Georg about adding it, even for 3.2. Alternatively,
if you like the idea, but we don't want to break the feature freeze
for it, then it may make a nice "see-also" link from the docs. I find
it useful from a "here are all the things I can configure on a handler
in one easy bundle" point of view, even aside from any benefits in
reduced typing.

Cheers,
Nick.

[1] 
http://code.activestate.com/recipes/577496-simple-creation-configuration-and-installation-of-/

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using logging in the stdlib and its unit tests

2010-12-11 Thread Vinay Sajip 
Glenn Linderman  g.nevcal.com> writes:

> Yep, agree.  But sadly, for each point, there may be multiple
> options (your StreamHandler, but I'd want a FileHandler; your
> separation of messages by level, my wanting them combined; etc.)

That's partly why logging hasn't made much effort in the "default" department:
everyone wants something different.

> Your comment about basicConfig setting the level on the root logger,
> but not on the default handler making it useless is opaque to me,

I think he just means that he can't use basicConfig because it just sets the
root logger's level to whatever's passed in, but what he actually wants to set
are two different levels: INFO on the logger, WARNING on the stderr handler.
On top of that, he wants to have a stdout handler which only outputs INFO
events, and to get that effect he has to add the filter to the stdout handler
(otherwise it would output INFO + anything at a higher level, too).

> but is there perhaps room for another basic setup API that could get
> the setup code down to a line or two in simple cases?  Maybe 3
> parameters:
[snip]
> Would that be a useful set of functionality to bundle?  And could it
> be extended, when the user wants more power, or would it have to be
> replaced, because it gets in the way of the user that wants more
> power?
> 

It's possible that perhaps there could be some 'canned configs', as I've
mentioned in another post on this thread. Some more thinking needs to be done
around this area.

Regards,

Vinay Sajip


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using logging in the stdlib and its unit tests

2010-12-11 Thread Nick Coghlan 
On Sat, Dec 11, 2010 at 3:06 PM, Nick Coghlan  wrote:
> However, the confusion that this setup will engender is that
> encountered by Bill: by default, info() messages are silenced rather
> than displayed on stdout.
>
> Notice that even the recommended "basicConfig" approach to resolving
> this is subtly flawed, since your choices are to either display info
> messages on stderr (by specifying just a level argument) or to
> redirect warning() et al to stdout instead of stderr (by also
> specifying a stream argument). The "correct" answer (info and below to
> stdout, warning and above to stderr) is actually quite tricky to set
> up, since handlers have no innate concept of a "maximum level", so you
> have to provide a filter function (more on that below [1]).

I need to correct this comment, since it's wrong and
basicConfig(level=logging.INFO) actually does the right thing.

As Glenn mentioned later in the thread, the output of logging.info and
logging.debug messages is *distinct* from an application's normal
operational output that is emitted on stdout. So making it easy to
emit such messages on stderr is the right thing to do - it's OK that
it requires a bit of additional set up to get them onto stdout instead
(since they really don't belong there most of the time).

I know my own rule of thumb is going to be updated along the lines of
"Am I writing to sys.stderr, or guarding output with a verbosity flag?
Then perhaps I should be using the logging module rather than print
statements".

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using logging in the stdlib and its unit tests

2010-12-11 Thread Vinay Sajip 
Nick Coghlan  gmail.com> writes:

> The switching between percent, str.format and string.Template
> formatting. It turns out that has been documented, but the table in
> question is still written from a percent-style formatting point of
> view.

Oh, right - yes. I presume you mean the table in the formatter documentation
which shows %(message)s etc.
 

> The "lazy" stream handler might be useful to make public in some way.
> For example, rather than hardcoding sys.stderr, it could take a
> callable that it uses to retrieve the stream. That kind of change can
> wait until 3.3 though - what is there now should be fine to let us get
> concurrent.futures and the logging tests to play nicely together.

Actually if we're to change things to print INFO to stdout and WARNING+ to
stderr, I'd change the last resort handler class to be more than just defining
a lazy stream property: perhaps something like [untested]

class _DefaultHandler(StreamHandler):
def __init__(self, level=NOTSET):
"""
Initialize the handler.
"""
Handler.__init__(self, level)

def emit(self, record):
if record.levelno == INFO:
self.stream = sys.stdout
else:
self.stream = sys.stderr
super(_DefaultHandler, self).emit(record)

> As far as whether anything else in the logging defaults needs to
> change, I don't think so. I'm now happy that if an application needs

I agree it can be refined later, but if we agreed that INFO goes to stdout,
then something like the above could be put in place for 3.2 (perhaps 3.2b2).
 
> output defaults to sys.stderr" philosophy). The change to the default
> behaviour was just to make logging.warning() etc a suitable
> replacement for writing directly to sys.stderr when exceptions can't
> be raised, which is in keeping with the existing spirit of the logging
> module.

Agreed.
 
> I don't think we should be adding more *code* at this stage. But
> documentation fixes and adjustments may be a good idea. I've learned
> quite a bit myself about the logging module in the course of this
> discussion, so it would be nice if some of that could be captured and
> used to improve the documentation.

Yes, you (and Glenn, and others) have given me quite a bit to work on, on that
score!
 
> That said, something that might be interesting is to see what my
> addHandler() recipe [1] would do for the examples in the logging
> documentation. If it makes a big enough difference, then it may be
> worth talking to Georg about adding it, even for 3.2. Alternatively,
> if you like the idea, but we don't want to break the feature freeze
> for it, then it may make a nice "see-also" link from the docs. I find
> it useful from a "here are all the things I can configure on a handler
> in one easy bundle" point of view, even aside from any benefits in
> reduced typing.

Actually I've been thinking about a handlers= optional argument for basicConfig
for some time (which, if provided, would override file and stream arguments).
But it seemed a little unwieldy and came across plainly as the afterthought it
admittedly was :-)

Regards,

Vinay Sajip


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using logging in the stdlib and its unit tests

2010-12-11 Thread Glenn Linderman 

On 12/11/2010 12:00 AM, Nick Coghlan wrote:

On Sat, Dec 11, 2010 at 4:25 PM, Glenn Linderman  wrote:

On 12/10/2010 9:24 PM, Nick Coghlan wrote:
This could actually make a reasonably good basic for a "task oriented"
subsection of the logging documentation. Something like:

Yep, agree.  But sadly, for each point, there may be multiple options (your
StreamHandler, but I'd want a FileHandler; your separation of messages by
level, my wanting them combined; etc.)

No, no, no, that's the *whole point* of using logging. The library
writer doesn't really *care* about where the messages end up - that is
entirely in the hands of the application developer when they choose
which handlers to install. The only situation that the library writer
cares about is the one that tripped up concurrent.futures and that has
already been changed for 3.2b2: that warnings and errors written to a
logger were silenced by default if the application never even called
basicConfig(). That has now been fixed so they will be emitted on
sys.stderr instead (including obeying any redirections of stderr over
the lifetime of the program).

Notice that my task list is *entirely* from the point of view of the
person emitting the messages. How those messages are later displayed
is then up to the application writer (or the logging module default
settings, if the application writer doesn't do anything specific).


Yes, yes, yes... that's what your list was!  I see that now!  But it is 
interesting that in missing that point, and reading your logging setup 
code, I saw that the same list of tasks is also exactly the sorts of 
things that you would want to control, for simple uses of the logger.  
Does basicConfig give that control?  I don't know, the first 8% of the 
logger documentation don't tell me that, they only give me a canned 
example (actually 3) without explanation of its full function.  Someday 
I'll read more.  Until then, basicConfig is opaque: I know only three 
incantations for using it, and not even any surety that they can be 
mixed, or that it can be called more than once in the same program.




Your comment about basicConfig setting the level on the root logger, but not
on the default handler making it useless is opaque to me,

An early version of my code used basicConfig to create the stderr
StreamHandler, but that turned out to be pointless since I wanted
different levels on the logger and the handler.


but is there
perhaps room for another basic setup API that could get the setup code down
to a line or two in simple cases?

For 3.2? No.

For 3.3? Maybe.


Would that be a useful set of functionality to bundle?  And could it be
extended, when the user wants more power, or would it have to be replaced,
because it gets in the way of the user that wants more power?

Logging already has powerful configuration mechanisms, especially
following the addition of dictConfig
(http://docs.python.org/dev/library/logging#logging-config-dictschema),
so it really doesn't need anything else along those lines.


Sure; that is the replacement technique.  And maybe that is OK, if there 
are no useful intermediates between basicConfig (whatever it might be 
capable of) and dictConfig (or the other various config techniques that 
preceded it and still exist, if I understand correctly from has been 
said here on python-dev).



At the simpler end, basicConfig already covers sending all messages to
a single stream with a single format - it's only split-stream handling
(such as stderr/stdout potentially being different endpoints) that it
can't cope with.


OK, so I'd probably be fine with basicConfig, even though you weren't, 
once I understand it a little better.



What may make more sense than yet another global config mechanism, is
a module level "addHandler" helper function along the following lines:


And once I learn enough to understand the addHandler code, I will have 
read a lot of the logger documentation than I have to date, or it will 
have been restructured enough that re-reading the first 8%, or maybe 12% 
if the easy-to-understand part grows!  But I like what you said here:



# Previous set up example is now only three lines long
import sys, logging
# Let root logger handlers see all messages
logging.getLogger().setLevel(logging.NOTSET)
# Send WARNING and above to stderr
addHandler(stream=sys.stderr, level=logging.WARNING)
# Send INFO to stdout
addHandler(stream=sys.stdout, level=logging.INFO, max_level=logging.INFO)

logging.info("Hello world!")
logging.warn("Hello world!")


And this subset usage of addHandler is reasonably understandable for 
beginners, although clearly the complexity of all the parameters must 
add power that is not needed in the simple case, that you feel could be 
useful in the complex case.  And I would change the third line from 
level=logging.INFO to level=logging.DEBUG to see lots more on stdout, 
correct?


Looking back at the logger's simple examples section, I see that 
getLogger is used before being define

Re: [Python-Dev] Using logging in the stdlib and its unit tests

2010-12-11 Thread Glenn Linderman 

On 12/11/2010 1:28 AM, Vinay Sajip wrote:

Nick Coghlan  gmail.com>  writes:

The "lazy" stream handler might be useful to make public in some way.
For example, rather than hardcoding sys.stderr, it could take a
callable that it uses to retrieve the stream. That kind of change can
wait until 3.3 though - what is there now should be fine to let us get
concurrent.futures and the logging tests to play nicely together.

Actually if we're to change things to print INFO to stdout and WARNING+ to
stderr, I'd change the last resort handler class to be more than just defining
a lazy stream property: perhaps something like [untested]

class _DefaultHandler(StreamHandler):
 def __init__(self, level=NOTSET):
 """
 Initialize the handler.
 """
 Handler.__init__(self, level)

 def emit(self, record):
 if record.levelno == INFO:
 self.stream = sys.stdout
 else:
 self.stream = sys.stderr
 super(_DefaultHandler, self).emit(record)


So I think Nick's configuration stuff is something that is supposed to 
be done by the application, not by the library, or am I missing that 
too?  And so if it is done by the application, then Nick's, or any other 
application writer's, preferences for where to send which messages can 
likely be accommodated.  But I think that by default anything that is 
not part of the application's defined output, should go to stderr... 
even INFO messages.  I have no problem with Nick wanting them on 
stdout,  but I would have a problem with libraries (or the logger of 
last resort) sending them there by default.




As far as whether anything else in the logging defaults needs to
change, I don't think so. I'm now happy that if an application needs

I agree it can be refined later, but if we agreed that INFO goes to stdout,
then something like the above could be put in place for 3.2 (perhaps 3.2b2).


I'd find that hard to agree to, but I probably don't count.  I like 
Nick's statement.  Libraries spewing messages to stderr is fine, spewing 
them to stdout is not, by default.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using logging in the stdlib and its unit tests

2010-12-11 Thread Glenn Linderman 

On 12/11/2010 1:07 AM, Nick Coghlan wrote:

As Glenn mentioned later in the thread, the output of logging.info and
logging.debug messages is*distinct*  from an application's normal
operational output that is emitted on stdout. So making it easy to
emit such messages on stderr is the right thing to do - it's OK that
it requires a bit of additional set up to get them onto stdout instead
(since they really don't belong there most of the time).

I know my own rule of thumb is going to be updated along the lines of
"Am I writing to sys.stderr, or guarding output with a verbosity flag?
Then perhaps I should be using the logging module rather than print
statements".


I'm in agreement with all this.  Thanks for the correction.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using logging in the stdlib and its unit tests

2010-12-11 Thread Nick Coghlan 
On Sat, Dec 11, 2010 at 7:28 PM, Vinay Sajip  wrote:
> Nick Coghlan  gmail.com> writes:
> Actually if we're to change things to print INFO to stdout and WARNING+ to
> stderr,

I suspect our messages crossed paths in mid-stream, but I actually
realised I was wrong on that front - the behaviour already provided by
logging.basicConfig is the better way to go. The missing piece in the
story of "how do I create X kind of output" is plain old vanilla print
statements and similar direct writes to sys.stdout. The logging module
is a tool to track what is happening in a program, not a tool for
providing a console based UI.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] porting python.org

2010-12-11 Thread Prashant Kumar 
I was wondering if we could contribute in porting of python.org website over
to python3k. If we can, what are the steps to be taken and other necessary
requirements?

regards,
prashant
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using logging in the stdlib and its unit tests

2010-12-11 Thread Vinay Sajip 
Glenn Linderman  g.nevcal.com> writes:


> the logger.  Does basicConfig give that control?  I don't know, the
> first 8% of the logger documentation don't tell me that, they only
> give me a canned example (actually 3) without explanation of its
> full function.  Someday I'll read more.  Until then, basicConfig is
> opaque: I know only three incantations for using it, and not even
> any surety that they can be mixed, or that it can be called more
> than once in the same program.

You've made some helpful observations about improving the logging documentation.
I'm grateful to you for those and would like to incorporate your feedback. But I
feel you're making somewhat heavy weather about basicConfig() and the "8%". It's
just 20 lines of code and not especially complicated code at that. It would
probably take you less time to look at that code than to post again about how
little information there is in the top 8% of the logging documentation, I think
you've got your point across. Here are a couple of hopefully useful links about
basicConfig():

From the 2.7.1 docs - documentation gleaned from the docstring:

http://docs.python.org/library/logging.html#logging.basicConfig

From the source code for 2.7.1, docstring followed by the code:

http://svn.python.org/view/python/branches/release27-maint/Lib/logging/__init__.py?revision=87157&view=markup#basicConfig

I will try to incorporate more basic examples at the top of the documentation,
but surely you don't want me to add more verbiage about basicConfig when your
overall feeling is that there's too much documentation?

Regards,

Vinay Sajip


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using logging in the stdlib and its unit tests

2010-12-11 Thread Vinay Sajip 
Nick Coghlan  gmail.com> writes:

> the logging module is a tool to track what is happening in a program, not a
> tool for providing a console based UI.

That was certainly the thinking behind how it worked before my recent changes,
but I completely accept that it wasn't helpful in the concurrent.futures
scenario, and agree with the comments you made around this in your earlier
posts. However, what Antoine has said on this thread (and others have concurred)
leads me to believe that he wants to use it not purely as a tracking tool (as
I've termed it, "an adjunct to doing real work") but also as a tool to provide
program output (e.g. doing some of the program's real work, say by calling
error() to print an error message).

Regards,

Vinay Sajip

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API for the new sysconfig module

2010-12-11 Thread Antoine Pitrou 
On Sat, 11 Dec 2010 12:55:25 +1000
Nick Coghlan  wrote:

> On Sat, Dec 11, 2010 at 12:44 PM, Terry Reedy  wrote:
> > On 12/10/2010 4:59 PM, R. David Murray wrote:
> >
> >> Like Éric, I'm not sure what the implications of the existing module
> >> having been released in 2.7 and 3.2 beta are in terms of making such an
> >> API change.
> >
> > I am with Raymond on this: the purpose of betas is so we can test *and* make
> > changes. No one should base production software on a beta release.
> 
> It's also the key difference between betas and release candidates.

Seems that both you and Terry have missed the part where the sysconfig
API is already part of 2.7, though.

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] futures API

2010-12-11 Thread Thomas Nagy 
--- El vie, 10/12/10, Brian Quinlan escribió:
> On Dec 10, 2010, at 10:51 AM, Thomas Nagy wrote:
> > --- El vie, 10/12/10, Brian Quinlan escribió:
> >> On Dec 10, 2010, at 5:36 AM, Thomas Nagy wrote:
> >>> I have a process running for a long time, and
> which
> >> may use futures of different max_workers count. I
> think it
> >> is not too far-fetched to create a new futures
> object each
> >> time. Yet, the execution becomes slower after each
> call, for
> >> example with http://freehackers.org/~tnagy/futures_test.py:
> >>>
> >>> """
> >>> import concurrent.futures
> >>> from queue import Queue
> >>> import datetime
> >>>
> >>> class counter(object):
> >>>      def __init__(self, fut):
> >>>          self.fut =
> fut
> >>>
> >>>      def run(self):
> >>>          def
> >> look_busy(num, obj):
> >>>
> >>    tot = 0
> >>>
> >>    for x in range(num):
> >>>
> >>    tot += x
> >>>
> >>    obj.out_q.put(tot)
> >>>
> >>>          start =
> >> datetime.datetime.utcnow()
> >>>          self.count =
> 0
> >>>          self.out_q
> =
> >> Queue(0)
> >>>          for x in
> >> range(1000):
> >>>
> >>    self.count += 1
> >>>
> >>    self.fut.submit(look_busy,
> self.count,
> >> self)
> >>>
> >>>          while
> >> self.count:
> >>>
> >>    self.count -= 1
> >>>
> >>    self.out_q.get()
> >>>
> >>>          delta =
> >> datetime.datetime.utcnow() - start
> >>>
> >>    print(delta.total_seconds())
> >>>
> >>> fut =
> >>
> concurrent.futures.ThreadPoolExecutor(max_workers=20)
> >>> for x in range(100):
> >>>      # comment the following
> line
> >>>      fut =
> >>
> concurrent.futures.ThreadPoolExecutor(max_workers=20)
> >>>      c = counter(fut)
> >>>      c.run()
> >>> """
> >>>
> >>> The runtime grows after each step:
> >>> 0.216451
> >>> 0.225186
> >>> 0.223725
> >>> 0.74
> >>> 0.230964
> >>> 0.240531
> >>> 0.24137
> >>> 0.252393
> >>> 0.249948
> >>> 0.257153
> >>> ...
> >>>
> >>> Is there a mistake in this piece of code?
> >>
> >> There is no mistake that I can see but I suspect
> that the
> >> circular references that you are building are
> causing the
> >> ThreadPoolExecutor to take a long time to be
> collected. Try
> >> adding:
> >>
> >>     c = counter(fut)
> >>     c.run()
> >> +    fut.shutdown()
> >>
> >> Even if that fixes your problem, I still don't
> fully
> >> understand this because I would expect the runtime
> to fall
> >> after a while as ThreadPoolExecutors are
> collected.
> >
> > The shutdown call is indeed a good fix :-) Here is the
> time response  
> > of the calls to counter() when shutdown is not
> called:
> > http://www.freehackers.org/~tnagy/runtime_futures.png
> 
> FWIW, I think that you are confusion the term "future"
> with  
> "executor". A future represents a single work item. An
> executor  
> creates futures and schedules their underlying work.

Ah yes, sorry. I have also realized that the executor is not the killer feature 
I was expecting, it can only replace a little part of the code I have: 
controlling the exceptions and the workflow is the most complicated part.

I have also observed a minor performance degradation with the executor 
replacement (3 seconds for 5000 work items). The amount of work items processed 
by unit of time does not seem to be a straight line: 
http://www.freehackers.org/~tnagy/runtime_futures_2.png . Out of curiosity, 
what is the "_thread_references" for?

The source file for the example is in:
http://www.freehackers.org/~tnagy/futures_test3.py

The diagram was created by:
http://www.freehackers.org/~tnagy/futures_test3.plot

Thomas



  
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] futures API

2010-12-11 Thread Scott Dial 
On 12/11/2010 9:44 AM, Thomas Nagy wrote:
> The amount of work items processed by unit of time does not seem to be a 
> straight line: http://www.freehackers.org/~tnagy/runtime_futures_2.png . Out 
> of curiosity, what is the "_thread_references" for?
> 
> The source file for the example is in:
> http://www.freehackers.org/~tnagy/futures_test3.py
> 
> The diagram was created by:
> http://www.freehackers.org/~tnagy/futures_test3.plot

You're test code does 50,000,000 of list appends. I suspect your
benchmark is telling you more about the behavior of large lists than the
overhead of the futures module. You should retry that experiment with
the list pre-allocated. Beyond that, the curve in that line is not
exactly a large amount of variance from a straight line.

-- 
Scott Dial
[email protected]
[email protected]
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using logging in the stdlib and its unit tests

2010-12-11 Thread Nick Coghlan 
On Sat, Dec 11, 2010 at 10:06 PM, Vinay Sajip  wrote:
> Nick Coghlan  gmail.com> writes:
>
>> the logging module is a tool to track what is happening in a program, not a
>> tool for providing a console based UI.
>
> That was certainly the thinking behind how it worked before my recent changes,
> but I completely accept that it wasn't helpful in the concurrent.futures
> scenario, and agree with the comments you made around this in your earlier
> posts. However, what Antoine has said on this thread (and others have 
> concurred)
> leads me to believe that he wants to use it not purely as a tracking tool (as
> I've termed it, "an adjunct to doing real work") but also as a tool to provide
> program output (e.g. doing some of the program's real work, say by calling
> error() to print an error message).

I think that's actually just a case where the lines get a little
blurry. From logging's point of view, it is still being used to say
"Hey, this particular event happened during program execution". We're
just adjusting our display policy to say that there are certain kinds
of events that should be made visible to the user by default, but
piping them into the logging machinery is preferable to writing them
directly to sys,stderr.

By contrast, most information that is written to sys.stdout isn't
really reporting events that happened in the program - it is actual
meaningful data that is related to the task the program accomplishes.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] porting python.org

2010-12-11 Thread Georg Brandl 
Am 11.12.2010 11:36, schrieb Prashant Kumar:
> I was wondering if we could contribute in porting of python.org
>  website over to python3k. If we can, what are the steps to
> be taken and other necessary requirements?

The SVN repository for python.org is currently not publicly available, but you
can get access asking on the [email protected] mailing list.

An excerpt from the README:

External packages required:

  Mako (http://www.makotemplates.org/)
  [Note that Mako requires setuptools to install out-of-the-box.  You can
   also just copy the lib/mako/ tree into your Python's site-packages
   directory if you prefer not to install setuptools.]

  PyYAML (http://pyyaml.org/wiki/PyYAML)

  Docutils (http://docutils.sf.net)

So the first step would be to determine whether Mako and PyYAML are ported
yet, and if not, help porting them.  Docutils is ported already.

cheers,
Georg

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API for the new sysconfig module

2010-12-11 Thread Nick Coghlan 
On Sun, Dec 12, 2010 at 12:17 AM, Antoine Pitrou  wrote:
> On Sat, 11 Dec 2010 12:55:25 +1000
> Nick Coghlan  wrote:
>
>> On Sat, Dec 11, 2010 at 12:44 PM, Terry Reedy  wrote:
>> > On 12/10/2010 4:59 PM, R. David Murray wrote:
>> >
>> >> Like Éric, I'm not sure what the implications of the existing module
>> >> having been released in 2.7 and 3.2 beta are in terms of making such an
>> >> API change.
>> >
>> > I am with Raymond on this: the purpose of betas is so we can test *and* 
>> > make
>> > changes. No one should base production software on a beta release.
>>
>> It's also the key difference between betas and release candidates.
>
> Seems that both you and Terry have missed the part where the sysconfig
> API is already part of 2.7, though.

I had missed that, yes*. I suspect Raymond may have missed it as well.

Cheers,
Nick.

*(I sometimes lose track of which changes were made in both branches
pre-2.7, which ones were mode post-2.7 release, and which ones went in
pre-2.7, but were 3.x only regardless)

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using logging in the stdlib and its unit tests

2010-12-11 Thread Tres Seaver 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 12/11/2010 04:28 AM, Vinay Sajip wrote:

> Actually if we're to change things to print INFO to stdout and WARNING+ to
> stderr,

ACK!  That would be an awful default -- stdout belongs to the
application, not to "meta" stuff like logging, which should only go to
stderr.  Much better to document the "how-to" for setting up something
like that than to make it a default.



Tres.
- -- 
===
Tres Seaver  +1 540-429-0999  [email protected]
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk0Ds8YACgkQ+gerLs4ltQ6BBACfTWWcfl6000DX9SWCYvd5Tf7n
qIcAniTD6EHRhrXA+eVpgyrlqvv12apq
=oR03
-END PGP SIGNATURE-

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] porting python.org

2010-12-11 Thread Sandro Tosi 
Hi,

On Sat, Dec 11, 2010 at 18:19, Georg Brandl  wrote:
>  Mako (http://www.makotemplates.org/)
>  [Note that Mako requires setuptools to install out-of-the-box.  You can
>   also just copy the lib/mako/ tree into your Python's site-packages
>   directory if you prefer not to install setuptools.]

Mako is compatible with py3k (via 2to3) since version 0.3 (but later
versions has a better support). [1] explains how to have a py3k
installation of mako.

[1] http://www.makotemplates.org/trac/browser/README.py3k

>  PyYAML (http://pyyaml.org/wiki/PyYAML)

"PyYAML also supports Python 3" from the homepage, the version
introducing py3k support is 3.08 .

Just my 2c,
-- 
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using logging in the stdlib and its unit tests

2010-12-11 Thread Nick Coghlan 
On Sun, Dec 12, 2010 at 3:24 AM, Tres Seaver  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> On 12/11/2010 04:28 AM, Vinay Sajip wrote:
>
>> Actually if we're to change things to print INFO to stdout and WARNING+ to
>> stderr,
>
> ACK!  That would be an awful default -- stdout belongs to the
> application, not to "meta" stuff like logging, which should only go to
> stderr.  Much better to document the "how-to" for setting up something
> like that than to make it a default.

Don't worry, that isn't going to happen. Vinay was just responding to
something he thought I was suggesting. I wasn't really, but I can
understand someone getting that impression (as I was using that
behaviour as my running example of "so just what *is* involved if an
applications wants to configure logging like that?").

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using logging in the stdlib and its unit tests

2010-12-11 Thread Glenn Linderman 

On 12/11/2010 3:52 AM, Vinay Sajip wrote:

I will try to incorporate more basic examples at the top of the documentation,
but surely you don't want me to add more verbiage about basicConfig when your
overall feeling is that there's too much documentation?


I try not to post unless I feel there is a new detail; yes the 8% is 
probably repetitive, but at present, going further, at least 
sequentially, does feel like a large mountain to climb... suddenly 
becoming significantly steeper there.


I really can't say much about the 92% of the documentation,  because of 
the mountain.


Whether the existing documentation can be reordered to avoid the steep 
part, but rather provide a gentle upward path, I cannot say.  If it is 
properly organized reference material, then probably adding more about 
something at the beginning is appropriate, to give the foundation, 
justification, and assistance for climbing the slope.


My overall feeling is not necessarily that there is too much 
documentation (there is a lot, but maybe that is a good thing, I should 
read it some week) but that it suffers in not providing useful 
information for the beginner at the front.  I've listed what I've 
gleaned here and tried to point out the things that I still don't know, 
and tried to point out where things became less than obvious along the 
way.  I do think it was primarily because the intro material is too 
brief and too disconnected, that I quit reading at the 8% mark (or so).  
If the rest of the large document were to be similarly disconnected, it 
wouldn't be worth my time to read it... and if I had to read it all, 
just to understand enough to get started with simple logging, then it 
would be faster to write my own, which is the path I took.  I suffer 
with my own being primitive, but I can tweak it when I need to.  I still 
don't fully understand the nuances between handlers and filters, and 
likely won't until I read more of the documentation.


Jumping directly to the documentation (at the 28% mark, I'd have had to 
read a long way to find it) for basicConfig per your links helps, but 
sadly,  because the first paragraph is full of the terms "logger", 
"handler", and "formatter", which I have only rudimentary ideas about 
what they do, being a beginner, I almost gave up again.  But the 
description of the valid keywords for **kwargs was actually 
understandable, at the end of that documentation.  I also learned that 
calling basicConfig more than once in a program is probably useless, and 
it should be nearly the first logger API called, if it is used.  The 
first paragraph should be split, perhaps: "Does basic configuration for 
the logging system." is almost redundant with the name, but is OK if 
first.  The rest of that paragraph, and the next three, should probably 
be moved after the list of kwargs keys and descriptions.  Yes, the word 
handler is used a bit, but it is associated there with concepts like 
file names and stream handles, and that eases the uncertainty; together 
with the simple example at the front, and the verbage there, I can see 
that I specify a file name, and it makes the logger log to that file :)  
And the level parameter is fine.  I'm less sure about format and datefmt 
and why I would want to specify them or how, and am surprised to find 
that format is to be used by a "handler" not a "formatter", but the 
simple example shows me I can leave those out for starters.


Personally, I suffer in writing documentation; when I attempt to do so, 
and people read it, they usually tell me that after about the third 
read, they finally understand it, but sadly it takes them three reads.  
It is interesting to be looking at the logger from the other side... 
maybe I'll be able to improve the documentation I write, after analyzing 
why the logger documentation was hard for me to approach.



___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Format factories (was Re: sWAPcASE Was: transform() and untransform() methods, and the codec registry)

2010-12-11 Thread Brett Cannon 
On Thu, Dec 9, 2010 at 16:26, Nick Coghlan  wrote:
> On Fri, Dec 10, 2010 at 9:29 AM, Antoine Pitrou  wrote:
>> On Thu, 09 Dec 2010 18:10:38 -0500
>> Eric Smith  wrote:
>>> If we're looking to reduce the number of methods on str, I wouldn't mind
>>> seeing center() and zfill() also go away, since they're trivially
>>> replaced by format().
>>
>> Well, it's only trivial when you know format()'s wicked mini-language by
>> heart :/  center() is easy and obvious. zfill() is arguably a bit too
>> specialized.
>
> I've occasionally wondered if string formatting [1] and the struct
> module [2] would benefit from format building functions that made them
> easier to use without necessarily learning the cryptic mini-languages
> off by heart.
>
> For example, a "string.make_format_spec" function might have a signature like:
>
> def make_format_spec(fill=None, align=None, sign=None, width=None,
> precision=None, display_type='s', alternate=False, commas=False,
> numeric_case=None)
>
> "align" would accept not only the actual format symbols ('<', '>',
> '=', '^'), but also the corresponding names ('left', 'right',
> 'numeric', 'center').
>
> "numeric_case" would accept None, 'upper' or 'lower' (affecting the
> formatting of hex and floating point values). If the stated numeric
> case differs from that specified by the display type, raise a
> ValueError. For an unspecified numeric case, the affected display
> types would default to 'lower'.
>
> Similarly, "display_type" would accept long names in addition to the
> character codes:
>
> 's': 'str'
> 'b': 'binary'
> 'c': 'chr'
> 'd': 'int'
> 'o': 'octal'
> 'x', 'X': 'hex' (numeric case controls display of digits A-F)
> 'n': 'locale'
> 'e', 'E': 'exponent' (numeric case controls display of exponent as
> well as infinite and NaN values)
> 'f', 'F': 'float' (numeric case controls display of exponent as well
> as infinite and NaN values)
> 'g', 'G': 'general' (numeric case controls display of exponent as well
> as infinite and NaN values)
> '%': 'percent' (numeric case controls display of exponent as well as
> infinite and NaN values)
>
> There could also be a corresponding parse_format_spec that produced a
> named tuple with the appropriate details.

But is this worth it since once you write it you won't be changing it
again, suggesting that taking the time to look up the formatting rules
isn't that much harder and wouldn't burden us w/ writing such
functions and trying to come up with a good API. I suspect what would
be more helpful is either have a rather detailed docstring for
str.format or to at least put the URL to the docs which explains the
mini-language to make it easier to find.

-Brett

>
> Cheers,
> Nick.
>
> [1] 
> http://docs.python.org/dev/library/string#format-specification-mini-language
> [2] http://docs.python.org/dev/library/struct#format-strings
>
> --
> Nick Coghlan   |   [email protected]   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using logging in the stdlib and its unit tests

2010-12-11 Thread Brett Cannon 
On Fri, Dec 10, 2010 at 22:21, Vinay Sajip  wrote:
> Nick Coghlan  gmail.com> writes:
>
>
>> This could actually make a reasonably good basic for a "task oriented"
>> subsection of the logging documentation. Something like:
>>
>
> Good suggestion, I'll see what I can do.

Just wanted to +1 on some task-oriented (or at least simple intro)
docs going into the logging module.

The length of the docs is rather daunting I always have a hard time
getting started (the first example is for file logging and the second
one does log rotation when all I want is printing to stderr!). All I
want is an opener saying, "here are the functions to call to send
stuff to the log; the defaults for each go here and here (e.g.,
stderr, /dev/null), and if you want to change those settings, here is
how to e.g., get info() to go to stderr. Read on for more if you need
something fancier (e.g., writing to a file)". That and making it not
an error when I call logging.error() w/o setting an explicit logger
would go a long way to making the logging module much more accessible
IMO.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Format factories (was Re: sWAPcASE Was: transform() and untransform() methods, and the codec registry)

2010-12-11 Thread Jeffrey Yasskin 
On Sat, Dec 11, 2010 at 11:21 AM, Brett Cannon  wrote:
> On Thu, Dec 9, 2010 at 16:26, Nick Coghlan  wrote:
>> On Fri, Dec 10, 2010 at 9:29 AM, Antoine Pitrou  wrote:
>>> On Thu, 09 Dec 2010 18:10:38 -0500
>>> Eric Smith  wrote:
 If we're looking to reduce the number of methods on str, I wouldn't mind
 seeing center() and zfill() also go away, since they're trivially
 replaced by format().
>>>
>>> Well, it's only trivial when you know format()'s wicked mini-language by
>>> heart :/  center() is easy and obvious. zfill() is arguably a bit too
>>> specialized.
>>
>> I've occasionally wondered if string formatting [1] and the struct
>> module [2] would benefit from format building functions that made them
>> easier to use without necessarily learning the cryptic mini-languages
>> off by heart.
>>
>> For example, a "string.make_format_spec" function might have a signature 
>> like:
>>
>> def make_format_spec(fill=None, align=None, sign=None, width=None,
>> precision=None, display_type='s', alternate=False, commas=False,
>> numeric_case=None)
>>
>> "align" would accept not only the actual format symbols ('<', '>',
>> '=', '^'), but also the corresponding names ('left', 'right',
>> 'numeric', 'center').
>>
>> "numeric_case" would accept None, 'upper' or 'lower' (affecting the
>> formatting of hex and floating point values). If the stated numeric
>> case differs from that specified by the display type, raise a
>> ValueError. For an unspecified numeric case, the affected display
>> types would default to 'lower'.
>>
>> Similarly, "display_type" would accept long names in addition to the
>> character codes:
>>
>> 's': 'str'
>> 'b': 'binary'
>> 'c': 'chr'
>> 'd': 'int'
>> 'o': 'octal'
>> 'x', 'X': 'hex' (numeric case controls display of digits A-F)
>> 'n': 'locale'
>> 'e', 'E': 'exponent' (numeric case controls display of exponent as
>> well as infinite and NaN values)
>> 'f', 'F': 'float' (numeric case controls display of exponent as well
>> as infinite and NaN values)
>> 'g', 'G': 'general' (numeric case controls display of exponent as well
>> as infinite and NaN values)
>> '%': 'percent' (numeric case controls display of exponent as well as
>> infinite and NaN values)
>>
>> There could also be a corresponding parse_format_spec that produced a
>> named tuple with the appropriate details.
>
> But is this worth it since once you write it you won't be changing it
> again,

Having a make_format_spec() would also help people trying to read the code.

> suggesting that taking the time to look up the formatting rules
> isn't that much harder and wouldn't burden us w/ writing such
> functions and trying to come up with a good API. I suspect what would
> be more helpful is either have a rather detailed docstring for
> str.format or to at least put the URL to the docs which explains the
> mini-language to make it easier to find.
>>
>> Cheers,
>> Nick.
>>
>> [1] 
>> http://docs.python.org/dev/library/string#format-specification-mini-language
>> [2] http://docs.python.org/dev/library/struct#format-strings
>>
>> --
>> Nick Coghlan   |   [email protected]   |   Brisbane, Australia
>> ___
>> Python-Dev mailing list
>> [email protected]
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: 
>> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>>
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/jyasskin%40gmail.com
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] futures API

2010-12-11 Thread Brian Quinlan 


On Dec 11, 2010, at 6:44 AM, Thomas Nagy wrote:


--- El vie, 10/12/10, Brian Quinlan escribió:

On Dec 10, 2010, at 10:51 AM, Thomas Nagy wrote:

--- El vie, 10/12/10, Brian Quinlan escribió:

On Dec 10, 2010, at 5:36 AM, Thomas Nagy wrote:

I have a process running for a long time, and

which

may use futures of different max_workers count. I

think it

is not too far-fetched to create a new futures

object each

time. Yet, the execution becomes slower after each

call, for

example with http://freehackers.org/~tnagy/futures_test.py:


"""
import concurrent.futures
from queue import Queue
import datetime

class counter(object):
  def __init__(self, fut):
  self.fut =

fut


  def run(self):
  def

look_busy(num, obj):



tot = 0



for x in range(num):



tot += x



obj.out_q.put(tot)


  start =

datetime.datetime.utcnow()

  self.count =

0

  self.out_q

=

Queue(0)

  for x in

range(1000):



self.count += 1



self.fut.submit(look_busy,

self.count,

self)


  while

self.count:



self.count -= 1



self.out_q.get()


  delta =

datetime.datetime.utcnow() - start



print(delta.total_seconds())


fut =



concurrent.futures.ThreadPoolExecutor(max_workers=20)

for x in range(100):
  # comment the following

line

  fut =



concurrent.futures.ThreadPoolExecutor(max_workers=20)

  c = counter(fut)
  c.run()
"""

The runtime grows after each step:
0.216451
0.225186
0.223725
0.74
0.230964
0.240531
0.24137
0.252393
0.249948
0.257153
...

Is there a mistake in this piece of code?


There is no mistake that I can see but I suspect

that the

circular references that you are building are

causing the

ThreadPoolExecutor to take a long time to be

collected. Try

adding:

 c = counter(fut)
 c.run()
+fut.shutdown()

Even if that fixes your problem, I still don't

fully

understand this because I would expect the runtime

to fall

after a while as ThreadPoolExecutors are

collected.


The shutdown call is indeed a good fix :-) Here is the

time response

of the calls to counter() when shutdown is not

called:

http://www.freehackers.org/~tnagy/runtime_futures.png


FWIW, I think that you are confusion the term "future"
with
"executor". A future represents a single work item. An
executor
creates futures and schedules their underlying work.


Ah yes, sorry. I have also realized that the executor is not the  
killer feature I was expecting, it can only replace a little part of  
the code I have: controlling the exceptions and the workflow is the  
most complicated part.


I have also observed a minor performance degradation with the  
executor replacement (3 seconds for 5000 work items). The amount of  
work items processed by unit of time does not seem to be a straight  
line: http://www.freehackers.org/~tnagy/runtime_futures_2.png .


That looks pretty linear to me.


Out of curiosity, what is the "_thread_references" for?


There is a big comment above it in the code:

# Workers are created as daemon threads. This is done to allow the  
interpreter
# to exit when there are still idle threads in a ThreadPoolExecutor's  
thread
# pool (i.e. shutdown() was not called). However, allowing workers to  
die with

# the interpreter has two undesirable properties:
#   - The workers would still be running during interpretor shutdown,
# meaning that they would fail in unpredictable ways.
#   - The workers could be killed while evaluating a work item, which  
could
# be bad if the callable being evaluated has external side-effects  
e.g.

# writing to a file.
#
# To work around this problem, an exit handler is installed which  
tells the
# workers to exit when their work queues are empty and then waits  
until the

# threads finish.

_thread_references = set()
_shutdown = False

def _python_exit():
global _shutdown
_shutdown = True
for thread_reference in _thread_references:
thread = thread_reference()
if thread is not None:
thread.join()

Is it still unclear why it is there? Maybe you could propose some  
additional documentation.


Cheers,
Brian


The source file for the example is in:
http://www.freehackers.org/~tnagy/futures_test3.py

The diagram was created by:
http://www.freehackers.org/~tnagy/futures_test3.plot

Thomas




___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of PEP 3143?

2010-12-11 Thread Ben Finney 
Benjamin Peterson  writes:

> 2010/12/8 Ben Finney :
> > So it's not abandoned, but I don't know which version should be the
> > current target. What change should I make to the PEP in such a case?
>
> Put 3.3 or 3.x if you're thinking really long term. :)

Done now (http://svn.python.org/projects/peps/trunk/>, revision
87170).

-- 
 \   “We must respect the other fellow's religion, but only in the |
  `\   sense and to the extent that we respect his theory that his |
_o__) wife is beautiful and his children smart.” —Henry L. Mencken |
Ben Finney

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API for the new sysconfig module

2010-12-11 Thread Raymond Hettinger 

On Dec 11, 2010, at 9:21 AM, Nick Coghlan wrote:

> On Sun, Dec 12, 2010 at 12:17 AM, Antoine Pitrou  wrote:
>> On Sat, 11 Dec 2010 12:55:25 +1000
>> Nick Coghlan  wrote:
>> 
>>> On Sat, Dec 11, 2010 at 12:44 PM, Terry Reedy  wrote:
 On 12/10/2010 4:59 PM, R. David Murray wrote:
 
> Like Éric, I'm not sure what the implications of the existing module
> having been released in 2.7 and 3.2 beta are in terms of making such an
> API change.
 
 I am with Raymond on this: the purpose of betas is so we can test *and* 
 make
 changes. No one should base production software on a beta release.
>>> 
>>> It's also the key difference between betas and release candidates.
>> 
>> Seems that both you and Terry have missed the part where the sysconfig
>> API is already part of 2.7, though.
> 
> I had missed that, yes*. I suspect Raymond may have missed it as well.
> 
> Cheers,
> Nick.
> 
> *(I sometimes lose track of which changes were made in both branches
> pre-2.7, which ones were mode post-2.7 release, and which ones went in
> pre-2.7, but were 3.x only regardless)

Right.  I missed that it was already in 2.7.
So, now we're stuck with it, forever.


Raymond

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] porting python.org

2010-12-11 Thread Terry Reedy 

On 12/11/2010 12:19 PM, Georg Brandl wrote:

Am 11.12.2010 11:36, schrieb Prashant Kumar:

I was wondering if we could contribute in porting of python.org
  website over to python3k.


I think this is an excellent idea. It will test Python3 and the modules 
and external packages used and once successful, advertise that all such 
are production-ready Py3 software. If and when there is a test version I 
can access (py3.python.org?), I will happily help test by browsing 
around and downloading stuff.


--
Terry Jan Reedy

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of PEP 3143?

2010-12-11 Thread Ben Finney 
Ben Finney  writes:

> Done now (http://svn.python.org/projects/peps/trunk/>, revision
> 87170).

Thanks to Georg Brandl for applying the change on my behalf.

-- 
 \   “As the most participatory form of mass speech yet developed, |
  `\the Internet deserves the highest protection from governmental |
_o__)   intrusion.” —U.S. District Court Judge Dalzell |
Ben Finney

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] porting python.org

2010-12-11 Thread Georg Brandl 
Am 11.12.2010 22:28, schrieb Terry Reedy:
> On 12/11/2010 12:19 PM, Georg Brandl wrote:
>> Am 11.12.2010 11:36, schrieb Prashant Kumar:
>>> I was wondering if we could contribute in porting of python.org
>>>   website over to python3k.
> 
> I think this is an excellent idea. It will test Python3 and the modules 
> and external packages used and once successful, advertise that all such 
> are production-ready Py3 software. If and when there is a test version I 
> can access (py3.python.org?), I will happily help test by browsing 
> around and downloading stuff.

I don't think a separate test site is useful.  Either the build process
works with Python 3 or it doesn't -- in the latter case you'll just get
an exception while running "make".

Georg

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] futures API

2010-12-11 Thread Steven D'Aprano 

Brian Quinlan wrote:


On Dec 11, 2010, at 6:44 AM, Thomas Nagy wrote:


I have also observed a minor performance degradation with the executor 
replacement (3 seconds for 5000 work items). The amount of work items 
processed by unit of time does not seem to be a straight line: 
http://www.freehackers.org/~tnagy/runtime_futures_2.png .


That looks pretty linear to me.


Close to, but not quite. The graph seems to be slightly curved, with the 
amount of work done per second decreasing for large amounts of work. 
Assuming that this performance degradation is real, and not an artifact 
of the measurement technique, it seems to be quite small. I'd be happy 
to describe it as "linear" in the same way we describe dictionary 
lookups as constant-time, even though technically that's not strictly 
true. (They're linear in the number of items with a matching hash, and 
there are probably other complications as well.)


As drawn, the curve seems to fall away like a log graph, which might 
suggest to the casual viewer that this is a good thing. It may be better 
to reverse the axes, that is to have the independent variable, number of 
tasks, on the horizontal axis, and the dependent variable, cost (time 
taken), vertically. This will make it clear that the incremental cost of 
doing one extra task increases (slightly) as the number of tasks goes up.




--
Steven

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Format factories (was Re: sWAPcASE Was: transform() and untransform() methods, and the codec registry)

2010-12-11 Thread Nick Coghlan 
On Sun, Dec 12, 2010 at 5:21 AM, Brett Cannon  wrote:
> But is this worth it since once you write it you won't be changing it
> again, suggesting that taking the time to look up the formatting rules
> isn't that much harder and wouldn't burden us w/ writing such
> functions and trying to come up with a good API. I suspect what would
> be more helpful is either have a rather detailed docstring for
> str.format or to at least put the URL to the docs which explains the
> mini-language to make it easier to find.

Yes, it may be worth it, since it greatly simplifies things like
generating a format spec programmatically rather than hardcoding it.
You can also run it once at the interactive prompt to figure out the
format string you want to hard code, then include the API call as a
comment for concise documentation of what your format string does. I
helped *define* the new format spec and I still need to reference the
docs to create trickier ones. A factory function would be far more
convenient than burdening str.format's docstring with a full
definition of the format spec syntax.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] futures API

2010-12-11 Thread Nick Coghlan 
On Sun, Dec 12, 2010 at 6:53 AM, Brian Quinlan  wrote:
> Is it still unclear why it is there? Maybe you could propose some additional
> documentation.

Did you get my question the other day as to whether a
weakref.WeakKeySet might be a better choice? I believe you would be
able to get rid of the periodic sweep for dead references if you did
that, and I didn't spot any obvious downsides.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] futures API

2010-12-11 Thread Thomas Nagy 
--- El sáb, 11/12/10, Brian Quinlan escribió:
> 
> On Dec 11, 2010, at 6:44 AM, Thomas Nagy wrote:
> 
> > --- El vie, 10/12/10, Brian Quinlan escribió:
> >> On Dec 10, 2010, at 10:51 AM, Thomas Nagy wrote:
> >>> --- El vie, 10/12/10, Brian Quinlan
> escribió:
>  On Dec 10, 2010, at 5:36 AM, Thomas Nagy
> wrote:
> > I have a process running for a long
> time, and
> >> which
>  may use futures of different max_workers
> count. I
> >> think it
>  is not too far-fetched to create a new
> futures
> >> object each
>  time. Yet, the execution becomes slower
> after each
> >> call, for
>  example with http://freehackers.org/~tnagy/futures_test.py:
> > 
> > """
> > import concurrent.futures
> > from queue import Queue
> > import datetime
> > 
> > class counter(object):
> >       def
> __init__(self, fut):
> >       
>    self.fut =
> >> fut
> > 
> >       def
> run(self):
> >       
>    def
>  look_busy(num, obj):
> > 
>      tot = 0
> > 
>      for x in
> range(num):
> > 
>      tot += x
> > 
>  
>    obj.out_q.put(tot)
> > 
> >       
>    start =
>  datetime.datetime.utcnow()
> >       
>    self.count =
> >> 0
> >       
>    self.out_q
> >> =
>  Queue(0)
> >       
>    for x in
>  range(1000):
> > 
>      self.count += 1
> > 
>  
>    self.fut.submit(look_busy,
> >> self.count,
>  self)
> > 
> >       
>    while
>  self.count:
> > 
>      self.count -= 1
> > 
>      self.out_q.get()
> > 
> >       
>    delta =
>  datetime.datetime.utcnow() - start
> > 
>  
>    print(delta.total_seconds())
> > 
> > fut =
>  
> >>
> concurrent.futures.ThreadPoolExecutor(max_workers=20)
> > for x in range(100):
> >       #
> comment the following
> >> line
> >       fut =
>  
> >>
> concurrent.futures.ThreadPoolExecutor(max_workers=20)
> >       c =
> counter(fut)
> >   
>    c.run()
> > """
> > 
> > The runtime grows after each step:
> > 0.216451
> > 0.225186
> > 0.223725
> > 0.74
> > 0.230964
> > 0.240531
> > 0.24137
> > 0.252393
> > 0.249948
> > 0.257153
> > ...
> > 
> > Is there a mistake in this piece of
> code?
>  
>  There is no mistake that I can see but I
> suspect
> >> that the
>  circular references that you are building
> are
> >> causing the
>  ThreadPoolExecutor to take a long time to
> be
> >> collected. Try
>  adding:
>  
>       c = counter(fut)
>       c.run()
>  +    fut.shutdown()
>  
>  Even if that fixes your problem, I still
> don't
> >> fully
>  understand this because I would expect the
> runtime
> >> to fall
>  after a while as ThreadPoolExecutors are
> >> collected.
> >>> 
> >>> The shutdown call is indeed a good fix :-)
> Here is the
> >> time response
> >>> of the calls to counter() when shutdown is
> not
> >> called:
> >>> http://www.freehackers.org/~tnagy/runtime_futures.png
> >> 
> >> FWIW, I think that you are confusion the term
> "future"
> >> with
> >> "executor". A future represents a single work
> item. An
> >> executor
> >> creates futures and schedules their underlying
> work.
> > 
> > Ah yes, sorry. I have also realized that the executor
> is not the killer feature I was expecting, it can only
> replace a little part of the code I have: controlling the
> exceptions and the workflow is the most complicated part.
> > 
> > I have also observed a minor performance degradation
> with the executor replacement (3 seconds for 5000 work
> items). The amount of work items processed by unit of time
> does not seem to be a straight line: 
> http://www.freehackers.org/~tnagy/runtime_futures_2.png
> .
> 
> That looks pretty linear to me.

Ok.
 
> > Out of curiosity, what is the "_thread_references"
> for?
> 
> There is a big comment above it in the code:
> 
> # Workers are created as daemon threads. This is done to
> allow the interpreter
> # to exit when there are still idle threads in a
> ThreadPoolExecutor's thread
> # pool (i.e. shutdown() was not called). However, allowing
> workers to die with
> # the interpreter has two undesirable properties:
> #   - The workers would still be running
> during interpretor shutdown,
> #     meaning that they would fail in
> unpredictable ways.
> #   - The workers could be killed while
> evaluating a work item, which could
> #     be bad if the callable being
> evaluated has external side-effects e.g.
> #     writing to a file.
> #
> # To work around this problem, an exit handler is installed
> which tells the
> # workers to exit when their work queues are empty and then
> waits until the
> # threads finish.
> 
> _thread_references = set()
> _shutdown = False
> 
> def _python_exit():
>     global _shutdown
>     _shutdown = True
>     for thread_reference in _thread_references:
>         thread = thread

Re: [Python-Dev] futures API

2010-12-11 Thread Brian Quinlan 


On Dec 11, 2010, at 6:33 PM, Nick Coghlan wrote:

On Sun, Dec 12, 2010 at 6:53 AM, Brian Quinlan   
wrote:
Is it still unclear why it is there? Maybe you could propose some  
additional

documentation.


Did you get my question the other day as to whether a
weakref.WeakKeySet might be a better choice? I believe you would be
able to get rid of the periodic sweep for dead references if you did
that, and I didn't spot any obvious downsides.


No I didn't, sorry! Could you resent it if it has more details then  
the paragraph above?


Cheers,
Brian


Cheers,
Nick.

--
Nick Coghlan   |   [email protected]   |   Brisbane, Australia


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] futures API

2010-12-11 Thread Nick Coghlan 
On Sun, Dec 12, 2010 at 12:36 PM, Brian Quinlan  wrote:
>
> On Dec 11, 2010, at 6:33 PM, Nick Coghlan wrote:
>
>> On Sun, Dec 12, 2010 at 6:53 AM, Brian Quinlan  wrote:
>>>
>>> Is it still unclear why it is there? Maybe you could propose some
>>> additional
>>> documentation.
>>
>> Did you get my question the other day as to whether a
>> weakref.WeakKeySet might be a better choice? I believe you would be
>> able to get rid of the periodic sweep for dead references if you did
>> that, and I didn't spot any obvious downsides.
>
> No I didn't, sorry! Could you resent it if it has more details then the
> paragraph above?

There wasn't much more detail, as there actually isn't a great deal to
the idea. This was the main comment in the previous email:

"It seems to me that using WeakSet would mean you could get rid of
_remove_dead_thread_references altogether (the implicit callbacks
would handle that part), and then a set() call in _python_exit would
give you concrete references to work with for cleanup purposes."

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com