Re: Python Programming Challenges for beginners?

2009-11-27 Thread Wolodja Wentland
On Thu, Nov 26, 2009 at 19:24 -0800, astral orange wrote:
> Hi-
> 
> I am reading the online tutorial along with a book I bought on Python.
> I would like to test out what I know so far by solving programming
> challenges. Similar to what O'Reilly Learning Perl has. I really
> enjoyed the challenges at the end of the chapter and it really help me
> test out if I was truly taking in the material like I should. Could I
> get some suggestions on resources? Is there anywhere where I can go
> online (for free or purchase) for programming problems? I am familiar
> with sites like Code Chef...etc...but at this stage that is not the
> right 'venue' for me. I mainly need challenges like the ones they have
> in Learning Perl.

Project Euler has already been pointed out to you, but I enjoyed solving
the riddles at:

http://www.pythonchallenge.com/

Enjoy! :-)
-- 
  .''`. Wolodja Wentland 
 : :'  :
 `. `'` 4096R/CAF14EFC 
   `-   081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Moving from Python 2 to Python 3: A 4 page "cheat sheet"

2009-12-02 Thread Wolodja Wentland
On Wed, Dec 02, 2009 at 00:10 -0800, Mark Summerfield wrote:
> On 1 Dec, 18:30, Lie Ryan  wrote:

> > Also, I'm not sure what this change is referring to:
> > Python 2                 Python 3
> > L = list(seq)            L = sorted(seq)
> > L.sort()
> >
> > L.sort is still available in python, and sorted() have been available
> > since python 2. Both list.sort() and sorted() are for different purpose,
> > and neither will be deprecated. What's the change here?
> 
> The document is about idioms as well as changes. In this case both
> approaches work in both versions, but it seems that there are still a
> lot of people who don't know about sorted(), so I put it in to show it
> as an idiom.

It would be quite nice if you could mark all the Python 3 idioms that
work in Python 2.X as well. This would allow readers that are still using
Python 2.X and are used to the 'old way' to adapt their coding style
accordingly. You could just add a little (2.X) after the idiom for
example.

And thanks for the nice cheat sheet! :-D

-- 
  .''`. Wolodja Wentland 
 : :'  :
 `. `'` 4096R/CAF14EFC 
   `-   081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Moving from Python 2 to Python 3: A 4 page "cheat sheet"

2009-12-03 Thread Wolodja Wentland
On Wed, Dec 02, 2009 at 08:03 -0800, Mark Summerfield wrote:
> On Dec 2, 11:20 am, Wolodja Wentland 

> > It would be quite nice if you could mark all the Python 3 idioms that
> > work in Python 2.X as well. This would allow readers that are still using
> > Python 2.X and are used to the 'old way' to adapt their coding style
> > accordingly. You could just add a little (2.X) after the idiom for
> > example.

> Yes it would be nice, but it isn't quite so simple. 

> To take sorted() as just one example, it was introduced in 2.4 so
> arguably using it isn't valid/idiomatic for Python 2.x programs where
> you care about backwards compatibility for the Python 2.x series... 

Yes, which is why you could include a 2.X and people who target, say
current +/- 0.1 can choose their poison.

> But my main reason for not wanting to do this is that the document is
> aimed at people who want to write Python 3, not to encourage people to
> stick with 2:-)

I actually think that it is the other way round. People should get
familiar with py3 features even if they are not yet ready to
"abandon" py2 (yet). I also think that using some of the backported/supported
features might spur interest in features that are 'py3 only' and therefore
encourage the adoption of py3.

It would also be nice to have a summary of things people can do *now* if
they want to keep the changes from 2to3 to a minimum, which will be with
us for some time. But that is not something *you* have to write .. :-)
-- 
  .''`. Wolodja Wentland 
 : :'  :
 `. `'` 4096R/CAF14EFC 
   `-   081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Why the expression "(1)" is not an one-arity tuple, but int ?

2009-12-04 Thread Wolodja Wentland
On Fri, Dec 04, 2009 at 15:17 +0300, Петров Александр wrote:
> In my code I try to use a generic approach to work with tuples. Let
> "X" be a tuple.
> When I want to access a first element of a tuple, I can write: "X[0]".
> And that is really working when X is a n-arity tuple, with n>1 (for
> example "foo( (1,2,3) )" ).
> But when I call my library function with a 1-arity tuple (for example
> "foo( (1) )" ) I have an error:
> 
> TypeError: 'int' object is unsubscriptable
> 
> How could I tell Python that "(1)" is not an integer, but an one-arity tuple ?

The following might clarify the issue:

>>> t = (1)
>>> type(t)

>>> t = (1,)
>>> type(t)

>>> t = 1,
>>> type(t)


It is the ',' not the '(' and ')' ...


-- 
  .''`. Wolodja Wentland 
 : :'  :
 `. `'` 4096R/CAF14EFC 
   `-   081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: [distutils] Install script under a different name

2009-12-05 Thread Wolodja Wentland
On Fri, Dec 04, 2009 at 19:34 -0500, Nikolaus Rath wrote:
> All my Python files have extension .py. However, I would like to install
> scripts that are meant to be called by the user without the suffix, i.e.
> the file scripts/doit.py should end up as /usr/bin/doit.

> Apparently the scripts= option of the setup() function does not support
> this directly. Is there a clever way to get what I want?

Just name the file you want to install 'doit' and not 'doit.py'. That
would be the easiest way. You might run into problems on Windows though,
which IIRC (it's been a while) heavily relies on file suffixes and not
on their content. That might have changed in the last 10 years though,
so better check that first.

You can also use entry points to create the executable at install time.
Have a look at [1] which explains how this is done. This requires using
Distribute/setuptools though, ...

[1] 
http://packages.python.org/distribute/setuptools.html#automatic-script-creation
-- 
  .''`. Wolodja Wentland 
 : :'  :
 `. `'` 4096R/CAF14EFC 
   `-   081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Which graph library is best suited for large graphs?

2009-12-11 Thread Wolodja Wentland
Hi all,

I am writing a library for accessing Wikipedia data and include a module
that generates graphs from the Link structure between articles and other
pages (like categories).

These graphs could easily contain some million nodes which are frequently
linked. The graphs I am building right now have around 300.000 nodes
with an average in/out degree of - say - 4 and already need around 1-2GB of
memory. I use networkx to model the graphs and serialise them to files on
the disk. (using adjacency list format, pickle and/or graphml).

The recent thread on including a graph library in the stdlib spurred my
interest and introduced me to a number of libraries I have not seen
before. I would like to reevaluate my choice of networkx and need some
help in doing so.

I really like the API of networkx but have no problem in switching to
another one (right now)  I have the impression that graph-tool might
be faster and have a smaller memory footprint than networkx, but am
unsure about that.

Which library would you choose? This decision is quite important for me
as the choice will influence my libraries external interface. Or is
there something like WSGI for graph libraries?

kind regards

-- 
  .''`.     Wolodja Wentland 
 : :'  :
 `. `'` 4096R/CAF14EFC 
   `-   081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Which graph library is best suited for large graphs?

2009-12-11 Thread Wolodja Wentland
On Fri, Dec 11, 2009 at 03:03 -0800, Bearophile wrote:
> Wolodja Wentland:
> > Which library would you choose?
> 
> This one probably uses low memory, but I don't know if it works still:
> http://osl.iu.edu/~dgregor/bgl-python/

That project looks not that maintained and graph-tool [1] is based on
boost as well, so I don't see the advantage in choosing bgl-python over
graph-tool.

The point is, that I am not sure if using graph-tool has any advantages
over networkx at all. It looks like a great library, supports filtered
graphs which I find pretty useful, but have not used it yet.

[1] http://projects.forked.de/graph-tool/
-- 
  .''`. Wolodja Wentland 
 : :'  :
 `. `'` 4096R/CAF14EFC 
   `-   081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Which graph library is best suited for large graphs?

2009-12-11 Thread Wolodja Wentland
On Fri, Dec 11, 2009 at 08:55 -0500, Neal Becker wrote:
> Bearophile wrote:
> > Wolodja Wentland:
> >> Which library would you choose?

> > This one probably uses low memory, but I don't know if it works still:
> > http://osl.iu.edu/~dgregor/bgl-python/

> How about python interface to igraph?

Don't know :-) as I have not yet worked with it. Why do you recommend it?
-- 
  .''`. Wolodja Wentland 
 : :'  :
 `. `'` 4096R/CAF14EFC 
   `-   081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Which graph library is best suited for large graphs?

2009-12-11 Thread Wolodja Wentland
On Fri, Dec 11, 2009 at 07:31 -0800, IngoognI wrote:
> On Dec 11, 11:12 am, Wolodja Wentland 
> wrote:

> > Which library would you choose?
> 
> looking at the galery at networx, it seems to be all balls 'n sticks,
> how about writing the data to a file POV-Ray can read and render it
> there?

Huh? I am not really concerned about rendering the graphs but after a
library with a small memory footprint. Preferably one that contains a
number of typical algorithms.
-- 
  .''`. Wolodja Wentland 
 : :'  :
 `. `'` 4096R/CAF14EFC 
   `-   081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Memory consumption of multiprocessing.Pool

2009-12-15 Thread Wolodja Wentland
Hi all,

I have a problem with the memory consumption of multiprocessing.Pool()'s
worker processes. I have a parent process that has to handle big data
structures and would like to use a pool of processes for computations.

The problem is, that all worker processes have the same memory
requirement as the parent one, although they do *not* use any the parent
processes data structures. The following snippet illustrates this
behaviour:

--- snip ---
#!/usr/bin/env python
# -*- coding: utf-8 -*-

import multiprocessing
import time

def worker(el):
time.sleep(10)

def main():
print 'Init Pool 1'
p_pool = multiprocessing.Pool()

print 'Call pool.map()'
p_pool.map(worker, range(multiprocessing.cpu_count()))

print 'Allocate memory'
eat_memory = range(100)

print 'Call pool.map()'
p_pool.map(worker, range(multiprocessing.cpu_count()))

print 'Delete pool 1'
del p_pool

print 'Init Pool 2'
p_pool = multiprocessing.Pool()

print 'Call pool.map()'
p_pool.map(worker, range(multiprocessing.cpu_count()))

print 'Delete pool 2'
del p_pool

if __name__ == '__main__':
main()
--- snip ---

You will see that the memory consumption of the worker processes will be
roughly the same for the first two calls to p_pool.map(), but rise for
the third.

How can I make sure that 'eat_memory' does not use any memory in the
pool processes? This is important as I don't always know when a pool is
instanciated and the pool processes should *not* have the same memory
requirements as the parent process.

Am I missing something here?

-- 
  .''`. Wolodja Wentland 
 : :'  :
 `. `'` 4096R/CAF14EFC 
   `-   081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: share dictionary between processes

2009-12-18 Thread Wolodja Wentland
On Thu, Dec 17, 2009 at 23:48 -0800, blumenkraft wrote:
> I want to share dictionary between two distinct processes.
> Something like this:
> 
> first.py
> import magic_share_module
> def create_dictionary():
> return {"a": 1}
> 
> magic_share_module.share("shared_dictionary",
> creator.create_dictionary)
> while True:
>  pass

Have a look at multiprocessing.Manager() it provides (among other
things) proxies for dictionaries that can be used in different threads.
These are even accessible on different hosts if configures correctly.
-- 
  .''`. Wolodja Wentland 
 : :'  :
 `. `'` 4096R/CAF14EFC 
   `-   081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: problem with multiprocessing and defaultdict

2010-01-12 Thread Wolodja Wentland
On Tue, Jan 12, 2010 at 11:48 +0100, wiso wrote:
> They sent back the object filled with data. The problem is very simple: I 
> have a container, the container has a method read(file_name) that read a 
> huge file and fill the container with datas. I have more then 1 file to read 
> so I want to parallelize this process. The reading method is quite slow 
> because it involves regex.

Take a look at multiprocessing.Manager and use one to proxy access to a
*shared* container to your container from all processes.

If your container is a dict it is as easy as:

manager = multiprocessing.Manager()
managed_dict = manager.dict()
...

-- 
  .''`. Wolodja Wentland 
 : :'  :
 `. `'` 4096R/CAF14EFC 
   `-   081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: a problem with writing a generator

2010-01-14 Thread Wolodja Wentland
On Thu, Jan 14, 2010 at 15:11 +0100, Paweł Banyś wrote:

> I seem to have some blackout in my mind because I cannot understand how
> to use a generator functionality to complete the task. If anybody has
> already done such thing I would be very grateful for any guidance.

I guess the following slides will help you a lot:

http://www.dabeaz.com/generators/

If you have further questions just ask and have fun playing with
generators in Python ...

-- 
  .''`.     Wolodja Wentland 
 : :'  :
 `. `'` 4096R/CAF14EFC 
   `-   081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: basic Class in Python

2010-01-17 Thread Wolodja Wentland
On Sun, Jan 17, 2010 at 15:05 -0800, BarryJOgorman wrote:
 [...]
> class Person:
> def _init_(self, name, job=None, pay=0):
  ^^ --> __init__(self, ...

It's __init__() not _init_()!

Have fun learning Python!

-- 
  .''`. Wolodja Wentland 
 : :'  :
 `. `'` 4096R/CAF14EFC 
   `-   081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: strange behaviour of colon within string

2009-10-19 Thread Wolodja Wentland
On Mon, Oct 19, 2009 at 05:44 -0700, khany wrote:
> i am relatively new to python and i am trying to convert a php app i
> have over to it using googleapps.

Welcome!

> anyway here is the problem. i poll ebay API which has in its XML " xml version="1.0" encoding="utf-8"?> xmlns="http://www.ebay.com/marketplace/search/v1/services";>   "

> however it fails to create the string UNLESS i remove the colon (:) in
> the http section. i tried to substitute it with chr(58) but it errors
> the same without showing why. does anyone know what i am doing wrong?
> is this peculiar to googleapps?

Could you please elaborate on "fails to create the string", preferably
in the form of a traceback. It might also help us if we knew what
behaviour you expected or what do you mean by "i remove the colon" .

Wolodja


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Unicode again ... default codec ...

2009-10-22 Thread Wolodja Wentland
On Thu, Oct 22, 2009 at 13:59 +0200, Lele Gaifax wrote:
> "Gabriel Genellina"  writes:

>> unittest, or ultimately, this bug: http://bugs.python.org/issue4947
> http://bugs.python.org/issue4947#msg87637 as the best fit, I think

You might also want to have a look at:

http://bugs.python.org/issue1293741

I hope this helps and that these bugs will be solved soon.

Wolodja


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: A new way to configure Python logging

2009-10-23 Thread Wolodja Wentland
l. If PEP 391 would have already been implemented right
now I would expose the logging configuration to the user in:

~/.foo/logging

load the dictionary and *programmatically* change the configuration to
meet the user demands (quiet, verbose, file, ...) stated with command
line options by adding/deleting/changing handlers in the dict before
passing it to dictConfig.

That seems suboptimal. ;-)

What I would *love* to see in the future would be:

* Default logging configuration in a YAML/JSON/... file somewhere in
  {/etc/foo/logging.conf, WindowsFooMagic/logging.conf} which describes
  all loggers/handlers/filters/... that *might* get used by the
  application eventually

* Additionally: The possibility to *override* some parts of the
  configuration in another file (files?). 

* The possibility to enable/disable certain parts of the configuration.

* Access to all parts of the logging infrastructure, so that I can adapt
  already configured parts to my actual needs.

* Easy configuration of a lower *and* upper bound for Handlers, so that
  I can easily add additional (more verbose) Handlers without fear of
  messages getting logged multiple times.

The point of all this is, that the final configuration of the logging
system is unknown until the configuration files *and* the command
line have been parsed and does not change (often) afterwards.

My main idea is to see the configuration files not as the final
configuration of the logging system but rather as a definition of the
building blocks that can be plucked together easily programmatically if
the developer sees the need to do so.

with kind regards

Wolodja Wentland

Post Scriptum

I just wrote what came to my mind. It might be that I am not aware of
better ways to deal with incremental configuration. And I read PEP 391
for the first time today, so I might have overlooked a lot of points. 

But this is how I do it right now. Please point out anything that might
make my life easier.


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: A new way to configure Python logging

2009-10-24 Thread Wolodja Wentland
On Sat, Oct 24, 2009 at 07:54 +, Vinay Sajip wrote:
> Wolodja Wentland  cl.uni-heidelberg.de> writes:
[snip]
> > foo
> >  |__bar
> >  |__baz
> >  |__newt
> > |___witch
> > 
> > I set every loggers log level to DEBUG and use the respective logger in

> You only need set foo's level to DEBUG and all of foo.bar, foo.baz etc.
> will inherit that level. 
OK, thanks for pointing that out!

[snip]
> > Among other levels specific to the application, like PERFORMANCE for
> > performance related unit tests, ...
> 
> I'm not sure what you mean here - is it that you've defined a custom level
> called PERFORMANCE?
Exactly. I used that particular level for logging within a unit test
framework for messages about performance related tests. Combined with a
Handler that generated HTML files from the LogRecord queue using various
templates (make, jinja, ...) it became a nice way to create nice looking
test reports.

Could a HTMLHandler be added to the standard set? Preferably one that
leaves the choice of the template engine to the user.

> > Application User Interface
> [snip]
> All of this sounds quite reasonable.
 Great :-)
>  
> > Implementation
> > --
> > 
> > You have rightfully noted in the PEP, that the ConfigParser method
> > is not really suitable for incremental configuration and I therefore
> > configure the logging system programmatically.

> Since you allow users the ability to control logging from the command-line,
> you need to do programmatic configuration anyway.
Yes, but that could be made easier. (see below)

> > I create all loggers with except the root (foo) with:
> > 
> > LOG = logging.getLogger(__name__)
> > LOG.setLevel(logging.DEBUG)
> >
> > within each module and then register suitable handlers *with the root
> > logger* to process incoming LogRecords. That means that I usually have a
> > StreamHandler, a FileHandler among other more specific ones.
> 
> See my earlier comment about setting levels for each logger explicitly. How
> do you avoid low-level chatter from all modules being displayed to users? Is
> it through the use of Filters?

Exactly. The Handlers will usually employ elaborate filtering, so they
can be "plugged together" easily:

- User wants html? Ah, just add the HTMLHandler to the root logger
- User wants verbose output? Ah, just add the VerboseHandler to ...
- ...

> There are times where specific handlers are attached lower down in the
> logger hierarchy (e.g. a specific subsystem) to send information to a relevant
> audience, e.g. the development or support team for that subsystem. 

Guess I never had the need for that. 

> Technically you can achieve this by attaching everything to the root
> and then attaching suitable Filters to those handlers, but it may be
> easier in some cases to attach the handlers to a lower-level logger
> directly, without the need for Filters.

Which is exactly what I do and I think that it fits my particular
mindset. I see the root handler basically as a multiplexer that feeds
LogRecords to various various co-routines (ie handlers) that decide what
to do with them. I like working on the complete set of LogRecords
accumulated from different parts of the application. The
handler/filter/... naming convention is just a more verbose/spelled out
way of defining different parts of the pipeline that the developer might
want to use. I guess I would welcome general purpose hook for each edge
in the logger tree and in particular one hook feeding different
co-routines at the root logger.

> though your configuration would have to leave out any handlers which
> are optionally specified via command-line arguments.
> > * Additionally: The possibility to *override* some parts of the
> >   configuration in another file (files?). 
> 
> That requirement is too broad to be able to give a one-size-fits-all
> implementation.

I was thinking along the line of ConfigParser.read([file1, file2, ...]),
so that you could have:

--- /etc/foo/logging.conf ---
...
formatters:
  default:
format: '%(asctime)s %(levelname)-8s %(name)-15s %(message)s'
datefmt: '%Y-%m-%d %H:%M:%S'
...
--- snip ---

and:

--- ~/.foo/logging.conf ---
formatters:
  # You can adapt the message and date format to your needs here.
  # The following placeholder can be used:
  # asctime- description
  # ...

  default:
format: '%(levelname)-8s %(name)-15s %(message)s'
datefmt: '%Y-%m-%d %H:%M:%S'
--- snip ---

So that if I call:

logging.config.fromFiles(['/etc/foo/logging.conf,
   os.path.expanduser(
   '~/.foo/logging.conf')])

The user adaptations will overrule the

Re: sequential multiple processes

2009-10-24 Thread Wolodja Wentland
On Sun, Oct 25, 2009 at 00:41 +0900, Hyunchul Kim wrote:
>    How to run multiple processes with sequential input of a thousand 
> of data
>    in a script run?
>  
>    I have a python script and 1,000 independent input data for it. 
>  
>    Previously, I divided input data into n groups and 
> ran a  same python
>    script n times to use n processors. 
>  
>    It's inconvenient.  
>  
>    How can I do same thing in a signle script running? 
>  

Have a look at [1] it describes a way to multiplex data to different
receivers. You can then combine that with the multiprocessing/threading
module in the stdlib.

kind regards

Wolodja

[1] http://www.dabeaz.com/generators/


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: A new way to configure Python logging

2009-10-25 Thread Wolodja Wentland
On Sun, Oct 25, 2009 at 10:48 +, Vinay Sajip wrote:
> Wolodja Wentland  cl.uni-heidelberg.de> writes:
[ HTMLHandler, multiple configuration files ]

OK! I agree that these parts are hard to standardise and do not really
belong in the *logging* module. 

Maybe a kind soul implements a "configuration" module in the future that
accepts configuration files in a plethora of formats and uses
dictionaries as the lingua franca for final configuration.

> > I will give an example.. The basic problem I have with *all* config file
> > based configuration right now is that I have to *register* every single
> > handler/filter with a logger *within* the configuration or their
> > configuration will be lost.

> You are right, unless handlers (and filters, formatters etc.) are given
> names which can be used to refer to them across multiple configuration calls.
> This is something I am thinking about and will probably update PEP 391
> with my thoughts.

[ usage example ]

> I think your way of working is entirely reasonable, but IMO is not likely to
> be so widespread as to make it worthwile baking into the stdlib. You can
> easily build your own configuration from which you build the dict to pass
> to dictConfig().

Are these two statements not a bit contradictory? If it would be
possible to refer to all major components in logging by *unique* names
would that not mean that the usage example I gave is possible?

I think we managed to single out the sole requirement I would have
towards 'logging' that is missing today.

Id est: The possibility to refer/retrieve/... all major components used
by logging (loggers, handlers, filters, formatters, adaptors) by a
*unique* name. That would enable the developer to deal with them in a
consistent way irregardless of the way they were initially defined
(configuration file, programmatically).

Is this way to deal with logging really that uncommon? I guess I have
to read a lot code to see how other people do it as this would be the
way that feels most natural to me.

BTW, the LoggerAdaptor class looks really useful. I just discovered it
and I have the feeling that I might use it frequently.

> > * Logging Expert
> > * Developer
> > * User

> Those three roles appear reasonable, but I would say that the expert-designed
> blocks would be specialised handlers, filters and formatters. That's not a 
> full-time job, though ;-)

I completely agree. I know that the logging expert and the developer
will most likely be the same person. I just wanted to point out that the
design of the logging system and its components is a different step in
program development than the usage of said system by a developer and
different users.

Thanks again for taking this discussion to the users list. I could have
commented in the -dev thread, but did not. (I ask myself: Why?) I
therefore appreciate it a lot that you try to figure out your users
requirements before implementing them! I just love open source software!

Have a great day and let me know whatever you come up with. 

Wolodja


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: PEP 391 (Dictionary-Based Configuration for Logging) Updated

2009-10-28 Thread Wolodja Wentland
On Wed, Oct 28, 2009 at 10:27 +, Vinay Sajip wrote:
> I've updated PEP 391 (Dictionary-Based Configuration for Logging):
> http://svn.python.org/view/peps/trunk/pep-0391.txt?r1=75599&r2=75918
+1

> All feedback gratefully received!

The PEP does not seem to specify how handler are retrieved by their
name/id. How should this work? Especially given the statement:

"The handler name lookup dictionary is for configuration use only
 and will not become part of the public API for the package."

thanks for incorporating these changes.

Wolodja


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: How to get the realpath of a symbolic link?

2009-10-31 Thread Wolodja Wentland
On Sat, Oct 31, 2009 at 14:48 -0500, Peng Yu wrote:
> On Sat, Oct 31, 2009 at 1:46 PM, Terry Reedy  wrote:
> > Peng Yu wrote:
[ snip ]

> I find the following two files that define realpath. But I don't find
> 'realpath' in os.py. I looked at 'os.py'. But I don't understand how
> the function realpath is introduced in the name space in os.path.
> Would you please let me know?

> gfind . ! -path '*backup*' -name "*.py" -type f -exec grep -n "def
> realpath" {} \; -printf %p\\n\\n
> 193:def realpath(path):
> ./macpath.py
> 
> 345:def realpath(filename):
> ./posixpath.py

The os module needs to support different platforms. The os.path module
is actually one of the platform specific ones (ntpath, posixpath,
...) that are imported 'as path' depending on the platform the code is
executed.

Have a look at the source code of the os module:

--- os.py - Python 2.6.3 ---
...
f 'posix' in _names:
...
import posixpath as path

elif 'nt' in _names:
...
import ntpath as path

import nt
__all__.extend(_get_exports_list(nt))
del nt

...

else:
raise ImportError, 'no os specific module found'

sys.modules['os.path'] = path

--- snip ---

If you really want to understand how a module is working then have a
look at its source code. Python is open source --> Use that privilige!

kind regards

Wolodja Wentland


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: How to import only one module in a package when the package __init__.py has already imports the modules?

2009-10-31 Thread Wolodja Wentland
On Sat, Oct 31, 2009 at 16:53 -0500, Peng Yu wrote:
> On Sat, Oct 31, 2009 at 4:14 PM, Robert Kern  wrote:

[ snip ]

> I know that multiple classes or functions are typically defined in one
> file (i.e. module in python). However, I feel this make the code not
> easy to read. Therefore, I insist on one class or function per file
> (i.e module in python).

Are you serious? Do you *really* put each function in its own file? How
exactly does this enhance the readability of the source code? Especially
if you compare that to a (sic!) modularisation scheme that groups
classes and functions together by task or semantic relatedness.

regards

Wolodja


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: How to import only one module in a package when the package __init__.py has already imports the modules?

2009-10-31 Thread Wolodja Wentland
On Sat, Oct 31, 2009 at 18:29 -0500, Peng Yu wrote:
> On Sat, Oct 31, 2009 at 5:45 PM, Wolodja Wentland
>  wrote:
> > On Sat, Oct 31, 2009 at 16:53 -0500, Peng Yu wrote:

> > Are you serious? Do you *really* put each function in its own file? How
> > exactly does this enhance the readability of the source code? Especially
> > if you compare that to a (sic!) modularisation scheme that groups
> > classes and functions together by task or semantic relatedness.

> If two functions are too long to put in file, I generally put them in
> two different files. 

If it should ever happen that two functions are too long to put in a
single file you should refactor your code. It is usually a good idea of
breaking problems down into single steps (ie functions) so you never end
up with a 5000 SLOC *function*.

How do functions of this length enhance the readability of your source
code?

> And I always put a single class in a file.

Why? What do you gain by that?

> Suppose that I have many functions in one file, it is not clear to see
> how many functions are in the file at first glance.

Use a better editor/IDE for that.

[snip]

I thought about answering your post in greater detail, but i would like
to evaluate your style of work first. Is there any place where I can
have a look at some of your source code? It would be perfect if it is a
medium sized project with said unit tests, packages and
function-modules and the rest you described.

Why does not a single module in in the stdlib follow your code layout
scheme?

regards

Wolodja


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: how to create a pip package

2009-11-10 Thread Wolodja Wentland
On Mon, Nov 09, 2009 at 19:48 -0800, Phlip wrote:
> I have a single file that I need my crew to pip install.

Where do you plan to host this file? Will it be available on PiPy?

> When I Google for "how to create a pip package" I don't hit anything.
> Of course that info is out there; I can't seem to pick up the trail of
> breadcrumbs to it.

As has already been noted in this thread you do not create a file
specifically for pip, but rather use the standard (or soon to be) way of
distributing Python distributions outlined in:

http://docs.python.org/library/distutils.html#module-distutils
http://packages.python.org/distribute/

If you do not plan to host your "file" on pypi you can easily create a
pip requirements file that references a VCS of your choice. Read how to
do this in the pip documentation on "editable" packages:

http://pypi.python.org/pypi/pip/

kind regards

Wolodja


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: how to create a pip package

2009-11-10 Thread Wolodja Wentland
On Tue, Nov 10, 2009 at 06:30 -0800, Phlip wrote:
> On Nov 10, 1:54 am, Wolodja Wentland 
> wrote:
> 
> > http://docs.python.org/library/distutils.html#module-distutils
> > http://packages.python.org/distribute/
> 
> ktx... now some utterly retarded questions to prevent false starts.

> the distutils page starts with "from distutils.core import setup".

[..]

> from setuptools import setup, find_packages

It will be enough to use the method outlined in the distutils
documentation. Setuptools is a third-party library that used to be the
de-facto standard for Python packaging. I don't want to go into detail
why setuptools might not be the best choice for a project and leave that
to [1] and [2].

The Distribute project was started in order to fix several bugs in
setuptools which was essentially unmaintained for a year and provides a
backward compatible fork in the 0.6 branch.

kind regards

Wolodja

[1] http://www.b-list.org/weblog/2008/dec/14/packaging/
[2] http://blog.ianbicking.org/2008/12/14/a-few-corrections-to-on-packaging/


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: how to create a pip package

2009-11-10 Thread Wolodja Wentland
On Tue, Nov 10, 2009 at 13:09 -0800, Phlip wrote:
> will pip pull from a simple GitHub repo? or do I need to package
> something up and put it in a pythonic repository somewhere?

I don't quite understand, but would say: both ;-)

You can't tell pip to pull from arbitrary git repositories, but only
from those that contain the packaging code as outlined in the distutils
documentation.

These git repositories do *not* have to be hosted on pypi. You should
however be able to do the following:

$ git clone git://example.com/repo.git
$ cd repo
$ python setup.py install

The pip requirement file would contain the following line:

-e git+git://example.com/repo.git#egg=rep

I hope this answers your questions :-D

--
  .''`. Wolodja Wentland 
 : :'  :
 `. `'` 4096R/CAF14EFC 
   `-   081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: how to create a pip package

2009-11-11 Thread Wolodja Wentland
On Tue, Nov 10, 2009 at 20:25 -0800, Phlip wrote:
> On Nov 10, 3:11 pm, Wolodja Wentland 
> wrote:
> 
> > The pip requirement file would contain the following line:

> > -e git+git://example.com/repo.git#egg=rep

> Let me ask it like this. What happens when a user types..?

>sudo pip install repo

pip will check for 'repo' on pypi, find nothing and stop processing.

> Is github one of the default sites pip scans?

No.

> If so, will it rip a requirements file in the root of repo.git? 

No.

>If so, what file name should that have?

You can choose any name you like. I think I have to explain a bit
more.

The requirement file is basically a list of *all installed
distributions* in a Python environment and is usually created by 'pip
freeze'. It is merely a way to tell pip later which distributions it
should try to install.

The basic way of working with requirement files if this:

1. Create virtual environment [1]

2. Install *your* package and all of its dependencies within the virtual
   environment

3. Run 'pip freeze' to get the list of installed distributions and write
   it to a file. This file is the requirement file.

4. Give the requirement file to someone else

5. Create a virtual environment

6. Run 'pip install -r requirement_file.txt' to install all
   distributions 'frozen' in the requirements file

7. PROFIT!!!

A requirements file is not really meant to state the dependencies of a
single distribution, but rather describe the complete state an
environment is in *so it can be reconstructed later* exactly like is has
been before. This is quite important if you want to deploy application
and you want to make sure that only tested versions of you dependency
get installed.

I think of it rather in the terms of:

pip freeze  --> dpkg --get-selections
pip install -r r.txt--> dpkg --set-selections
aptitude install

AFAIK you can also host the distribution on another site than pypi, but
I am not sure how to tell pip to check there for distributions as well.
You might want to ask this on the virtualenv list.

--
  .''`. Wolodja Wentland 
 : :'  :
 `. `'` 4096R/CAF14EFC 
   `-   081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: How can pip install a GitHub code drop?

2009-11-15 Thread Wolodja Wentland
On Sun, Nov 15, 2009 at 09:16 -0800, Phlip wrote:
> How can I point pip at the repo to install the library?

> sudo pip -e git+git://github.com/Phlip/Kozmiq.git

Make that:

pip -e git+git://github.com/Phlip/Kozmiq.git#egg=Kozmiq

and (preferably) don't install into system paths ;-)

kind regards

Wolodja
-- 
  .''`. Wolodja Wentland 
 : :'  :
 `. `'` 4096R/CAF14EFC 
   `-   081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: How can pip install a GitHub code drop?

2009-11-15 Thread Wolodja Wentland
On Mon, Nov 16, 2009 at 01:11 +0100, Wolodja Wentland wrote:
> On Sun, Nov 15, 2009 at 09:16 -0800, Phlip wrote:
> > How can I point pip at the repo to install the library?

> > sudo pip -e git+git://github.com/Phlip/Kozmiq.git

> pip -e git+git://github.com/Phlip/Kozmiq.git#egg=Kozmiq

err...

pip install -e git+git://github.com/Phlip/Kozmiq.git#egg=Kozmiq
^^^ ^^^
Hope it works

-- 
  .''`. Wolodja Wentland 
 : :'  :
 `. `'` 4096R/CAF14EFC 
   `-   081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Imitating "tail -f"

2009-11-22 Thread Wolodja Wentland
On Sun, Nov 22, 2009 at 03:43 +0100, Ivan Voras wrote:
> I'm trying to simply imitate what "tail -f" does, i.e. read a file, wait
> until it's appended to and process the new data, but apparently I'm
> missing something.
[..]
> Any advice?

Have a look at [1], which mimics "tail -f" perfectly. It comes from a
talk by David Beazley on generators which you can find at [2] and
[3].

Enjoy!

[1] http://www.dabeaz.com/generators/follow.py
[2] http://www.dabeaz.com/generators-uk/
[3] http://www.dabeaz.com/coroutines/

-- 
  .''`. Wolodja Wentland 
 : :'  :
 `. `'` 4096R/CAF14EFC 
   `-   081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: creating pipelines in python

2009-11-23 Thread Wolodja Wentland
On Sun, Nov 22, 2009 at 14:49 -0800, per wrote:
> i am looking for a python package to make it easier to create a
> "pipeline" of scripts (all in python). what i do right now is have a
> set of scripts that produce certain files as output, and i simply have
> a "master" script that checks at each stage whether the output of the
> previous script exists, using functions from the os module. this has
> several flaws and i am sure someone has thought of nice abstractions
> for making these kind of wrappers easier to write.
> does anyone have any recommendations for python packages that can do
> this?

There are various possibilities. I would suggest you have a look at [1]
which details the creation of pipelines with generators that can be used
within *one* program. If you want to chain different programs together
you can use the subprocess package in the stdlib of Python 2.6.

[1] http://www.dabeaz.com/generators/
-- 
  .''`. Wolodja Wentland 
 : :'  :
 `. `'` 4096R/CAF14EFC 
   `-   081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: determining which value is the first to appear five times in a list?

2010-02-06 Thread Wolodja Wentland
On Sat, Feb 06, 2010 at 13:24 -0500, Chris Colbert wrote:
> [(match_val_291, identifier_b), (match_val_23, identifier_b), (match_val_22,
> identifer_k)  ]
> 
> Now, what I would like to do is step through this list and find the identifier
> which appears first a K number of times.

I think you can use the itertools.groupby(L, lambda el: el[1]) to group
elements in your *sorted* list L by the value el[1] (i.e. the
identifier) and then iterate through these groups until you find the
desired number of instances grouped by the same identifier.

Let me exemplify this:

>>> from itertools import groupby
>>> instances = [(1, 'b'), (2, 'b'), (3, 'a'), (4, 'c'), (5, 'c'), (6, 'c'), 
>>> (7, 'd')]
>>> k = 3
>>> grouped_by_identifier = groupby(instances, lambda el: el[1])
>>> grouped_by_identifier = ((identifier, list(group)) for identifier, group in 
>>> grouped_by_identifier)
>>> k_instances = (group for identifier, group in grouped_by_identifier if 
>>> len(group) == k)
>>> next(k_instances)
[(4, 'c'), (5, 'c'), (6, 'c')]
>>> next(k_instances)
Traceback (most recent call last):
  File "", line 1, in 
StopIteration

There are certainly millions of ways to do this and most of them will be
better than my proposal here, but you might like this approach. Another
approach would use itertools.takewhile() or itertools.ifilter() ... Just
have a look :-)

yours sincerely
-- 
  .''`. Wolodja Wentland 
 : :'  :
 `. `'` 4096R/CAF14EFC 
   `-   081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: determining which value is the first to appear five times in a list?

2010-02-06 Thread Wolodja Wentland
On Sat, Feb 06, 2010 at 14:42 -0500, Terry Reedy wrote:
> On 2/6/2010 2:09 PM, Wolodja Wentland wrote:

> >I think you can use the itertools.groupby(L, lambda el: el[1]) to group
> >elements in your *sorted* list L by the value el[1] (i.e. the
> >identifier) and then iterate through these groups until you find the
> >desired number of instances grouped by the same identifier.

> This will generally not return the same result. It depends on
> whether OP wants *any* item appearing at least 5 times or whether
> the order is significant and the OP literally wants the first.

Order is preserved by itertools.groupby - Have a look:

>>> instances = [(1, 'b'), (2, 'b'), (3, 'a'), (4, 'c'), (5, 'c'), (6, 'c'), 
>>> (7, 'b'), (8, 'b')]
>>> grouped_by_identifier = groupby(instances, lambda el: el[1])
>>> grouped_by_identifier = ((identifier, list(group)) for identifier, group in 
>>> grouped_by_identifier)
>>> k_instances = (group for identifier, group in grouped_by_identifier if 
>>> len(group) == 2)
>>> for group in k_instances:
... print group
... 
[(1, 'b'), (2, 'b')]
[(7, 'b'), (8, 'b')]

So the first element yielded by the k_instances generator will be the
first group of elements from the original list whose identifier appears
exactly k times in a row. 

> Sorting the entire list may also take a *lot* longer.
Than what? 

Am I missing something? Is the "*sorted*" the culprit? If yes -> Just
forget it as it is not relevant.
-- 
  .''`. Wolodja Wentland 
 : :'  :
 `. `'` 4096R/CAF14EFC 
   `-   081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Finding application data after install - a solution?

2009-09-19 Thread Wolodja Wentland
Hi all,

reliably finding distribution data from your program seems to be an
unsolved issue for programs packaged with distutils. 

I have seen a lot of code that manipulates mod.__file__ to solve this
problem, but this *will* break for some installation schemes and has the
following problems:

* it breaks if the user specified '--install-data' to have a different
  value than '--install-{pure,plat}lib'
* it makes the life of linux distribution package maintainers
  unneccessarily hard, because they have to patch your code so it works
  with their file system hierarchy.
* it does not work inside eggs
* it is ugly ;-)

Good news everyone! I spend some time to solve this problems and would
like to share my snippet and ask for comments.

The idea is to fill a python module 'build_info.py' with installation
prefix information *at build time* and access the data within that
module.

--- snip ---
from distutils.command.build_py import build_py as _build_py
from types import StringType, ListType, TupleType

import distutils.core as core
import sys
import os.path
import string

class build_py(_build_py):
"""build_py command

This specific build_py command will modify module 'build_config' so that it
contains information on installation prefixes afterwards.
"""
def build_module (self, module, module_file, package):
if type(package) is StringType:
package = string.split(package, '.')
elif type(package) not in (ListType, TupleType):
raise TypeError, \
  "'package' must be a string (dot-separated), list, or tuple"

if ( module == 'build_info' and len(package) == 1 and package[0] == 
'mwdb'):
iobj = self.distribution.command_obj['install']

with open(module_file, 'w') as module_fp:
module_fp.write('# -*- coding: UTF-8 -*-\n\n')
module_fp.write("DATA_DIR = '%s'\n"%(
os.path.join(iobj.install_data, 'share')))
module_fp.write("LIB_DIR = '%s'\n"%(iobj.install_lib))
module_fp.write("SCRIPT_DIR = '%s'\n"%(iobj.install_scripts))

_build_py.build_module(self, module, module_file, package)

core.setup(name='foo',
   cmdclass={'build_py': build_py},
   ...
   )
--- snip ---

This works for installers based on distutils and those based on
setuptools and is IMHO a much cleaner and nicer solution than having to
use a complicated API that relies on externally managed information or
__file__ hacks.

Before you copy this i should note that i plan to use string templates
within 'build_info' and just substitute the wanted information and not
generate the whole file from scratch. 

The module detection logic and exception handling might need some work
as well. ;-)

I have the following questions:

1. Is the distutils 'API' i use here likely to break?
2. Can you think of a better way to do this?

with kind regards

Wolodja Wentland


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Where are python module installed?

2009-09-20 Thread Wolodja Wentland
On Sun, Sep 20, 2009 at 20:30 -0500, Peng Yu wrote:
> I configured python-2.6.2 with my own --prefix, then 'make' and 'make
> install'. I only find the following dirs and I don't find any python
> modules in the directory. Do python modules come with python-2.6.2?
Yes.

> $ ls
> bin  include  lib  share
    ^^^ here

with kind regards

Wolodja Wentland


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Finding application data after install - a solution?

2009-09-21 Thread Wolodja Wentland
On Mon, Sep 21, 2009 at 23:52 -0300, Gabriel Genellina wrote:
> En Sat, 19 Sep 2009 12:03:52 -0300, Wolodja Wentland
>  escribió:
> 
> >reliably finding distribution data from your program seems to be an
> >unsolved issue for programs packaged with distutils.

[...]

> Isn't pkgutil.get_data() what you're looking for?
> I'm not sure it covers all your use cases though.

No that pkgutil.get_data() is exactly *not* what I am looking for ;-)

I have a library which *could* use external data and use this additional
information to cover more cases, but works perfectly fine without this
data. The data could change more frequently than i want to release the
library.

I use distutils data_files=... to have the data installed, but there
is/was (until the recipe I posted here) no way to find that data again,
because the admin could define arbitrary prefixes for libs, scripts and
data. That means that there might be absolutely no relation between the
module path and the data file path.

I want to:

1. Give administrators the freedom to install the data wherever they
   want
2. Adhere to the FHS (installing data within modules breaks it)
3. Be able to find that data again regardless of the installation
  scheme used

1 and 2 are easily solved... It was just not possible to find the data
again. The snippet in the original code solves that.

I still don't think that relying on these internal structures within
distutils is supported, but can't think of a better way.

so long and thanks for all the fish

Wolodja


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Finding application data after install - a solution?

2009-09-22 Thread Wolodja Wentland
On Tue, Sep 22, 2009 at 07:42 -0700, Aahz wrote:
> >I want to:
> >
> >1. Give administrators the freedom to install the data wherever they
> >   want
> >2. Adhere to the FHS (installing data within modules breaks it)
> >3. Be able to find that data again regardless of the installation
> >  scheme used
> >
> >1 and 2 are easily solved... It was just not possible to find the data
> >again. The snippet in the original code solves that.
> Given your mention of FHS, it sounds like you are focused on Unix-like
> systems, in which case why not rely on the standard mechanisms for config
> files?

I do not intentionally focus on UNIX type systems, but I have grown up
with *nix and I rather follow one scheme than none at all. But the
proposed way works on Windows as well, although the users might find
previously unseen directories like 'PREFIX/share/foo/doc' and the like
on their system.

How to solve this platform independently? And what standard mechanisms
for config files do you mean? Are there library functions I overlooked?

But the data I am speaking about are not config files. I will explain in
more detail so you can understand the problem.

I have a library that parses markup that has a canonical form and a
localised one. 'Image:' for example is the canonical form and the one
used for English. In German the canonical form *and* a localised one is
used ('Bild:'). This scheme (localised+canonical) holds true for all
(260+) languages this markup is used in.

The library itself works fine with the canonical forms, but loads
language specific markup definitions *if they are available* . I
therefore don't want to install this data along with the library for the
following reasons:

1. The data is not needed if the user just want to work with English
   data

2. The data could change much more frequently than the library that
   works with the data.

   If I couple library and data together I would have to make a new
   release *every time* the data changes, even if the user does not
   care about the other languages.

I therefore opted to provide a seperate (Python) distribution foo-data
which contains the language specific markup definitions and install it
to '$DATA_PREFIX/share/foo/languageinfo' .

As $DATA_PREFIX is only known at build time there was (until now) no
reliable way to find the data if the only information one can get is
$LIB_PREFIX, because these two might be totally unrelated.

I hope that clarifies my point

Wolodja

P.S Which markup format am I speaking about? First one to answer gets a
brand new Internet!


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: pycopg2 build problems

2009-09-23 Thread Wolodja Wentland
On Wed, Sep 23, 2009 at 12:24 -0700, devaru wrote:
> I'm trying to install psycopg2 on my system. I followed the
> instruction in INSTALL file and gave the command
> python setup.py build
> running build
> running build_py
> running build_ext
> error: No such file or directory

I ran into this some days ago. The problem is not related to the
distribution you downloaded, but to missing information about PostgreSQL
itself.

IIRC the file in question is "/usr/bin/pg_config". The file is probably packaged
in some lib*-dev package on your distribution.

--- Debian example ---
$ apt-file search /usr/bin/pg_config
libpq-dev: /usr/bin/pg_config
--- snip ---

thanks for all the fish

Wolodja


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: pycopg2 build problems

2009-09-23 Thread Wolodja Wentland
On Wed, Sep 23, 2009 at 22:38 +0200, Wolodja Wentland wrote:
> --- Debian example ---
> $ apt-file search /usr/bin/pg_config
> libpq-dev: /usr/bin/pg_config
> --- snip ---

Just wanted to note that libpq-dev is a dependency of
postgresql-server-dev-8.4 on squeeze. So you *might* want to consider
installing that instead of libpq-dev.

thanks for all the fish

Wolodja


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Finding application data after install - a solution?

2009-09-24 Thread Wolodja Wentland
On Thu, Sep 24, 2009 at 04:07 -0300, Gabriel Genellina wrote:
> >I do not intentionally focus on UNIX type systems, but I have grown up
> >with *nix and I rather follow one scheme than none at all. But the
> >proposed way works on Windows as well, although the users might find
> >previously unseen directories like 'PREFIX/share/foo/doc' and the like
> >on their system.

> I think many Windows users would say WTF!? when seeing those
> directories - and send cordial greetings to you, your parents and
> your whole family :)

That is probably true, but Windows has a 'etc' directory
(c:\windows\system32\drivers\etc) which AFAIK contains the hosts file
and/or other goodies from *NIX.

> Instead of ending with, e.g., a directory like c:\usr\share\foo\doc,
> your program should ask the OS for the special folder
> CSIDL_COMMON_APPDATA and add the foo\doc part. Or any other suitable
> standard folder. 

Is CSIDL_COMMON_APPDATA and environment variable set on all Windows
flavours? Are there versions that relying on this env var (if it is one)
will cause havoc and the death of millions of kittens? (ie. is it
available on all versions)

Where can I find Information on these "other" standard folders? Sorry,
but I have not used a Windows machine in ages.

> A directory like c:\usr\share\foo on Windows is as ridiculous as
> /Documents\ and\ Settings/All\ Users/Application\ Data/foo on any unix
> like system.

Agreed! Although I find the *NIX flavour more palatable to my eye ;-)

> >As $DATA_PREFIX is only known at build time there was (until now) no
> >reliable way to find the data if the only information one can get is
> >$LIB_PREFIX, because these two might be totally unrelated.
> 
> You should probably raise this issue at the distutils-sig mailing
> list: http://www.python.org/community/sigs/current/distutils-sig/

I already did [1] but the distutils developers are quite busy discussing
a the new format for setup.cfg. I did not got any feedback on the
solution I developed and thought that other Python users might have a
better solution to this problem.

How do *you* deal with application data in your programs? Is there a way
that works on as many platforms as possible?

so long

Wolodja

[1] http://mail.python.org/pipermail/distutils-sig/2009-September/013238.html


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Finding application data after install - a solution?

2009-09-24 Thread Wolodja Wentland
On Thu, Sep 24, 2009 at 12:51 +0100, Tim Golden wrote:
> Wolodja Wentland wrote:
> >Is CSIDL_COMMON_APPDATA and environment variable set on all Windows
> >flavours?
> 
> Certainly all those which Python currently supports. There are
> some small subtleties which have changed between older and
> newer versions.

Great!

> >Where can I find Information on these "other" standard folders? Sorry,
> >but I have not used a Windows machine in ages.

> http://msdn.microsoft.com/en-us/library/bb762494%28VS.85%29.aspx

Thank you! I will definitely consider adding better Windows installation
support in one of the next releases.

so long

Wolodja


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Finding application data after install - a solution?

2009-09-25 Thread Wolodja Wentland
On Fri, Sep 25, 2009 at 02:24 -0300, Gabriel Genellina wrote:
> En Thu, 24 Sep 2009 08:21:59 -0300, Wolodja Wentland
> >How do *you* deal with application data in your programs? Is there a way
> >that works on as many platforms as possible?

> On linux, using a configuration file like /etc/fooapplication.conf
> On Windows, using a registry key under
> HKEY_LOCAL_MACHINE\Software\My Bussiness Name\Foo Application.
> In either place, there is an entry pointing to the actual directory
> where data is stored.

How do you you *install* this file within /etc ? Do users have to copy
it themselves into /etc from DATA_DIR/foo/etc/fooapplication.conf.sample
?

> I never had the problem you want to solve, probably I'm just lucky
> (or perhaps my users are "geek" enough to know how to edit a config
> file, or dumb enough to never change the system settings...)

I am just trying to support all installation schemes supported by
distutils. That's all ;-)

so long

Wolodja


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list