Re: Override a method but inherit the docstring

[Peter Otten]

Ben Finney wrote:

> Howdy all,
> 
> The following is a common idiom::
> 
> class FooGonk(object):
> def frobnicate(self):
> """ Frobnicate this gonk. """
> basic_implementation(self.wobble)
> 
> class BarGonk(FooGonk):
> def frobnicate(self):
> special_implementation(self.warble)
> 
> The docstring for ‘FooGonk.frobnicate’ is, intentionally, perfectly
> applicable to the ‘BarGonk.frobnicate’ method also. Yet in overriding
> the method, the original docstring is not associated with it.
> 
> Ideally there would be a way to specify that the docstring should be
> inherited. The best I can come up with is::
> 
> class BarGonk(FooGonk):
> def frobnicate(self):
> special_implementation(self.warble)
> frobnicate.__doc__ = FooGonk.frobnicate.__doc__
> 
> but that violates DRY (the association between BarGonk and FooGonk is
> being repeated), puts the docstring assignment awkwardly after the end
> of the method instead of at the beginning where docstrings normally go,
> and reads poorly besides.
> 
> What is the most Pythonic, DRY-adherent, and preferably least-ugly
> approach to override a method, but have the same docstring on both
> methods?

Just thinking aloud: Write a patch for pydoc that looks up the base-class 
documentation.

B.f.__doc__ will continue to return None, but

help(B.f) will show something like

No documentation available for B.f. 

Help for A.f:
yadda yadda


Of course that might be misleading when A.f and B.f are up to something 
completely different...

Peter

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: UnicodeEncodeError: 'ascii' codec can't encode character u'\xb7' in position 13: ordinal not in range(128)

[Nobody]

On Thu, 16 Jul 2009 20:26:39 -0700, akhil1988 wrote:

> Well, you were write: unintentionally I removed strip(). But the problem does
> not ends here:
> 
> I get this error now:
> 
>  File "./temp.py", line 488, in 
> main()
>   File "./temp.py", line 475, in main
> for line in sys.stdin:
>   File "/usr/local/lib/python3.1/codecs.py", line 300, in decode
> (result, consumed) = self._buffer_decode(data, self.errors, final)
> UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-2: invalid
> data
> 
> for this line:
> â

Right. You're running in a locale whose encoding is UTF-8, but feeding
data which isn't valid UTF-8 to stdin. If you want to use data with a
different encoding, you need to replace sys.stdin, e.g.:

import sys
import io
sys.stdin = io.TextIOWrapper(sys.stdin.detach(), encoding = 'iso-8859-1')

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: urllib with x509 certs

[Lacrima]

Hello!

I've solved this problem, using pyCurl.
Here is sample code.

import pycurl
import StringIO
b = StringIO.StringIO()
c = pycurl.Curl()
url = 'https://example.com/'
c.setopt(pycurl.URL, url)
c.setopt(pycurl.WRITEFUNCTION, b.write)
c.setopt(pycurl.CAINFO, 'cert.crt')
c.setopt(pycurl.SSLKEY, 'mykey.key')
c.setopt(pycurl.SSLCERT, 'mycert.cer')
c.setopt(pycurl.SSLKEYPASSWD , 'pass phrase')
c.perform()

This also allow to specify CA, so your requests are more secure then
with urllib.

With regards, Max.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Einstein summation notation (was: question of style)

[Steven D'Aprano]

On Thu, 16 Jul 2009 23:13:51 -0700, koranthala wrote:

>> That test was designed to treat None as a boolean False, without
>> noticing that numeric 0 is also treated as False and could make the
>> test do the wrong thing.  This is an extremely common type of error.
> 
> Actually, I felt that 0 not being considered False would be a better
> option.
> I had lot of instances where 0 is a valid value and always I had to put
> checks specifically for that.
> For me, None and False should be equal to False in if checks, every
> thing else being considered True.

I believe it would be a mistake for None == False to return True.

As far as having only None and False to be considered equivalent in truth 
contexts, that's awfully limiting. It is very useful to be able to write 
e.g.:

if header or body or footer:
print assemble_page(header, body, footer)

and have empty strings to be equivalent to False.


Lisp and (I think) Ruby behaves as you want. Python doesn't. We think 
it's Lisp and Ruby that got it wrong :)


> Can someone let me know why 0 is considered equal to False? There should
> be real good reasons for it, only that I cannot think of one.

Having 0 considered false goes back to branching in machine language, 
which I understand are typically. The Null value (e.g. nil in Pascal, nul 
in C) are usually implemented as pointer values equal to an address of 
zero.

It essentially boils down to this: the behaviour of if...else is a 
convention. The language designer might choose to have an explicit 
Boolean type, and prohibit "if None" (e.g. Pascal does this). It might 
choose to treat flags as integers, and branch according to the integer 
being zero (false) or non-zero (true) (e.g. Forth does this). Python 
allows any object to be tested by if, and branches according to whether 
it is Something or Nothing. It's a logical dichotomy, and treating 0 and 
[] as Something instead of Nothing would make it arbitrary and strange.



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Einstein summation notation (was: question of style)

[Paul Rubin]

Steven D'Aprano  writes:
> It is very useful to be able to write e.g.:
> if header or body or footer:
> print assemble_page(header, body, footer)
> and have empty strings to be equivalent to False.

Why doesn't assemble_page properly handle the case where header, body,
and footer are all empty?  That would let you eliminate the if.  "Make
sure your code 'does nothing' gracefully" (Kernighan and Plauger).
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: invoke method on many instances

[Steven D'Aprano]

On Fri, 17 Jul 2009 05:19:50 +, Alan G Isaac wrote:

> As a recurrent situation, I need to invoke the same method on many
> instances.  Speed matters, but the solution should be pure Python.  Is
> the following convenience function a reasonable approach?
>
> def apply2(itr, methodname, *args, **kwargs):
> f = operator.methodcaller(methodname, *args, **kwargs)
> for item in itr:
> f(item)

I don't particularly like your naming conventions, but never mind.

You could also try this:

for obj in objects:
getattr(obj, methodname)(*args, **kwargs)




See also these recipes from the "Python Cookbook":

http://code.activestate.com/recipes/52289/
http://code.activestate.com/recipes/87370/




-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Override a method but inherit the docstring

[Steven D'Aprano]

On Fri, 17 Jul 2009 12:58:48 +1000, Ben Finney wrote:

>> Using a decorator in this manner requires repeating the super class
>> name.  Perhaps there is a way to get the bases of BarGonk, but I don't
>> think so, because at the time that the decorator is called, BarGonk is
>> not yet fully defined.
> 
> Yes, I tried a few different ways, but within the decorator it seems the
> function object is quite unaware of what class it is destined for.


When the decorator is called, the function object is just a function 
object, not a method, so there is no concept of "what class it is 
destined for".

>>> def dec(func):
... print type(func)
... try:
... print func.im_class
... except:
... print "no im_class"
... return func
...
>>> class Test(object):
... @dec
... def spam(self):
... pass
...

no im_class
>>> type(Test.spam)

>>> Test.spam.im_class



I suppose you could try to determine what namespace you're currently when 
the class is created, but that's surely going to be fragile and messy.



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: TypeError: unbound method

[Steven D'Aprano]

On Thu, 16 Jul 2009 18:08:45 -0700, Chris Rebert wrote:

> You're trying to call an instance method on the class itself, which
> doesn't make sense.

Oh I don't know, people do it all the time -- just not the way the OP 
did :)

> You need to first create an instance of the class to invoke the method
> on. i.e.:
> 
> instance = TheClass(constuctor_arguments_here)
> var1 = instance.method()

Or explicitly pass an instance to the method when calling from the class:

TheClass.method(TheClass(constuctor_arguments_here))


Which strangely enough, is useful at times.

class K(parent):
def method(self, args):
x = parent.method(self, args)
return x + 1



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to check if any item from a list of strings is in a big string?

[Steven D'Aprano]

On Thu, 16 Jul 2009 11:02:57 -0500, Pablo Torres N. wrote:

> On Thu, Jul 9, 2009 at 22:07, Steven
> D'Aprano wrote:
>> On Thu, 09 Jul 2009 18:36:05 -0700, inkhorn wrote:
>>
>>> def list_items_in_string(list_items, string):
>>>     for item in list_items:
>>>         if item in string:
>>>             return True
>>>     return False
>> ...
>>> Any ideas how to make that function look nicer? :)
>>
>> Change the names. Reverse the order of the arguments. Add a docstring.
>>
>>
> Why reverse the order of the arguments? Is there a design principle
> there?

It's just a convention. Before strings had methods, you used the string 
module, e.g.:

string.find(source, target)
=> find target in source

This became source.find(target).

In your function:

list_items_in_string(list_items, string)

"list_items" is equivalent to target, and "string" is equivalent to 
source. It's conventional to write the source first.



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

ann: servicestation - run your linux/mac services on windows without needing pywin...

[Oisin Mulvihill]

Hi There,

I gave a lightning talk about my ServiceStation project at Europython  
2009. While it
isn't written in Python (C++/Windows), I did write it with python in  
mind. Its

released under the CDDL license.

I'm hoping it will be of use to other Python users who will need to  
run their services
on windows, but don't want to learn much/anything about windows  
service programming.


The ServiceStation blurb is:

  ServiceStation allows you to run arbitrary programs as a
  service on the Windows platform. The program you wish to
  run does not need to be changed to allow it to work with
  ServiceStation or windows services.

  This project was developed with an eye to running Python web
  services on Windows, without the need to use and include Pywin32.
  This meant we could take services running on Linux/Mac and
  run them unmodified on Windows."

If your interested in this check out the projects trac page for docs  
and download


  http://www.foldingsoftware.com/servicestation

All the best,

Oisin

--
http://mail.python.org/mailman/listinfo/python-list

Re: question of style

[Piet van Oostrum]

> koranthala  (k) wrote:

>>> That test was designed to treat None as a boolean False, without
>>> noticing that numeric 0 is also treated as False and could make the
>>> test do the wrong thing.  This is an extremely common type of error.

>k> Actually, I felt that 0 not being considered False would be a better
>k> option.
>k> I had lot of instances where 0 is a valid value and always I had to
>k> put checks specifically for that.
>k> For me, None and False should be equal to False in if checks, every
>k> thing else being considered True.

>k> Can someone let me know why 0 is considered equal to False?
>k> There should be real good reasons for it, only that I cannot think of
>k> one.

Originally Python didn't have booleans. Integers were used instead (but
not exclusively). 0 was False, everything else True (with a slight
preference for 1 as True). Same as in C. Nowadays this is reflected in
bool being a subtype of int with False == 0 and True == 1. Actually it
is even closer: False and True are just 0 and 1 cast to bool, so to say.
-- 
Piet van Oostrum 
URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
Private email: p...@vanoostrum.org
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Persistent variable in subprocess using multiprocessing?

[Piet van Oostrum]

There is stil something not clear in your description.

>m> I'm using multiprocessing to spawn several subprocesses, each of which
>m> uses a very large data structure (making it impractical to pass it via
>m> pipes / pickling). I need to allocate this structure once when the
>m> process is created and have it remain in memory for the duration of
>m> the process. 

I have read this as that every subprocess has its own large
data structure and that there is no connection between these.

But seeing where the discussion is going I guess there might be
different interpretations. So can you enlighten us how the situation is?

1. Each subprocess has a copy of a data structure that is prepared by the
   master process. Therefore you want it to be passed by the fork
   1a. the data structure is constant i.e. the subprocess doesn't change it
   1b. the subprocess makes changes in its copy
2. Each subprocess has a seperate data structure not equal to the others
3. Something else.
-- 
Piet van Oostrum 
URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
Private email: p...@vanoostrum.org
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: ann: servicestation - run your linux/mac services on windows without needing pywin...

[Tim Golden]

Oisin Mulvihill wrote:

Hi There,

I gave a lightning talk about my ServiceStation project at Europython 
2009. While it
isn't written in Python (C++/Windows), I did write it with python in 
mind. Its

released under the CDDL license.

I'm hoping it will be of use to other Python users who will need to run 
their services
on windows, but don't want to learn much/anything about windows service 
programming.



If your interested in this check out the projects trac page for docs and 
download
  http://www.foldingsoftware.com/servicestation 


It's not clear at a glance how this differs from, say, the
venerable SRVANY.EXE and a slew of other "Run-things-as-a-service"
programs. Absolutely not wishing to be disparaging, but what
is servicestation offering to distinguish itself?

TJG
--
http://mail.python.org/mailman/listinfo/python-list

Re: ctype performance benchmark

[Stefan Behnel]

auror...@gmail.com wrote:
> My overall observation is that ctypes function has an overhead that is
> 2 to 3 times to a similar C extension function. This may become a
> limiting factor if the function calls are fine grained. Using ctypes
> for performance enhancement is a lot more productive if the interface
> can be made to medium or coarse grained.

I think ctypes is ok for its niche: easy interfacing with C stuff from
straight Python code, without further dependencies. There will always (and
necessarily) be a call overhead involved. Anyone who needs to care about
per-call performance would use something else anyway (do I need to mention
Cython here?)

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: no return value for threading.Condition.wait(timeout)?

[Piet van Oostrum]

> Gabriel Rossetti  (GR) wrote:

>GR> I have a 1-1 relation, I have a thread reading msgs from a network socket
>GR> and a method waiting for an answer to arrive (see my thread
>GR> "Threading.Condition problem"). I would like to be able to have a timeout
>GR> as to not block forever, so the idea is to check if I returned because of a
>GR> timeout or not.

So that case is easy I think. After the wait just check if the answer
has arrived.
-- 
Piet van Oostrum 
URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
Private email: p...@vanoostrum.org
-- 
http://mail.python.org/mailman/listinfo/python-list

ANN: psyco V2

[Christian Tismer]

Announcing Psyco V2 source release
--

This is the long awaited announcement of Psyco V2.

Psyco V2 is a continuation of the well-known psyco project,
which was called finished and was dis-continued by its author
Armin Rigo in 2005, in favor of the PyPy project.

This is a new project, using Psyco's code base with permission
of Armin. Questions aqnd complaints should go to me
(tis...@stackless.com) or the mailing list
(psyco-de...@lists.sourceforge.net);
Armin is explicitly not in charge of (t)his project any longer!

As one of the founders and an active member of the PyPy
project, I was very happy to be invited to work on Psyco
V2, by FATTOC, LLC. Psyco V2 tries to extend on the original Psyco
approach "an extension module that just makes Python faster".

Psyco is a just-in-time compiler that accelerates arbitrary
Python code by specialization. We believe that Psyco's approach
can be carried out much further than it was tried so far, when
it's first version was abandoned.

This first V2 release is source-only. There is no web-site, yet,
and there are no binaries for download. These will be available
in a few days on http://www.psyco.org .

For the time being, please stick with subversion access,
building the extension module from source code. The repository
is here:

http://codespeak.net/svn/psyco/v2/dist

Check-out the repository, and run the setup.py script,
given that you have access to a C compiler.

Psyco V2 will run on X86 based 32 bit Linux, 32 bit Windows,
and Mac OS X. Psyco is not supporting 64 bit, yet. But it
is well being considered.

The current improvements are, shortly:

  - Support for Python 2.4, 2.5 and 2.6
  - a lot of new builtins
  - generators, fast and fully supported.

More information is coming soon on http://www.psyco.org .

This is the beginning of a series of new Psyco versions.
Many more improvements are prepared and about to be published,
soon, starting with the current version 2.0.0 .

Stay tuned, this is just the beginning of psyco's re-birth!

For questions about Psyco V2, please join the mailing list

psyco-de...@lists.sourceforge.net

or contact me on IRC:

#psyco on irc.freenode.net .

Psyco V2 is fundamentally supported by FATTOC, LLC.
See http://www.fattoc.com .

Without their continuous support, this work would not have
been possible at all. I wish to express my deepest thanks
to FATTOC, for allowing me to continue on Psyco with all the
energy that this ambitious project needs, and will need.

Further special thanks are going to
Armin Rigo, John Benediktsson, David Salomon, Miki Tebeka,
Raymond Hettinger, Fabrizio Milo, Michael Foord,
Dinu Gherman, Stephan Diehl, Laura Creighton and Andrea Tismer,
for all the support and discussions.

Looking forward to a great future of Psyco!

July 17, 2009
--
Christian Tismer :^)   
tismerysoft GmbH : Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9A :*Starship* http://starship.python.net/
14109 Berlin : PGP key -> http://wwwkeys.pgp.net/
work +49 30 802 86 56  mobile +49 173 24 18 776  fax +49 30 80 90 57 05
PGP 0x57F3BF04   9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
  whom do you want to sponsor today?   http://www.stackless.com/
--
http://mail.python.org/mailman/listinfo/python-list

Re: Why not enforce four space indentations in version 3.x?

[Nobody]

On Thu, 16 Jul 2009 09:18:47 -0500, Tim Chase wrote:

> Yes, the dictatorial "a tab always equals 8 spaces"

Saying "always" is incorrect; it is more accurate to say that tab stops
are every 8 columns unless proven otherwise, with the burden of proof
falling on whoever wants to use something different.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Clarity vs. code reuse/generality

[Albert van der Horst]

In article ,
Dennis Lee Bieber   wrote:
>On 07 Jul 2009 05:05:12 GMT, Steven D'Aprano
> declaimed the following in
>gmane.comp.python.general:
>
>> Paraphrasing the Collins Dictionary of Mathematics:
>>
>> opposite sense. Sense is also used to distinguish clockwise and anti-
>> clockwise.
>>
>   Which, I believe, is the only usage I've encountered of it... In
>regards to quantum spin states in such things as Scientific American
>(though that magazine has either gone down hill in the last 30 years, or
>my expectations have gone up... Current issues read like the first years
>of Discover magazine)

I dropped my subscription when power was expressed in multiples
of US hair dryers. (I could calculate that back, and was appalled
by the energy wastage of US hair dryers. ;-) )

>--
>   WulfraedDennis Lee Bieber   KD6MOG

Groetjes Albert

--
-- 
Albert van der Horst, UTRECHT,THE NETHERLANDS
Economic growth -- being exponential -- ultimately falters.
alb...@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: UnicodeEncodeError: 'ascii' codec can't encode character u'\xb7' in position 13: ordinal not in range(128)

[Piet van Oostrum]

> akhil1988  (a) wrote:

>a> Well, you were write: unintentionally I removed strip(). But the problem 
>does
>a> not ends here:

>a> I get this error now:

>a>  File "./temp.py", line 488, in 
>a> main()
>a>   File "./temp.py", line 475, in main
>a> for line in sys.stdin:
>a>   File "/usr/local/lib/python3.1/codecs.py", line 300, in decode
>a> (result, consumed) = self._buffer_decode(data, self.errors, final)
>a> UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-2: invalid
>a> data

>a> for this line:
>a> â

Your Python assumes stdin uses utf-8 encoding, probably because your
locale says so. But it seems the input is not really utf-8 but some
other encoding.
-- 
Piet van Oostrum 
URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
Private email: p...@vanoostrum.org
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: ann: servicestation - run your linux/mac services on windows without needing pywin...

[Oisin Mulvihill]

On Jul 17, 9:34 am, Tim Golden  wrote:
> Oisin Mulvihill wrote:
> > Hi There,
>
> > I gave a lightning talk about my ServiceStation project at Europython
> > 2009. While it
> > isn't written in Python (C++/Windows), I did write it with python in
> > mind. Its
> > released under the CDDL license.
>
> > I'm hoping it will be of use to other Python users who will need to run
> > their services
> > on windows, but don't want to learn much/anything about windows service
> > programming.
> > If your interested in this check out the projects trac page for docs and 
> > download
> >  http://www.foldingsoftware.com/servicestation
>
> It's not clear at a glance how this differs from, say, the
> venerable SRVANY.EXE and a slew of other "Run-things-as-a-service"
> programs. Absolutely not wishing to be disparaging, but what
> is servicestation offering to distinguish itself?
>
> TJG

Hi There,

I have to admit I didn't come across srvany before. I may not have
used the right google fu the day I decided to do this ;) So maybe it
does all the following, however servicestation distingusihes itself
by:

 * Properly tracking all child processes launched by the command it
runs and closes them on stop/restart.

 * Monitoring the command its running and keeping it alive.

 * Allows you to set the description / name from the config file.

 * It logs useful information to the event viewer so you can see why
it couldn't run the command under its care.

 * Can interact with the desktop or not so you can run programs with a
gui but not actually see it.

 * Does not disconnect the service if you log-out as some runners do.

Thats all I can think of. I should probably add this to this wiki too,

Thanks for prompting me on this.

Oisin


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: error when compiling source on linux

[Peter Fodrek]

On Friday 17 July 2009 06:44:26 aj wrote:
> when I try to compile Python 2.6 from source code on ubuntu, I get the
> message
> /usr/bin/ld: cannot open output file python: Is a directory

This says that there is directory in the sources with same name as will be 
named output

It can only occur when you are not compiling software correctly via INSTALL or 
README file

I think you made 

./configure
make

as compiling process

if this is true try 

mkdir bulid
cd build
../configure
make
make install


Kind regards 

Peter 
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: ann: servicestation - run your linux/mac services on windows without needing pywin...

[Tim Golden]

Oisin Mulvihill wrote:
I have to admit I didn't come across srvany before. 


It's been part of the Windows Resource Kit since way back,
but like many such things is probably known more by word-of-mouth
than by Internet Searchability


 * Properly tracking all child processes launched by the command it
runs and closes them on stop/restart.

 * Monitoring the command its running and keeping it alive.

 * Allows you to set the description / name from the config file.

 * It logs useful information to the event viewer so you can see why
it couldn't run the command under its care.

 * Can interact with the desktop or not so you can run programs with a
gui but not actually see it.

 * Does not disconnect the service if you log-out as some runners do.


Looks very interesting. Definitely put this on the front page,
otherwise people may think as I did that this is just another
srvany clone. 


TJG
--
http://mail.python.org/mailman/listinfo/python-list

Re: Passing python list from C to python

[hartley]

On Jul 16, 9:26 pm, a...@pythoncraft.com (Aahz) wrote:
> In article 
> <0afc5c4d-1af5-4d0e-9442-26b51b12e...@m11g2000yqh.googlegroups.com>,
>
> hartley   wrote:
>
> >If you had loosened up on the sarcasm I would probably have read what
> >you wrote more thoroughly instead of just skimming through it. Thanks
> >for the help, but you should seriously consider doing something with
> >your patronizing attitude.
>
> http://www.mikeash.com/getting_answers.htmlhttp://www.catb.org/~esr/faqs/smart-questions.html

>From "http://www.mikeash.com/getting_answers.html":

How To Ask Questions The Smart Way is pretty condescending, especially
if you already feel insulted by somebody telling you that you need
help asking questions. If you're pointed at a guide with a filename of
smart-questions, that means this person thinks you have stupid
questions, and who needs that?

Cheers mate :)

-Hartley
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Override a method but inherit the docstring

[Ben Finney]

Peter Otten <__pete...@web.de> writes:

> Just thinking aloud: Write a patch for pydoc that looks up the
> base-class documentation.

That doesn't scale; I want the docstring to be discovered by the normal
interface (the ‘__doc__’ attribute) by *every* tool that gathers
docstrings from methods.

Also, I want to have this behaviour not for every method missing a
docstring, but only for selected methods.

> Of course that might be misleading when A.f and B.f are up to
> something completely different...

Exactly.

-- 
 \ “Facts do not cease to exist because they are ignored.” —Aldous |
  `\Huxley |
_o__)  |
Ben Finney
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Why aren't OrderedDicts comparable with < etc?

[Sion Arrowsmith]

Jack Diederich   wrote:
>It isn't an OrderedDict thing, it is a comparison thing.  Two regular
>dicts also raise an error if you try to LT them.

Since when?

Python 2.5.2 (r252:60911, Jan  4 2009, 17:40:26)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> d1 = dict((str(i), i) for i in range (10))
>>> d2 = dict((str(i), i) for i in range (20))
>>> d1 < d2
True
>>>

(Don't have a 2.6 or 3 to hand.)

Mind you, it makes even less sense for a regular dict than an
OrderedDict. And it's not like d1.items() < d2.items() is a
huge burden, if that's what you want dictionary comparison to
mean.

-- 
\S

   under construction

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: missing 'xor' Boolean operator

[Jean-Michel Pichavant]

Luis Alberto Zarrabeitia Gomez wrote:

Quoting Jean-Michel Pichavant :

  

Emile van Sebille wrote:


On 7/16/2009 7:04 AM Unknown said...
  

On 2009-07-16, Emile van Sebille  wrote:


daysInAdvance = int(inputVar) or 25
  

I don't get it.  That doesn't work right when inputVar == "0".


Aah, but you didn't get to define right.  :)  For that particular 
example 0 is not a valid response.
  
When I was talking about such error prone form of boolean operations, I 
didn't expect to be right so quickly :p



What do you mean by being "right so quickly", and "error prone" in this context?
I would also ask "Unknown" why he believes that "int(intputVar) or 25" doesn't
work right when inputVar == "0". The only false value that int() may return is
zero, so the "or 25" clause is there only for that case. I can't see then how
you think that is an error.
  


I was saying that using boolean operators with object instead of boolean 
values is error prone, cause no language behaves he same way, and all 
behaviors are conventions difficult to figure out without diving deeply 
into the documentation (or being explained as it happened to me).


I think the initialization trick is an error, because I don't want 
foo(0) to set daysInAdvance to 25. I'll want it to set the attribute to 
0, cause 0 is a valid integer. 0 is a valid integer content, None 
wouldn't be a valid integer content.



JM
--
http://mail.python.org/mailman/listinfo/python-list

Re: ANN: psyco V2

[Bearophile]

Very good, thank you. I'll try it when I can.

Is Psyco3 going to borrow/steal some ideas/code from Unladen Swallow?

The problem I have with Psyco1.6 is that you can't use the normal
profilers to know how much seconds of running time is taken by each
function/method of your code.
Psyco1.6 has a profile() function, but I am not much able to use it
yet.
Can you tell me how to find a sorted list of the running time of all
functions/methods of a Psyco-digested program?

Bye and thank you,
bearophile
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Try... except....Try again?

[Andre Engels]

On Fri, Jul 17, 2009 at 7:31 AM, Xavier Ho wrote:
> I have a simple class that generates prime numbers, using memorisation and
> iteration to generate the next prime number.
>
> In the class, I have defined a function that gives, say, the 5th prime
> number. Or the 1000th, it's still very fast. But the code isn't what I
> really like.
[...]
> My problem is that it's a "While True" loop, which I get a lot of "Don't do
> it!" In the light of making the code better, what could I do?

Where do you get these "Don't do it!"s? As far as I know, "while true"
loops are a very acceptable idiom for programming in Python (and
several other languages).


-- 
André Engels, andreeng...@gmail.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Try... except....Try again?

[Xavier Ho]

On Fri, Jul 17, 2009 at 7:33 PM, Andre Engels  wrote:

>
> Where do you get these "Don't do it!"s? As far as I know, "while true"
> loops are a very acceptable idiom for programming in Python (and
> several other languages).
>

Hey Andre,

Friends, internet, common sense. Also... "while True" has a bad reputation.
But mostly outside influences.

But if it's okay, it's okay by me.. I can't think of an easier way to code
the "try loop" above, anyhow.

Ching-Yun "Xavier" Ho, Technical Artist

Contact Information
Mobile: (+61) 04 3335 4748
Skype ID: SpaXe85
Email: cont...@xavierho.com
Website: http://xavierho.com/
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Override a method but inherit the docstring

[David Stanek]

On Fri, Jul 17, 2009 at 2:58 AM, Peter Otten<__pete...@web.de> wrote:
> Ben Finney wrote:
>
>> Howdy all,
>>
>> The following is a common idiom::
>>
>>     class FooGonk(object):
>>         def frobnicate(self):
>>             """ Frobnicate this gonk. """
>>             basic_implementation(self.wobble)
>>
>>     class BarGonk(FooGonk):
>>         def frobnicate(self):
>>             special_implementation(self.warble)
>>
>> The docstring for ‘FooGonk.frobnicate’ is, intentionally, perfectly
>> applicable to the ‘BarGonk.frobnicate’ method also. Yet in overriding
>> the method, the original docstring is not associated with it.
>>
>> Ideally there would be a way to specify that the docstring should be
>> inherited. The best I can come up with is::
>>
>>     class BarGonk(FooGonk):
>>         def frobnicate(self):
>>             special_implementation(self.warble)
>>         frobnicate.__doc__ = FooGonk.frobnicate.__doc__
>>
>> but that violates DRY (the association between BarGonk and FooGonk is
>> being repeated), puts the docstring assignment awkwardly after the end
>> of the method instead of at the beginning where docstrings normally go,
>> and reads poorly besides.
>>
>> What is the most Pythonic, DRY-adherent, and preferably least-ugly
>> approach to override a method, but have the same docstring on both
>> methods?
>
> Just thinking aloud: Write a patch for pydoc that looks up the base-class
> documentation.
>
> B.f.__doc__ will continue to return None, but
>
> help(B.f) will show something like
>
>    No documentation available for B.f.
>
>    Help for A.f:
>    yadda yadda
>
>
> Of course that might be misleading when A.f and B.f are up to something
> completely different...
>

This should never be the case. It violates LSP and would be very confusing to
readers of the code.


-- 
David
blog: http://www.traceback.org
twitter: http://twitter.com/dstanek
-- 
http://mail.python.org/mailman/listinfo/python-list

setup.py not found

[larry.mart...@gmail.com]

I'm trying to install a package (cx_Oracle) on a mac running 10.5.7.
I've done this on other platforms, but never on a mac. I followed the
instructions given, but when I try and run setup I get:

Apollo:instantclient_10_2 user$ python setup.py build
/System/Library/Frameworks/Python.framework/Versions/2.5/Resources/
Python.app/Contents/MacOS/Python: can't open file 'setup.py': [Errno
2] No such file or directory

Is there something else I have to install first to have this?

Thanks!
-- 
http://mail.python.org/mailman/listinfo/python-list

guessing file type

[Seldon]

Hello,  I need to determine programmatically a file type from its 
content/extension (much like the "file" UNIX command line utility)


I searched for a suitable Python library module, with little luck. Do 
you know something useful ?


Thanks in advance.
--
Seldon
--
http://mail.python.org/mailman/listinfo/python-list

Re: Override a method but inherit the docstring

[David Stanek]

On Fri, Jul 17, 2009 at 3:52 AM, Steven
D'Aprano wrote:
> On Fri, 17 Jul 2009 12:58:48 +1000, Ben Finney wrote:
>
>>> Using a decorator in this manner requires repeating the super class
>>> name.  Perhaps there is a way to get the bases of BarGonk, but I don't
>>> think so, because at the time that the decorator is called, BarGonk is
>>> not yet fully defined.
>>
>> Yes, I tried a few different ways, but within the decorator it seems the
>> function object is quite unaware of what class it is destined for.
>
>
> When the decorator is called, the function object is just a function
> object, not a method, so there is no concept of "what class it is
> destined for".
>
 def dec(func):
> ...     print type(func)
> ...     try:
> ...             print func.im_class
> ...     except:
> ...             print "no im_class"
> ...     return func
> ...
 class Test(object):
> ...     @dec
> ...     def spam(self):
> ...             pass
> ...
> 
> no im_class
 type(Test.spam)
> 
 Test.spam.im_class
> 
>
>
> I suppose you could try to determine what namespace you're currently when
> the class is created, but that's surely going to be fragile and messy.
>

It isn't too bad. I got the idea to use the method's enclosing
scope[1] in my decorator[2] from DecoratorTools. I am working to
remove it because it make me sad, but it does work.

[1] 
http://code.google.com/p/snake-guice/source/browse/snakeguice/decorators.py#51
[2] 
http://code.google.com/p/snake-guice/source/browse/snakeguice/decorators.py#58


-- 
David
blog: http://www.traceback.org
twitter: http://twitter.com/dstanek
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: guessing file type

[Ben Finney]

Seldon  writes:

> Hello,  I need to determine programmatically a file type from its
> content/extension (much like the "file" UNIX command line utility)

The Unix ‘file(1)’ program does its magic with a library called “magic”
and a corresponding database.

> I searched for a suitable Python library module, with little luck. Do
> you know something useful ?

This package http://pypi.python.org/pypi/python-magic> provides
Python bindings to libmagic.

-- 
 \ “Unix is an operating system, OS/2 is half an operating system, |
  `\Windows is a shell, and DOS is a boot partition virus.” —Peter |
_o__)H. Coffin |
Ben Finney
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: guessing file type

[Tim Chase]

Hello,  I need to determine programmatically a file type from its 
content/extension (much like the "file" UNIX command line utility)


I searched for a suitable Python library module, with little luck. Do 
you know something useful ?


Are you looking for something comprehensive?  Or are you just 
looking for particular file-types that your application can 
handle?  I'd start with the python mimetypes library[1] which 
does detection based on extensions (to which you can add your own 
mappings).  For sniffing by content, there are a wide variety of 
document types, I don't know of any pre-existing library.  The 
first couple bytes can often tell you something, but you'd have 
to go digging into the source for "file" to see what it does.


-tkc

[1]
http://docs.python.org/library/mimetypes.html



--
http://mail.python.org/mailman/listinfo/python-list

Re: guessing file type

[VanL]

Seldon wrote:
Hello,  I need to determine programmatically a file type from its 
content/extension (much like the "file" UNIX command line utility)


I searched for a suitable Python library module, with little luck. Do 
you know something useful ?


Python-magic (http://pypi.python.org/pypi/python-magic/0.1) wraps 
libmagic -- the same thing that "file" uses -- but getting it on windows 
is a pain. I have never been able to do it, anyway.


A more cross-platform library with a similar scope is hachoir. Still 
seems to be available from PyPI, but the website seems to have fallen 
off the 'net.


--
http://mail.python.org/mailman/listinfo/python-list

Re: Try... except....Try again?

[MRAB]

Xavier Ho wrote:
I have a simple class that generates prime numbers, using memorisation 
and iteration to generate the next prime number.


In the class, I have defined a function that gives, say, the 5th prime 
number. Or the 1000th, it's still very fast. But the code isn't what I 
really like.


def nPrime(self, n):
"Returns nth prime number, the first one being 2, where n = 0. 
When n = 1, it returns 3."

while True:
try:
return self.primes[n]
except:
self.next()

The next() function generates the next prime, and appends it into 
primes. This way, it keeps trying to append until the nth prime 
requested exist, and returns it.


My problem is that it's a "While True" loop, which I get a lot of "Don't 
do it!" In the light of making the code better, what could I do?



I wouldn't use a bare "except". What if you had written "prims" instead
of "primes"? It would catch AttributeError and produce the next prime, 
repeating until it ran out of memory.


--
http://mail.python.org/mailman/listinfo/python-list

Re: Einstein summation notation (was: question of style)

[Steven D'Aprano]

On Fri, 17 Jul 2009 00:34:00 -0700, Paul Rubin wrote:

> Steven D'Aprano  writes:
>> It is very useful to be able to write e.g.: 
>>
>> if header or body or footer:
>> print assemble_page(header, body, footer)
>>
>> and have empty strings to be equivalent to False.
> 
> Why doesn't assemble_page properly handle the case where header, body,
> and footer are all empty?  That would let you eliminate the if.  "Make
> sure your code 'does nothing' gracefully" (Kernighan and Plauger).

You mean something like this?

def assemble_page(header, body, footer):
if header or body or footer:
do_lots_of_expensive_processing()
else:
do_nothing_gracefully()

Sure, why not?

Whatever way you do it, my point is that it is very useful to perform 
branching tests on non-Boolean objects. Wherever you put the test, being 
able to write:

if header or body or footer:

is a huge improvement over:

if (header != '') or (body != '') or (footer != ''):



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

ANN: Leo 4.6 final released

[Edward K Ream]

Leo 4.6 final is now available at:
http://sourceforge.net/project/showfiles.php?group_id=3458&package_id=29106

Leo is a text editor, data organizer, project manager and much more. See:
http://webpages.charter.net/edreamleo/intro.html

The highlights of Leo 4.6:
--

- Cached external files *greatly* reduces the time to load .leo files.
- Leo now features a modern Qt interface by default.
  Leo's legacy Tk interface can also be used.
- New --config, --file and --gui command-line options.
- Leo tests syntax of .py files when saving them.
- Leo can now open any kind of file into @edit nodes.
- @auto-rst nodes allow easy editing of reStructuredText files.
- Properties of commanders, positions and nodes simplify programming.
- Improved Leo's unit testing framework.
- Leo now requires Python 2.5 or later.
- Dozens of small improvements and bug fixes.

Links:
--
Leo:  http://webpages.charter.net/edreamleo/front.html
Forum:http://groups.google.com/group/leo-editor
Download: http://sourceforge.net/project/showfiles.php?group_id=3458
Bzr:  http://code.launchpad.net/leo-editor/
Quotes:   http://webpages.charter.net/edreamleo/testimonials.html


Edward K. Ream   email:  edream...@yahoo.com
Leo: http://webpages.charter.net/edreamleo/front.html



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: missing 'xor' Boolean operator

[Steven D'Aprano]

On Thu, 16 Jul 2009 15:53:45 +0200, Jean-Michel Pichavant wrote:

> Python has extended the algebra definition of "or" and "and" top any
> type, but it is so unintuitive (I'm no LISP programmer).

I disagree. The Something/Nothing dichotomy is so intuitive to me that I 
would hate to go back to a language that only accepted booleans as 
arguments to `if`.


> I think than 
> using the short-circuiting mechanism of bool operators along with the 
> python tricks is just error prone and may result in bug difficult to 
> spot, unless you are very aware of all python boolean mechanisms.

In other words, if you don't know how Python behaves, you will make 
mistakes.

Of course you will. That applies to *anything* -- if you don't know how 
it works, you will make mistakes.

Given three result codes, where 0 means "no error" and an arbitrary non-
zero integer means some error, it is simple and easy to write:

failed = result_1 or result_2 or result_3

The equivalent:

failed = (result_1 != 0) or (result_2 != 0) or (result_3 != 0)
# or if you prefer:
succeeded = (result_1 == 0) and (result_2 == 0) and (result_3 == 0)

are longer and more difficult to read and easier to get wrong. Even worse 
are tricks like this:

failed = (result_1 + result_2 + result_3) != 0

This obscures the fact that the result codes are flags and makes it seem 
like (flag + flag) is meaningful.



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: guessing file type

[Justin Ezequiel]

On Jul 17, 8:39 pm, VanL  wrote:
> Seldon wrote:
> > Hello,  I need to determine programmatically a file type from its
> > content/extension (much like the "file" UNIX command line utility)
>
> > I searched for a suitable Python library module, with little luck. Do
> > you know something useful ?
>

I've used the magic.py module by "Jason Petrone" before.
http://tinyurl.com/ldho8m
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Try... except....Try again?

[Steven D'Aprano]

Apologies for breaking threading, but the original post hasn't showed up 
on my ISP's news server.

Xavier Ho wrote:
> I have a simple class that generates prime numbers, using memorisation
> and iteration to generate the next prime number.
[...]
> The next() function generates the next prime, and appends it into
> primes. This way, it keeps trying to append until the nth prime
> requested exist, and returns it.
> 
> My problem is that it's a "While True" loop, which I get a lot of
>> "Don't do it!"

There's nothing wrong with "while True" loops. They are useful in two 
situations:

(1) Where you want to loop forever.

(2) Where you want to loop until some complex event takes place, which 
can't easily be written using "while condition".

Neither of these cases holds for your code, so although there's nothing 
wrong with "while True", in your case I'd avoid it.


> In the light of making the code better, what could I do?


(1) Explicitly loop the correct number of times.
(2) Only catch IndexError.

def nPrime(self, n):
for i in xrange(len(self.primes), n+1):
self.next()
return self.primes[n]



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Einstein summation notation (was: question of style)

[Paul Rubin]

Steven D'Aprano  writes:
> def assemble_page(header, body, footer):
> if header or body or footer:
> do_lots_of_expensive_processing()
> else:
> do_nothing_gracefully()

Why should the processing be expensive if all three fields are empty?

> if header or body or footer:
> is a huge improvement over:
> if (header != '') or (body != '') or (footer != ''):

Doesn't really seem any better.  There's always something like

   if any(p != '' for p in [header, body, footer, otherthing1, ...])

if the number of components gets larger.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: missing 'xor' Boolean operator

[Luis Zarrabeitia]

On Friday 17 July 2009 07:06:26 am Jean-Michel Pichavant wrote:
>
> I was saying that using boolean operators with object instead of boolean
> values is error prone, cause no language behaves he same way,

I don't know of many languages that actively promote the duck typing concept, 
are as highly dynamic as python, have the "metaclass" concept, treats almost 
all language concepts as first class citizens (including functions and 
classes), and so on. And don't get me started on assignment (which I consider 
very natural, by the way, but apparently most of the popular languages have 
pretty unnatural assignments).

It is error prone if you are expecting the result to be a bool instead of just
behaving like one (it is certainly unexpected, but you shouldn't be expecting 
to get an instance of certain class anyway, for most of python's operations). 
And it is confusing if you are reading the "int(inputVar) or 25" line and 
have no idea of what it means (but once you know it, it becomes very 
readable, almost plain english).

> and all 
> behaviors are conventions difficult to figure out without diving deeply
> into the documentation (or being explained as it happened to me).

That happens with many new concepts. Welcome to python, I guess... if you are 
willing to shake some of your expectations from previous programming 
languages, you will enjoy it. My [little] experience teaching python tells me 
that "duck typing", "non-enforced encapsulation" and "everything is an 
object" are the hardest to accept for the C# folk at my faculty, but once 
past it, they (the few of them who don't leave the course after that) really 
enjoy the language.

> I think the initialization trick is an error, because I don't want
> foo(0) to set daysInAdvance to 25. I'll want it to set the attribute to
> 0, cause 0 is a valid integer. 0 is a valid integer content, None
> wouldn't be a valid integer content.

Well, that wouldn't be a problem with "or", but with the programmer.

The exact same behaviour could be obtained with

if int(inputValue) == 0:
inputValue = 25

and no "or" involved.

However, using only

inputValue = inputValue or 25

could have been an error if you only wanted 25 in case inputValue is None.

(the "or trick" implies certain trust in that the object you have in hand has 
a reasonable definition of truth value).

-- 
Luis Zarrabeitia (aka Kyrie)
Fac. de Matemática y Computación, UH.
http://profesores.matcom.uh.cu/~kyrie
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: missing 'xor' Boolean operator

[Jean-Michel Pichavant]

Steven D'Aprano wrote:

On Thu, 16 Jul 2009 15:53:45 +0200, Jean-Michel Pichavant wrote:


Given three result codes, where 0 means "no error" and an arbitrary non-
zero integer means some error, it is simple and easy to write:

failed = result_1 or result_2 or result_3

The equivalent:

failed = (result_1 != 0) or (result_2 != 0) or (result_3 != 0)
# or if you prefer:
succeeded = (result_1 == 0) and (result_2 == 0) and (result_3 == 0)

  

[snip]

This is, I guess, where we disagree. I find the second proposal less 
error prone, and universally understandable unlike the first one. It may 
be verbose, it may look even lame to some people, but in the end this is 
perfectly reliable, because you manipulate only False or True within the 
boolean operations.


The first form does not clearly show what is the failed criteria. It 
just happens by coincidence that in this case the failed criteria 
matches the Nothingness of result_1, result_2, result_3. What if results 
may be 'OK' or 'KO'.


failed = result_1 or result_2 or result_3
won't work.

failed = (result_1 =='KO') or (result_2 =='KO') or (result_3 =='KO') is 
lame but reliable.



JM
--
http://mail.python.org/mailman/listinfo/python-list

Try... except....Try again?

[Xavier Ho]

oops, wrong address.

When will reply-to tag appear on the Python mailing list? =/

Good idea, Steven and MRAB.

I changed my code to the following:

def nPrime(self, n):
"Returns nth prime number, the first one being 2, where n = 0. When
n = 1, it returns 3."
for x in range(n+2):
try:
return self.primes[n]
except IndexError:
self.next()

And it's definitely better. Thanks a ton guys.

(n+2 because after looping the maximum number of times, it needs to try
returning the value again.)

It works just as well as the previous version, but slightly safer on my
part... and I can adapt this in my future coding habits.

Thanks again.


Ching-Yun "Xavier" Ho, Technical Artist

Contact Information
Mobile: (+61) 04 3335 4748
Skype ID: SpaXe85
Email: cont...@xavierho.com
Website: http://xavierho.com/
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: missing 'xor' Boolean operator

[Ethan Furman]

Jean-Michel Pichavant wrote:

Steven D'Aprano wrote:


On Thu, 16 Jul 2009 15:53:45 +0200, Jean-Michel Pichavant wrote:


Given three result codes, where 0 means "no error" and an arbitrary non-
zero integer means some error, it is simple and easy to write:

failed = result_1 or result_2 or result_3

The equivalent:

failed = (result_1 != 0) or (result_2 != 0) or (result_3 != 0)
# or if you prefer:
succeeded = (result_1 == 0) and (result_2 == 0) and (result_3 == 0)

  


[snip]

This is, I guess, where we disagree. I find the second proposal less 
error prone, and universally understandable unlike the first one. 


Careful!  The very few (if any) things in this world that can be 
considered "universally understandable" do *not* include any programming 
language!  :)


It may 
be verbose, it may look even lame to some people, but in the end this is 
perfectly reliable, because you manipulate only False or True within the 
boolean operations.


The first form does not clearly show what is the failed criteria. It 
just happens by coincidence that in this case the failed criteria 
matches the Nothingness of result_1, result_2, result_3. What if results 
may be 'OK' or 'KO'.


As the programmer, particularly a Python programmer, you should be 
taking advantage of Python's strengths, of which this is one.  If, as 
you say, some function uses a "something" value to indicate an 
undesirable result, then you have to use something akin to your last 
statement.



~Ethan~



failed = result_1 or result_2 or result_3
won't work.

failed = (result_1 =='KO') or (result_2 =='KO') or (result_3 =='KO') is 
lame but reliable.



JM


--
http://mail.python.org/mailman/listinfo/python-list

Re: Einstein summation notation (was: question of style)

[Steven D'Aprano]

On Fri, 17 Jul 2009 07:12:51 -0700, Paul Rubin wrote:

> Steven D'Aprano  writes:
>> def assemble_page(header, body, footer):
>> if header or body or footer:
>> do_lots_of_expensive_processing()
>> else:
>> do_nothing_gracefully()
> 
> Why should the processing be expensive if all three fields are empty?

Since I haven't specified an implementation for assemble_page, it could 
be doing *anything*. Perhaps it has to talk to a remote database over a 
slow link, perhaps it generates 300 lines of really inefficient HTML code 
with no content, perhaps it sends a print job to a printer which then 
warms up, cleans the print heads, and ejects a blank piece of paper.

Why does it matter? The example is about the `if` test, not about the 
function assemble_page().



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Try... except....Try again?

[Jack Diederich]

On Fri, Jul 17, 2009 at 10:35 AM, Xavier Ho wrote:

> I changed my code to the following:
>
>     def nPrime(self, n):
>     "Returns nth prime number, the first one being 2, where n = 0. When
> n = 1, it returns 3."
>     for x in range(n+2):
>     try:
>     return self.primes[n]
>     except IndexError:
>     self.next()
>
> And it's definitely better. Thanks a ton guys.
>
> (n+2 because after looping the maximum number of times, it needs to try
> returning the value again.)

def nPrime(self, n):
while len(self.primes) < n:
 self.next()
return self.primes[n]

If you use an off-the-shelf prime number generator fucntion[1] that
returns consecutive primes the method collapses into a simple
function.

def nPrime(n, primes=[], next_prime=eratosthenes().next):
while len(primes) < n:
primes.append(next_prime())
return primes[n]

-Jack

[1] 
http://www.catonmat.net/blog/solving-google-treasure-hunt-prime-number-problem-four/#comment-3075
 There is a deque implementation that is faster still but I can't
find a link.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Einstein summation notation

[MRAB]

koranthala wrote:

That test was designed to treat None as a boolean False, without
noticing that numeric 0 is also treated as False and could make the
test do the wrong thing.  This is an extremely common type of error.


Actually, I felt that 0 not being considered False would be a better
option.
I had lot of instances where 0 is a valid value and always I had to
put checks specifically for that.
For me, None and False should be equal to False in if checks, every
thing else being considered True.

Can someone let me know why 0 is considered equal to False?
There should be real good reasons for it, only that I cannot think of
one.


Python did always have True and False.
--
http://mail.python.org/mailman/listinfo/python-list

Re: no return value for threading.Condition.wait(timeout)?

[Gabriel Rossetti]

Piet van Oostrum wrote:

Gabriel Rossetti  (GR) wrote:



  

GR> I have a 1-1 relation, I have a thread reading msgs from a network socket
GR> and a method waiting for an answer to arrive (see my thread
GR> "Threading.Condition problem"). I would like to be able to have a timeout
GR> as to not block forever, so the idea is to check if I returned because of a
GR> timeout or not.



So that case is easy I think. After the wait just check if the answer
has arrived.
  
The problem is other answers could arrive that are not the one I want. 
Maybe I could check if I got the answer, if not then check how much time 
has elapsed, and if it's grater or equal to the timeout consider it a 
timeout. I still think a return value would be better though.


--
http://mail.python.org/mailman/listinfo/python-list

Re: Einstein summation notation

[Jerry Hill]

On Fri, Jul 17, 2009 at 11:09 AM, MRAB wrote:

> Python did always have True and False.

Only if "always" means "since python 2.2.1".

See: http://www.python.org/doc/2.3/whatsnew/section-bool.html and
http://www.python.org/dev/peps/pep-0285/ for details.

-- 
Jerry
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: missing 'xor' Boolean operator

[Steven D'Aprano]

On Fri, 17 Jul 2009 16:34:57 +0200, Jean-Michel Pichavant wrote:

> Steven D'Aprano wrote:
>> On Thu, 16 Jul 2009 15:53:45 +0200, Jean-Michel Pichavant wrote:
>>
>>
>> Given three result codes, where 0 means "no error" and an arbitrary
>> non- zero integer means some error, it is simple and easy to write:
>>
>> failed = result_1 or result_2 or result_3
>>
>> The equivalent:
>>
>> failed = (result_1 != 0) or (result_2 != 0) or (result_3 != 0) # or if
>> you prefer:
>> succeeded = (result_1 == 0) and (result_2 == 0) and (result_3 == 0)
>>
>>
> [snip]
> 
> This is, I guess, where we disagree. I find the second proposal less
> error prone, and universally understandable unlike the first one. It may
> be verbose, it may look even lame to some people, but in the end this is
> perfectly reliable, because you manipulate only False or True within the
> boolean operations.

Because it is verbose, it is more error-prone. The more code you have to 
write, the more opportunities you have to make mistakes.

(This holds up to a point, beyond which being more and more terse also 
leads to more errors.)

Boolean algebra is notorious for programmer errors. The more complicated 
the boolean expression, the more often programmers get it wrong. The more 
book-keeping they have to do, the easier it is to get it wrong. All those 
(result != 0) comparisons are mere book-keeping, they don't add anything 
to the code except to force the flag to be True or False.



> The first form does not clearly show what is the failed criteria.

Of course it does: at least one of the three result codes is non-zero. As 
a bonus, failed will be assigned the first non-zero result code (if any).

Your preferred form, on the other hand, folds all the possible error 
codes into True, throwing away useful information:

>>> result_1 = 0  # success
>>> result_2 = 0
>>> result_3 = 15  # failure
>>> failure = result_1 or result_2 or result_3
>>> failure
15
>>> failure = (result_1 != 0) or (result_2 != 0) or (result_3 != 0)
>>> failure
True



> It
> just happens by coincidence that in this case the failed criteria
> matches the Nothingness of result_1, result_2, result_3. What if results
> may be 'OK' or 'KO'.

Obviously the test you write depends on the data you have to work with. 
Do you think that's surprising?


> failed = result_1 or result_2 or result_3 won't work.

Not in the example you gave, no. Although if I had to work with lots and 
lots of strings of the form 'OK' and 'KO', I'd consider sub-classing 
string:

>>> class OKString(str):
... def __nonzero__(self):
... if self == 'OK': return True
... elif self == 'KO': return False
... else:
... raise ValueError('invalid result code')
...
>>>
>>> a = OKString('OK')
>>> b = OKString('KO')
>>> c = OKString('KO')
>>> if a and b and c:
... print "Success!"
... else:  print "Failed!"
...
Failed!


(I wouldn't bother if I only needed one or two such tests, but if I had 
lots of them, I'd consider it.)

 
> failed = (result_1 =='KO') or (result_2 =='KO') or (result_3 =='KO') is
> lame but reliable.


Yes, it's reliably complicated, reliably easy to get wrong, reliably 
harder to read, and it reliably throws away information.

But apart from those disadvantages, it does the job you want it to do.


-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Einstein summation notation

[Steven D'Aprano]

On Fri, 17 Jul 2009 16:09:03 +0100, MRAB wrote:

> Python did always have True and False.

$ python1.5
Python 1.5.2 (#1, Apr  1 2009, 22:55:54)  [GCC 4.1.2 20070925 (Red Hat 
4.1.2-27)] on linux2
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> True, False
Traceback (innermost last):
  File "", line 1, in ?
NameError: True



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

'del' function for Dictionary

[mayank gupta]

Hi all,

I wanted to know whether there is a more efficient way to delete an entry
from a dictionary (instead of using the 'del' function), because after
analyzing the time taken by the code, it seems to me that the 'del' function
takes most of the time. I might be incorrect as well.
Kindly help me in this regard.

Cheers,
Mayank

-- 
I luv to walk in rain bcoz no one can see me crying
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Try... except....Try again?

[Xavier Ho]

On Fri, Jul 17, 2009 at 11:02 PM, Jack Diederich  wrote:

>
> If you use an off-the-shelf prime number generator fucntion[1] that
> returns consecutive primes the method collapses into a simple
> function.
>
> def nPrime(n, primes=[], next_prime=eratosthenes().next):
>while len(primes) < n:
>primes.append(next_prime())
>return primes[n]
>
> -Jack
>
> [1]
> http://www.catonmat.net/blog/solving-google-treasure-hunt-prime-number-problem-four/#comment-3075
> There is a deque implementation that is faster still but I can't
> find a link.


Jack, thanks for the link, I'll definitely look into it.

The function is currently implemented in a class I wrote myself, which is a
generator class that does exactly the same thing in the next() function. I
didn't post the code here because it wasn't in the scope of this thread, but
I'm sure others have done a lot better and much more efficient algorithms.

The nPrime method uses the nature of the iteration and checks if a value
already exists (ie generated before) and return that value; otherwise it
iterates again. This way, I don't have to iterate from n=2 and on every
time, which makes generation time linear.

However, I just gave the link's code a quick try. It was at least 10x times
faster than my code to generate 10 prime numbers - I'll have a closer
look.

Thanks a great deal.

Ching-Yun "Xavier" Ho, Technical Artist

Contact Information
Mobile: (+61) 04 3335 4748
Skype ID: SpaXe85
Email: cont...@xavierho.com
Website: http://xavierho.com/
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: 'del' function for Dictionary

[Michiel Overtoom]

mayank gupta wrote:

after analyzing the time taken by the code,

What code?


--
http://mail.python.org/mailman/listinfo/python-list

Find first matching substring

[Eloff]

Almost every time I've had to do parsing of text over the last 5 years
I've needed this function:

def find_first(s, subs, start=None, end=None):
results = [s.find(sub, start, end) for sub in subs]
results = [r for r in results if r != -1]
if results:
return min(results)

return -1

It finds the first matching substring in the target string, if there
is more than one match, it returns the position of the match that
occurs at the lowest index in the string.

Has anyone else had problems where they could have applied this
function?

It seems to me that python's find (and rfind, index, rindex) could be
modified (the same way that startswith and endswith have been) to
behave this way if a tuple were passed. Do other's agree that this
would be desirable?

Thanks,
Dan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Einstein summation notation

[Ethan Furman]

Paul Rubin wrote:

Steven D'Aprano  writes:


def assemble_page(header, body, footer):
   if header or body or footer:
   do_lots_of_expensive_processing()
   else:
   do_nothing_gracefully()



Why should the processing be expensive if all three fields are empty?



   if header or body or footer:
is a huge improvement over:
   if (header != '') or (body != '') or (footer != ''):



Doesn't really seem any better.  There's always something like

   if any(p != '' for p in [header, body, footer, otherthing1, ...])

if the number of components gets larger.


Or if any(p for p in [header, body, footer, whatever, ...])

Which is even nicer!  :)  Particularly if header, et al, are some more 
complicated objects which don't easily generate strings, but do easily 
know whether they are Something or Nothing.


I suppose some of these discussions are based more on one's personal 
requirements (using several languages, just a couple, or just Python), 
and whether one wants to achieve Mastery of Python, or just be 
Proficient.  Neither choice is wrong, and wanting Mastery, of course, 
does not mean not wanting improvements, etc.  It does, however, require 
a deep understanding of not just the mechanics, but the philosophy of 
the language.


Mind you, I am nowhere near Mastery just yet (okay, okay, I have a 
_long_ way to go! ;), but I love the philosophy of Python, it's 
simplicity, and the *entirety* of the Zen.


~Ethan~
--
http://mail.python.org/mailman/listinfo/python-list

Re: Einstein summation notation (was: question of style)

[Paul Rubin]

Steven D'Aprano  writes:
> Since I haven't specified an implementation for assemble_page, it could 
> be doing *anything*. Perhaps it has to talk to a remote database over a 
> slow link, perhaps it generates 300 lines of really inefficient HTML code 
> with no content, perhaps it sends a print job to a printer which then 
> warms up, cleans the print heads, and ejects a blank piece of paper.

That sounds like a code smell to me.  If the function is supposed to
print a piece of paper when any of those strings are nonempty, it
should also print one when they are all empty.  Of the different
values a string can have, 'foo', 'bar', and '' are all perfectly good
values and should all be treated the same way.  

If you want an absent string to work differently than a present one,
that's separate from empty vs nonempty, and overloading the empty
string to mean "absent" is another antipattern.  Maybe what you really
want is a list of page constituents that are present:

   page_parts = [header, body, footer]

If all the constituents are absent, then page_parts is the empty list,
so you would just test for page_parts being empty.

> Why does it matter? The example is about the `if` test, not about the 
> function assemble_page().

Because of the notion that the code should "do nothing" gracefully.
The presence of that 'if' test is ungraceful, so its better to look
for ways to eliminate it than slickify it.
-- 
http://mail.python.org/mailman/listinfo/python-list

Question regarding style/design..

[Wells Oliver]

Sometimes I see relatively small application, generally task scripts,
written as essentially a list of statements. Other times, I see them neatly
divided into functions and then the "if __name__ == '__main__':" convention.
Is there a preference? Is there an... application scope such that the
preference shifts from the former to the latter? I understand the use of the
__name__ == 'main' convention for building unit tests, but I'm mixed on
using it in scripts/small applications.
Thanks for any thoughts!

-- 
Wells Oliver
we...@submute.net
-- 
http://mail.python.org/mailman/listinfo/python-list

A multi-threaded file searcher for processing large text files

[Mahmoud Abdelkader]

I'm building a threaded file searcher that uses some of Fredrik Lundh's (
http://effbot.org/zone/wide-finder.htm) suggestions for parsing text very
quickly in pure python, as I have about a 10GB log file to parse every day.
A naiive approach would be to just parse the 1MB chunks, add the results
into a list, and just traverse that list.

I want to take this idea a bit further. I want to process results as they're
being found. A great way to implement this is to use the Queue class that
python provides. My idea is to exploit the iterator protocol to have it
block until a result is found, if any, and return the result until we're
finished parsing the file then we can raise StopIteration.

My idea is sort of similar to a producer / consumer, but it follows
something of this idiom:
  producer produces the file chunks
  consumer consumes the file chunks
 -> consumer parsers the file chunks and produces results
  class waits on the production of the original consumer and processes it as
they come.

I am having a bit of trouble with the concurrency, but I'm using this as an
exercise to understand how concurrency works from a broader scale. I am not
trying to get into a debate of whether this is really needed or a
python-concurrency debate:)

Without further ado, my class is as follows:

class ThreadedFileSearcher(object):
def __init__(self, filename, rx_pat, blocking_timeout = 10):
self.name = filename
self.pattern = rx_pat
self.blocking_timeout = blocking_timeout

#need to find a better way to do this with more threads that can
return
#stable results (aka chunks are in order)
self._thread_count = 1

#the queues
self._results = Queue.Queue()
self._queue = Queue.Queue()

#returns the get_chunk() implementation
self._engine = LogParsingEngine(filename)

#start acquiring file offsets for the file
#as daemon threads
self._initialize_worker_threads(self._prime_queue)

#experimental...should probably be some type of conditional variable
self._stop_processing = False

def __iter__(self):
#start the worker threads
self._initialize_worker_threads(self._target_worker)
return self.next()

def _initialize_worker_threads(self, callback):
#should really use just one thread
for i in xrange(self._thread_count):
t = threading.Thread(target=callback)
t.setDaemon(True)
t.start()

def _prime_queue(self):
"""put code chunk offsets on the queue"""
#get_chunks() just returns 1MB offsets in the file
for chunk in self._engine.get_chunks():
self._queue.put(chunk)

def _target_worker(self):
"""code chunk to parse queue"""
#loop infinitely
while True:
try:
#get next chunk offset from the queue
start_pos, bytes_to_read = self._queue.get(
  timeout=self.blocking_timeout
   )
except (TypeError, Queue.Empty):
#None was returned from the .get()
#this will throw a TypeError as it tries to unpack None
#or the Queue was empty
self._stop_processing = True
#exit loop
break

#process the cunk here
f = open(self.name, 'r')
f.seek(start_pos)
#find all matching lines in the chunk
for chunk in self.pattern.findall(f.read(bytes_to_read)):
#an non-overlapping matches of self.pattern
#placed on the queue as a string
self._results.put(chunk)
f.close()

#done!
self._queue.task_done()

def next(self):
while True:
try:
#wait for the results to be put on
matchedlist =
self._results.get(timeout=self.blocking_timeout)
except Queue.Empty:
#if the worker thread finished
if self._stop_processing:
raise StopIteration
else:
self._results.task_done()
yield matchedlist

To use the following class, I wanted to have some kind of interface like
this:

regex = re.compile("-{3}Processing..-{3}") #---Processing..---
f = ThreadedFileSearcher("LogFile.log", regex)
for line in f:
#guaranteed to be a line that matches regex
#process something...
print line


I am looking for some suggestions, comments, and better ways to modify this.


One thing someone will realize when using this class is that the initial
parsing will be incredibly quick, but if the blocking_timeout is set to 10,
then there will be a 10second wait at the end to test if the worker threads
should set the stop conditions. A glaring hole leading from this is if the
blocking_timeout is set 

Re: missing 'xor' Boolean operator

[Emile van Sebille]

On 7/17/2009 7:34 AM Jean-Michel Pichavant said...

Steven D'Aprano wrote:

On Thu, 16 Jul 2009 15:53:45 +0200, Jean-Michel Pichavant wrote:


Given three result codes, where 0 means "no error" and an arbitrary non-
zero integer means some error, it is simple and easy to write:

failed = result_1 or result_2 or result_3

The equivalent:

failed = (result_1 != 0) or (result_2 != 0) or (result_3 != 0)
# or if you prefer:
succeeded = (result_1 == 0) and (result_2 == 0) and (result_3 == 0)

  

[snip]

This is, I guess, where we disagree. I find the second proposal less 
error prone, and universally understandable unlike the first one. It may 
be verbose, it may look even lame to some people, but in the end this is 
perfectly reliable, because you manipulate only False or True within the 
boolean operations.


The first form does not clearly show what is the failed criteria. It 
just happens by coincidence 


No -- it happens by design because the premise is 'where 0 means "no 
error" and an arbitrary non-zero integer means some error'.


that in this case the failed criteria 
matches the Nothingness of result_1, result_2, result_3. What if results 
may be 'OK' or 'KO'.


Which by definition won't happen for the example cited...



failed = result_1 or result_2 or result_3
won't work.


... so you certainly wouldn't write a test that couldn't properly 
determine success or failure.




failed = (result_1 =='KO') or (result_2 =='KO') or (result_3 =='KO') is 
lame but reliable.


In this case I'd write something that meets the specs of the problem 
your addressing...


failed = 'KO' in (result_1,result_2,result_3)

Emile


--
http://mail.python.org/mailman/listinfo/python-list

Regular exprssion for IRC messages

[nohics nohics]

Hello,

When an IRC client receive a messages, this message in in a special format
defined here: Message format in pseudo
BNF(
http://www.irchelp.org/irchelp/rfc/chapter2.html#c2_3_1 ).

For example we can have:

:mynickname!n=myusern...@41.238.129.121 JOIN :#AnIrcChannel
:mynickname!n=myusern...@41.238.129.121 PRIVMSG #AnIrcChannel :Hello here
:mynickname!n=myusern...@41.238.129.121 NICK :TheNewNickName
:holmes.freenode.net 372 UnNickName :- a message here ...

I want to transforme the message format defined in the irc protocole in the
last link, to a regular expression to get the nickname, the username, the
host, the commad, it's arguments if exist, and the message after ":"

I'm programming in python and I have tried this regular expression:

^(:(\w+)(![^ \r\n]+)?(\...@[^ \r\n]+)? )?([a-zA-Z]+|\d\d\d)(( [^: \r\n][^
\r\n]*)+)( :[^\r\n]*)?

Python code:

nicknamePattern = re.compile(r'^(:(\w+)(![^ \r\n]+)?(\...@[^ \r\n]+)?
)?([a-zA-Z]+|\d\d\d)(( [^: \r\n][^ \r\n]*)+)( :[^\r\n]*)?')
matches = nicknamePattern.search(data)
if matches != None:  print matches.groups()

But it doesn't seems to work all the time. So can you help me to make the
correct regular expression please ?

You can see some example of messages working or not with this pattren (when
I try to connect to an irc server):
http://www.developpez.net/forums/d779736/php/langage/regex/format-message-pseudo-bnf-expression-reguliere/#post4494010
Thanks.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Question regarding style/design..

[Tim Chase]

Sometimes I see relatively small application, generally task scripts,
written as essentially a list of statements. Other times, I see them neatly
divided into functions and then the "if __name__ == '__main__':" convention.
Is there a preference? Is there an... application scope such that the
preference shifts from the former to the latter? I understand the use of the
__name__ == 'main' convention for building unit tests, but I'm mixed on
using it in scripts/small applications.


This may circle around to the "is Python a scripting language" 
bruhaha a while back.  I tend to skip the __name__ == '__main__' 
in what I'd consider scripts (one-file hacks to solve an 
immediate problem, often disposable or one-use wonders).  However 
for non-scripts (which I'd define as code that has more than one 
source-code file I've written or that I'd have to maintain) and 
for modules, I tend to put in the __name__ guard.


-tkc




--
http://mail.python.org/mailman/listinfo/python-list

Re: ctype performance benchmark

[Wai Yip]

I started with ctypes because it is the battery included with the
Python standard library. My code is very fluid and I'm looking for
easy opportunity to optimize it. One example is to find the longest
common prefix among two strings. Right now I am comparing it character
by character with pure Python. It seems like an ideal low hanging
fruit. With ctype I can rewriting a few lines of Python into a few
lines of C. All the tools are available and no third party library is
needed.

It turned out the performance fail to match pure Python. This function
is called million of times in an inner loop. The overhead overwhelm
any performance gain with C. Eventually I have found success grouping
the data into larger chunk for each API call. This is what I
originally planned to do anyway. I am only sharing my experience here
that doing fine grained ctype function call has its limitation.

I have looked into Pyrex before and I was quite confused by the fusion
of C and Python langauage. Perhaps it is time for me to give it a
second look. I just heard of Cython now and it also look interesting.
I think it is helpful for both project to articulate more clearly what
they are, how they achieve the performance gain and to give more
guidance on good use cases. On the other hand, perhaps it is just me
are confused because I don't know enough of the Python internal.

Wai Yip
-- 
http://mail.python.org/mailman/listinfo/python-list

Installing python-gnutls on Windows

[IronyOfLife]

Hi,

I'm new to python and would like to request your help installing
python-gnutls on windows. Are there binaries available which can be
installed ?

Looks like the setup.py file which came with the package is for
Debian. I tried modifying it for windows but am still getting errors.

get_options() is not getting the include directories and libraries so
I harcoded the include directories. I'm getting the following error
now:
//---
C:\Development\python\python-gnutls-1.1.8>python setup.py build --
compiler=mingw32
running build
running build_py
running build_ext
building 'gnutls.library._gnutls_init' extension
Traceback (most recent call last):
  File "setup.py", line 86, in 
libraries='-lgnutls -lgnutls-extra')])
  File "C:\Python26\lib\distutils\core.py", line 152, in setup
dist.run_commands()
  File "C:\Python26\lib\distutils\dist.py", line 975, in run_commands
self.run_command(cmd)

  

  File "C:\Python26\lib\distutils\command\build_ext.py", line 471, in
build_exte
nsions
self.build_extension(ext)

  ...

depends=ext.depends)
  File "C:\Python26\lib\distutils\ccompiler.py", line 689, in compile
depends, extra_postargs)
  File "C:\Python26\lib\distutils\ccompiler.py", line 363, in
_setup_compile
"'include_dirs' (if supplied) must be a list of strings"
TypeError: 'include_dirs' (if supplied) must be a list of strings
//-

I would really appreciate your help

Thanks

Ramg
-- 
http://mail.python.org/mailman/listinfo/python-list

getopt code NameError exception on logTail.py

[LoD MoD]

I am having trouble extending my option parsing.

Any help is appreciated:


import sys

import getopt

import time


def tail(file):

while 1:

where = file.tell()

line = file.readline()

if not line:

time.sleep(1)

file.seek(where)

else:

print line, # already has newline


def main():



# parse command line options

try:

opts, args = getopt.getopt(sys.argv[1:], "hf:", ["help", "filename="
])

except getopt.error, msg:

print msg

print "for help use --help"

sys.exit(2)

# process options

for o, a in opts:

if o in ("-h", "--help"):

print __doc__

sys.exit(0)

if o in ("-f", "--filename"):

print "Parsing F argument"

file = open(filename, 'r')

print file






# process arguments

for arg in args:

process(arg) # process() is defined elsewhere


if __name__ == "__main__":

main()

Yields this error:

localhost:src gsery$ python logTail.py  /var/log/system.log
Traceback (most recent call last):
  File "logTail.py", line 52, in 
main()
  File "logTail.py", line 49, in main
process(arg) # process() is defined elsewhere
NameError: global name 'process' is not defined
-- 
http://mail.python.org/mailman/listinfo/python-list

Propagate import for all modules in a package.

[Phil]

I'm really new to Python and I am absolutely stumped trying to figure
this out. I have searched plenty, but I am either searching for the
wrong keywords or this isn't possible.

What I want to do is have one import be global for the entire package.
Here is an example...


  __init__.py
  module1.py
  module2.py
  ...
  moduleN.py

I was thinking that I could just, for example, 'from datetime import
datetime' in __init__.py and have the ability to use 'datetime'
anywhere in any of the modules in 'package'.

This didn't work for me. Did I just do something wrong? Is what I am
trying to do possible?

Thanks in advance.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: UnicodeEncodeError: 'ascii' codec can't encode character u'\xb7' in position 13: ordinal not in range(128)

[Scott David Daniels]

akhil1988 wrote:
>

Nobody-38 wrote:

On Thu, 16 Jul 2009 15:43:37 -0700, akhil1988 wrote:

...

In Python 3 you can't decode strings because they are Unicode strings
and it doesn't make sense to decode a Unicode string. You can only
decode encoded things which are byte strings. So you are mixing up byte
strings and Unicode strings.

... I read a byte string from sys.stdin which needs to converted to unicode
string for further processing.

In 3.x, sys.stdin (stdout, stderr) are text streams, which means that they
read and write Unicode strings, not byte strings.


I cannot just remove the decode statement and proceed?
This is it what it looks like:
for line in sys.stdin:
line = line.decode('utf-8').strip()
if line == '': #do something here

If I remove the decode statement, line == '' never gets true. 

Did you inadvertently remove the strip() as well?

... unintentionally I removed strip()
I get this error now:
 File "./temp.py", line 488, in 
main()
  File "./temp.py", line 475, in main
for line in sys.stdin:
  File "/usr/local/lib/python3.1/codecs.py", line 300, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-2: invalid
data


(1) Do not top post.
(2) Try to fully understand the problem and proposed solution, rather
than trying to get people to tell you just enough to get your code
going.
(3) The only way sys.stdin can possibly return unicode is to do some
decoding of its own.  your job is to make sure it uses the correct
decoding.  So, if you know your source is always utf-8, try
something like:

import sys
import io

sys.stdin = io.TextIOWrapper(sys.stdin.detach(), encoding='utf8')

for line in sys.stdin:
line = line.strip()
if line == '':
#do something here


--Scott David Daniels
scott.dani...@acm.org
--
http://mail.python.org/mailman/listinfo/python-list

Re: Einstein summation notation

[Matthew Barnett]

Steven D'Aprano wrote:

On Fri, 17 Jul 2009 16:09:03 +0100, MRAB wrote:


Python did always have True and False.


Oops! I meant "didn't", of course.


$ python1.5
Python 1.5.2 (#1, Apr  1 2009, 22:55:54)  [GCC 4.1.2 20070925 (Red Hat 
4.1.2-27)] on linux2

Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam

True, False

Traceback (innermost last):
  File "", line 1, in ?
NameError: True


--
http://mail.python.org/mailman/listinfo/python-list

Re: Question regarding style/design..

[Robert Kern]

On 2009-07-17 12:11, Tim Chase wrote:

Sometimes I see relatively small application, generally task scripts,
written as essentially a list of statements. Other times, I see them
neatly
divided into functions and then the "if __name__ == '__main__':"
convention.
Is there a preference? Is there an... application scope such that the
preference shifts from the former to the latter? I understand the use
of the
__name__ == 'main' convention for building unit tests, but I'm mixed on
using it in scripts/small applications.


This may circle around to the "is Python a scripting language" bruhaha a
while back. I tend to skip the __name__ == '__main__' in what I'd
consider scripts (one-file hacks to solve an immediate problem, often
disposable or one-use wonders). However for non-scripts (which I'd
define as code that has more than one source-code file I've written or
that I'd have to maintain) and for modules, I tend to put in the
__name__ guard.


I definitely suggest always using the __name__ guard no matter what. Habits are 
important, and it's usually more of a waste of time trying to determine if you 
are writing a "script" or a "program" than it is to just write the clause all 
the time. One of the ex-DivMod guys (Glyph?) has a good blog post on the subject 
of writing docstrings and the Suzuki Method of violin; however, I cannot find 
it. With a good programmer's editor that can do macros or "snippets", this 
particular habit can be basically cost-free.


There are also technical limitations to be aware of. Even if you don't think you 
will ever import the script, other tools might. For example, on Windows, 
multiprocessing needs to import the main module of the parent process in order 
to do some of its operations. If you do not have a __name__ guard, you will go 
into an infinite regress as your subprocesses execute the main code and create 
subsubprocesses until you finally manage to kill them all. Extra fun if it is a 
GUI and you have to click all of the windows closed.


Don't waste half a day trying to figure out why your script mysteriously doesn't 
work. Learn from my mistakes. :-)


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

--
http://mail.python.org/mailman/listinfo/python-list

Re: Propagate import for all modules in a package.

[Diez B. Roggisch]

Phil schrieb:

I'm really new to Python and I am absolutely stumped trying to figure
this out. I have searched plenty, but I am either searching for the
wrong keywords or this isn't possible.

What I want to do is have one import be global for the entire package.
Here is an example...


  __init__.py
  module1.py
  module2.py
  ...
  moduleN.py

I was thinking that I could just, for example, 'from datetime import
datetime' in __init__.py and have the ability to use 'datetime'
anywhere in any of the modules in 'package'.

This didn't work for me. Did I just do something wrong? Is what I am
trying to do possible?


Each module has it's own namespace, there is no real global one. So 
there is no obvious and recommended way to do what you want.


You can do it through, by stuffing names into the __builtins__-module, 
which acts as "namespace of last resort"



from datetime import datetime
__builtins__['datetime'] = datetime

However, this is strongly discouraged - you can easily get 
name-conflicts, and undefined behavior in your modules if you rely on this.


Diez
--
http://mail.python.org/mailman/listinfo/python-list

Re: Find first matching substring

[MRAB]

Eloff wrote:
> Almost every time I've had to do parsing of text over the last 5 years
> I've needed this function:
>
> def find_first(s, subs, start=None, end=None):
> results = [s.find(sub, start, end) for sub in subs]
> results = [r for r in results if r != -1]
> if results:
> return min(results)
>
> return -1
>
> It finds the first matching substring in the target string, if there
> is more than one match, it returns the position of the match that
> occurs at the lowest index in the string.
>
One possible optimisation for your code is to note that if you find that
one of the substrings starts at a certain position then you're not
interested in any subsequent substring which might start at or after
that position, so you could reduce the search space for each substring
found.

Has anyone else had problems where they could have applied this 
function?


It seems to me that python's find (and rfind, index, rindex) could be
 modified (the same way that startswith and endswith have been) to 
behave this way if a tuple were passed. Do other's agree that this 
would be desirable?



Possibly. I think that allowing a tuple in the partition and rpartition
methods might also be useful.

--
http://mail.python.org/mailman/listinfo/python-list

Generator Expressions and CSV

[Zaki]

Hey all,

I'm really new to Python and this may seem like a really dumb
question, but basically, I wrote a script to do the following, however
the processing time/memory usage is not what I'd like it to be. Any
suggestions?


Outline:
1. Read tab delim files from a directory, files are of 3 types:
install, update, and q. All 3 types contain ID values that are the
only part of interest.
2. Using set() and set.add(), generate a list of unique IDs from
install and update files.
3. Using the set created in (2), check the q files to see if there are
matches for IDs. Keep all matches, and add any non matches (which only
occur once in the q file) to a queue of lines to be removed from teh q
files.
4. Remove the lines in the q for each file. (I haven't quite written
the code for this, but I was going to implement this using csv.writer
and rewriting all the lines in the file except for the ones in the
removal queue).

Now, I've tried running this and it takes much longer than I'd like. I
was wondering if there might be a better way to do things (I thought
generator expressions might be a good way to attack this problem, as
you could generate the set, and then check to see if there's a match,
and write each line that way).


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Propagate import for all modules in a package.

[Albert Hopkins]

On Fri, 2009-07-17 at 10:28 -0700, Phil wrote:
> I'm really new to Python and I am absolutely stumped trying to figure
> this out. I have searched plenty, but I am either searching for the
> wrong keywords or this isn't possible.
> 
> What I want to do is have one import be global for the entire package.
> Here is an example...
> 
> 
>   __init__.py
>   module1.py
>   module2.py
>   ...
>   moduleN.py
> 
> I was thinking that I could just, for example, 'from datetime import
> datetime' in __init__.py and have the ability to use 'datetime'
> anywhere in any of the modules in 'package'.
> 
> This didn't work for me. Did I just do something wrong? Is what I am
> trying to do possible?
> 

That's not how packages (were designed to) work.  A package is basically
a namespace.  It doesn't do anything special outside of providing a
namespace for common modules.  There are some special features (e.g.
__init__.py which is basically the "body" of the package and '__all__'
which handles wildcard imports of a package) but other than that it's
just a namespace.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Question regarding style/design..

[Tim Chase]

Robert Kern wrote:
[sage advice snipped]

Don't waste half a day trying to figure out why your script
mysteriously doesn't work. Learn from my mistakes. :-)


I find that's the best type of mistake to learn from:  other
people's ;-)


-tkc
--
http://mail.python.org/mailman/listinfo/python-list

Writing a single list of numbers to a .m (matlab) file

[Hanna Michelsen]

Hi,

I was wondering if I could get some suggestions on how to write a single
list of numbers to a .m file (for matlab) so that I can create a matlab
vector out of the list of numbers from my python program. I have been using
a csv writer to create .m files from lists of lists, but I'm not sure how to
write just a single list to the file. (if I use the same code I've been
using, I get an error: csv.Error sequence expected)

Thanks for the help!
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: getopt code NameError exception on logTail.py

[MRAB]

LoD MoD wrote:

I am having trouble extending my option parsing.
Any help is appreciated:

import sys
import getopt
import time

def tail(file):
while 1:
where = file.tell()
line = file.readline()
if not line:
time.sleep(1)
file.seek(where)
else:
print line, # already has newline

def main():

# parse command line options
try:
opts, args = getopt.getopt(sys.argv[1:], "hf:", ["help", 
"filename="])

except getopt.error, msg:
print msg
print "for help use --help"
sys.exit(2)
# process options
for o, a in opts:
if o in ("-h", "--help"):
print __doc__
sys.exit(0)
if o in ("-f", "--filename"):
print "Parsing F argument"
file = open(filename, 'r')
print file



# process arguments

for arg in args:
process(arg) # process() is defined elsewhere

if __name__ == "__main__":
main()

Yields this error:

localhost:src gsery$ python logTail.py  /var/log/system.log
Traceback (most recent call last):
  File "logTail.py", line 52, in 
main()
  File "logTail.py", line 49, in main
process(arg) # process() is defined elsewhere
NameError: global name 'process' is not defined


The trackback tells you what's wrong: you haven't defined 'process'. The
comment says it's defined elsewhere, but neither I nor Python can see
it! :-)
--
http://mail.python.org/mailman/listinfo/python-list

Re: Generator Expressions and CSV

[Emile van Sebille]

On 7/17/2009 10:58 AM Zaki said...


Now, I've tried running this and it takes much longer than I'd like. I
was wondering if there might be a better way to do things 


Suppose, for the sake of argument, that you've written highly efficient 
code.  Then the processing time would already be entirely optimized and 
no improvements possible.  It's running as fast as it can.  We can't help.


On the other hand, maybe you didn't.  In that case, you'll need to 
profile your code to determine where the time is consumed.  At a 
minimum, you'll need to post the slow parts so we can see the 
implementation and suggest improvements.


Emile

--
http://mail.python.org/mailman/listinfo/python-list

Re: getopt code NameError exception on logTail.py

[LoD MoD]

In this instance the trackback was somewhat unhelpful.There problem was
here:

   file = open(filename, 'r')

should be

   file = open(a, 'r')

args should be passed within the getopt riff

On Fri, Jul 17, 2009 at 11:08 AM, MRAB  wrote:

> LoD MoD wrote:
>
>> I am having trouble extending my option parsing.
>> Any help is appreciated:
>>
>> import sys
>> import getopt
>> import time
>>
>> def tail(file):
>>while 1:
>>where = file.tell()
>>line = file.readline()
>>if not line:
>>time.sleep(1)
>>file.seek(where)
>>else:
>>print line, # already has newline
>>
>> def main():
>>
>># parse command line options
>>try:
>>opts, args = getopt.getopt(sys.argv[1:], "hf:", ["help",
>> "filename="])
>>except getopt.error, msg:
>>print msg
>>print "for help use --help"
>>sys.exit(2)
>># process options
>>for o, a in opts:
>>if o in ("-h", "--help"):
>>print __doc__
>>sys.exit(0)
>>if o in ("-f", "--filename"):
>>print "Parsing F argument"
>>file = open(filename, 'r')
>>print file
>>
>># process arguments
>>for arg in args:
>>process(arg) # process() is defined elsewhere
>>
>> if __name__ == "__main__":
>>main()
>>
>> Yields this error:
>>
>> localhost:src gsery$ python logTail.py  /var/log/system.log
>> Traceback (most recent call last):
>>  File "logTail.py", line 52, in 
>>main()
>>  File "logTail.py", line 49, in main
>>process(arg) # process() is defined elsewhere
>> NameError: global name 'process' is not defined
>>
>>  The trackback tells you what's wrong: you haven't defined 'process'. The
> comment says it's defined elsewhere, but neither I nor Python can see
> it! :-)
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: ctype performance benchmark

[Nick Craig-Wood]

Wai Yip  wrote:
>  I started with ctypes because it is the battery included with the
>  Python standard library. My code is very fluid and I'm looking for
>  easy opportunity to optimize it. One example is to find the longest
>  common prefix among two strings. Right now I am comparing it character
>  by character with pure Python. It seems like an ideal low hanging
>  fruit. With ctype I can rewriting a few lines of Python into a few
>  lines of C. All the tools are available and no third party library is
>  needed.

Yes ctypes is very easy for this and I've used it like this in the
past too.  It makes interfacing C code very easy.

You might consider just writing a plain bit of C code.  If you make a
setup.py too (not hard) then it is very easy to distribute and use on
other platforms.

The Python C API is very easy to use - documentation is extensive.
You need to be very careful with references and the error checking is
boring and tedious!

>  It turned out the performance fail to match pure Python. This function
>  is called million of times in an inner loop. The overhead overwhelm
>  any performance gain with C. Eventually I have found success grouping
>  the data into larger chunk for each API call. This is what I
>  originally planned to do anyway. I am only sharing my experience here
>  that doing fine grained ctype function call has its limitation.

Interesting but not too unexpected - that translation of types is
non-trivial.

>  I have looked into Pyrex before and I was quite confused by the fusion
>  of C and Python langauage. Perhaps it is time for me to give it a
>  second look. I just heard of Cython now and it also look interesting.
>  I think it is helpful for both project to articulate more clearly what
>  they are, how they achieve the performance gain and to give more
>  guidance on good use cases. On the other hand, perhaps it is just me
>  are confused because I don't know enough of the Python internal.

Pyrex evolved/was forked into Cython.

Cython allows you to write a subset of python code that is compiled
into C.  You can also optionally provide some type annotations to get
it to produce better C code at the cost of making your code look less
like python.

I find Cython alternatively brilliant and frustrating!  Sometimes it
works really well, and sometimes I spend hours working out why my
program doesn't compile, or to find out the exact syntax I need to
interface with such and such a library.

I did a C library interfacing project with both ctypes and cython
recently as a comparison.  It came out to be about the same number of
lines of code for both.  I decided to run with the ctypes version as
it was part python.

Here is a short Cython example which interfaces with opendir /
closedir / readdir which python doesn't do natively.  Note the way you
interface with C structures (the real ones are used in the C code -
the annotations just tell cython how to use them).  Note also that
Directory is a class defined in C which I don't think you can't do
with ctypes without a wrapper class.

This compiles into 1320 lines of C, which in turn compile into 11380
bytes of shared object (when stripped).

import cython

cdef extern from "dirent.h":
struct dirent:
char d_name[0]
ctypedef struct DIR
DIR *opendir(char *name)
int closedir(DIR *dirp)
dirent *readdir(DIR *dirp)

cdef extern from "errno.h":
int errno

cdef extern from "string.h":
char *strerror(int errnum)

cdef class Directory:
"""Represents an open directory"""

cdef DIR *handle

def __init__(self, path):
self.handle = opendir(path)
if self.handle is NULL:
raise OSError(errno, "Failed to open directory: %s" % 
strerror(errno))

def readdir(self):
"""Read the next name in the directory"""
cdef dirent *p
p = readdir(self.handle)
if p is NULL:
return None
return p.d_name

def close(self):
"""Close the directory"""
if self.handle is not NULL:
closedir(self.handle)
self.handle = NULL

def __dealloc__(self):
self.close()


-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Propagate import for all modules in a package.

[Phil]

Thanks to both of you for the fast and detailed responses. I will just
treat the package for what it is, a namespace.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Propagate import for all modules in a package.

[Phil]

Thanks to both of you for the fast and detailed responses. I will
just
use the package for what it is, a namespace.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Propagate import for all modules in a package.

[Phil]

Thanks to both of you for the fast and detailed responses. I will just
use the package for what it is, a namespace.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Generator Expressions and CSV

[MRAB]

Zaki wrote:

Hey all,

I'm really new to Python and this may seem like a really dumb
question, but basically, I wrote a script to do the following, however
the processing time/memory usage is not what I'd like it to be. Any
suggestions?


Outline:
1. Read tab delim files from a directory, files are of 3 types:
install, update, and q. All 3 types contain ID values that are the
only part of interest.
2. Using set() and set.add(), generate a list of unique IDs from
install and update files.
3. Using the set created in (2), check the q files to see if there are
matches for IDs. Keep all matches, and add any non matches (which only
occur once in the q file) to a queue of lines to be removed from teh q
files.
4. Remove the lines in the q for each file. (I haven't quite written
the code for this, but I was going to implement this using csv.writer
and rewriting all the lines in the file except for the ones in the
removal queue).

Now, I've tried running this and it takes much longer than I'd like. I
was wondering if there might be a better way to do things (I thought
generator expressions might be a good way to attack this problem, as
you could generate the set, and then check to see if there's a match,
and write each line that way).


Why are you checking and removing lines in 2 steps? Why not copy the
matching lines to a new q file and then replace the old file with the
new one (or, maybe, delete the new q file if no lines were removed)?
--
http://mail.python.org/mailman/listinfo/python-list

Mechanize not recognized by py2exe

[Stephen M. Olds]

I have a Python script getData.py that uses Mechanize, and runs fine
under the interpreter. It was installed using easy_install - and the
install seemed to indicate it was completed.

The problem is, when I try to compile it using py2exe while in the
folder of the script, and using the run line command:

python getData.py py2exe

I get the warning: "Import error: No Module named mechanize"...

I checked the environmental path and find the following:
%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;C:\Program
Files (x86)\Python;C:\Python25\Lib\site-packages;C:\Program Files
(x86)\Common Files\Adobe\AGL

I did a search for mechanize and find an egg file in
C:\Python25\Lib\site-packages named mechanize-0.1.7b-py2.5.

Not really understanding the "egg" thing, what do I have here that
needs to be done?

Stephen







-- 
http://mail.python.org/mailman/listinfo/python-list

Re: ctype performance benchmark

[Duncan Booth]

Wai Yip  wrote:

> I started with ctypes because it is the battery included with the
> Python standard library. My code is very fluid and I'm looking for
> easy opportunity to optimize it. One example is to find the longest
> common prefix among two strings. Right now I am comparing it character
> by character with pure Python. It seems like an ideal low hanging
> fruit. With ctype I can rewriting a few lines of Python into a few
> lines of C. All the tools are available and no third party library is
> needed.

Alternatively you could just use os.path.commonprefix().

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: A Bug By Any Other Name ...

[Albert van der Horst]

In article ,
Gabriel Genellina  wrote:
>En Mon, 06 Jul 2009 00:28:43 -0300, Steven D'Aprano
> escribi?:
>> On Mon, 06 Jul 2009 14:32:46 +1200, Lawrence D'Oliveiro wrote:
>>
>>> I wonder how many people have been tripped up by the fact that
>>>
>>> ++n
>>>
>>> and
>>>
>>> --n
>>>
>>> fail silently for numeric-valued n.
>>
>> What do you mean, "fail silently"? They do exactly what you should
>> expect:
> ++5  # positive of a positive number is positive
>>
>> I'm not sure what "bug" you're seeing. Perhaps it's your expectations
>> that are buggy, not Python.
>
>Well, those expectations are taken seriously when new features are
>introduced into the language - and sometimes the feature is dismissed just
>because it would be confusing for some.
>If a += 1 works, expecting ++a to have the same meaning is very reasonable
>(for those coming from languages with a ++ operator, like C or Java) -
>more when ++a is a perfectly valid expression.
>If this issue isn't listed under the various "Python gotchas" articles, it
>should...

In algol 68 there was one thing strictly forbidden: putting two
operators behind each other:
x :=  y** -b .comment  must be x := y**(-b) .comment

This is quite a sensible rule, especially when, like in Python,
two special characters can be run together to denote a different
operator.
Examples abound :   +:=  **
A consequence is that  'a*-b'
would be illegal. It would become 'a*(-b)'
Worse is that x=-q would be illegal.

Because unlike in algol 68 in python whitespace is relevant,
we could get by with requiring whitespace:
x= -q   # okay
a--
>Gabriel Genellina

Groetjes Albert


--
-- 
Albert van der Horst, UTRECHT,THE NETHERLANDS
Economic growth -- being exponential -- ultimately falters.
alb...@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

-- 
http://mail.python.org/mailman/listinfo/python-list

proposal: add setresuid() system call to python

[travis+ml-python]

Hello,

Historically, I have used scripting languages like python for typical
uses, but they tend to not fare very well at system programming; for
close interfacing with the operating system, I'm often forced to use a
language like C.  This is undesirable to me.

I do not think this has to be the case; I see no reason why a
scripting language can't implement more of the system call API, at the
risk of having some OS-dependent modules.  I would actually like to
see more network servers written in scripting languages, as they
neatly avoid buffer overflow and integer overflow issues with no extra
effort.

One BIG roadblock to doing this is when they can't manage to drop
permissions properly.

I am suggesting that the setresuid function be added to python,
perhaps in the OS module, because it has the clearest semantics for
manipulating user ids.  The reason why is best described in the
following paper:

http://www.eecs.berkeley.edu/~daw/papers/setuid-usenix02.pdf

One argument against this is that it is not specified by POSIX, and
thus might be dismissed as "implementation dependent".

However, as the paper above demonstrates, even though the setuid
system call is defined by POSIX, it already has system-dependent
behavior.  POSIX provides for at least two different behaviors of the
setuid call, and even more if you consider that it leaves what
constitutes "appropriate privileges" up to the OS kernel.

I humbly propose that python implement all the routines necessary to
securely drop privileges, to enable construction of network daemons
that might need to drop privileges from root to some non-root userid
(e.g. mail transfer agents, or POP/IMAP servers).

Furthermore, where there are multiple system calls to achieve this
effect, it should implement the ones with the clearest semantics, and
setresuid fits that bill.  To see what an utter mess the uid-manipulation
routines are in, I refer you once again to this paper, as the situation
is too complicated to describe in this email:

http://www.eecs.berkeley.edu/~daw/papers/setuid-usenix02.pdf

Opinions?

Best,
Travis
-- 
Obama Nation | My emails do not have attachments; it's a digital signature
that your mail program doesn't understand. | 
http://www.subspacefield.org/~travis/ 
If you are a spammer, please email j...@subspacefield.org to get blacklisted.


pgpVyRpXxMRRP.pgp
Description: PGP signature
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: A Bug By Any Other Name ...

[J. Cliff Dyer]

On Fri, 2009-07-17 at 20:53 +, Albert van der Horst wrote:
> Because unlike in algol 68 in python whitespace is relevant,
> we could get by with requiring whitespace:
> x= -q   # okay
> a 8 ** -2 # okay

This is actually quite thoroughly untrue.  In python, *indentation* is
significant.  Whitespace (internal to a line) is not.  You can even call
methods like this if you want:

>>> s = 'abc'
>>> s. upper()
ABC

Obviously, that's A Bad Idea(tm), but python's parser won't stop you.
The ++ operator gotcha is so minor that I can't remember anyone actually
asking about it on the list (who was actually facing it as a
problem--this thread was started by idle speculation).  Can we not
change the language syntax to address non-issues?

Practicality beats purity, a.k.a. don't you have something better to do?


Cheers,
Cliff



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Generator Expressions and CSV

[Zaki]

On Jul 17, 2:49 pm, MRAB  wrote:
> Zaki wrote:
> > Hey all,
>
> > I'm really new to Python and this may seem like a really dumb
> > question, but basically, I wrote a script to do the following, however
> > the processing time/memory usage is not what I'd like it to be. Any
> > suggestions?
>
> > Outline:
> > 1. Read tab delim files from a directory, files are of 3 types:
> > install, update, and q. All 3 types contain ID values that are the
> > only part of interest.
> > 2. Using set() and set.add(), generate a list of unique IDs from
> > install and update files.
> > 3. Using the set created in (2), check the q files to see if there are
> > matches for IDs. Keep all matches, and add any non matches (which only
> > occur once in the q file) to a queue of lines to be removed from teh q
> > files.
> > 4. Remove the lines in the q for each file. (I haven't quite written
> > the code for this, but I was going to implement this using csv.writer
> > and rewriting all the lines in the file except for the ones in the
> > removal queue).
>
> > Now, I've tried running this and it takes much longer than I'd like. I
> > was wondering if there might be a better way to do things (I thought
> > generator expressions might be a good way to attack this problem, as
> > you could generate the set, and then check to see if there's a match,
> > and write each line that way).
>
> Why are you checking and removing lines in 2 steps? Why not copy the
> matching lines to a new q file and then replace the old file with the
> new one (or, maybe, delete the new q file if no lines were removed)?

That's what I've done now.

Here is the final code that I have running. It's very much 'hack' type
code and not at all efficient or optimized and any help in optimizing
it would be greatly appreciated.

import csv
import sys
import os
import time

begin = time.time()

#Check minutes elapsed
def timeElapsed():
current = time.time()
elapsed = current-begin
return round(elapsed/60)


#USAGE: python logcleaner.py  

inputdir = sys.argv[1]
outputdir = sys.argv[2]

logfilenames = os.listdir(inputdir)



IDs = set() #IDs from update and install logs
foundOnceInQuery = set()
#foundTwiceInQuery = set()
#IDremovalQ = set() Note: Unnecessary, duplicate of foundOnceInQuery;
Queue of IDs to remove from query logs (IDs found only once in query
logs)

#Generate Filename Queues For Install/Update Logs, Query Logs
iNuQ = []
queryQ = []

for filename in logfilenames:
if filename.startswith("par1.install") or filename.startswith
("par1.update"):
iNuQ.append(filename)
elif filename.startswith("par1.query"):
queryQ.append(filename)

totalfiles = len(iNuQ) + len(queryQ)
print "Total # of Files to be Processed:" , totalfiles
print "Install/Update Logs to be processed:" , len(iNuQ)
print "Query logs to be processed:" , len(queryQ)

#Process install/update queue to generate list of valid IDs
currentfile = 1
for file in iNuQ:
print "Processing", currentfile, "install/update log out of", len
(iNuQ)
print timeElapsed()
reader = csv.reader(open(inputdir+file),delimiter = '\t')
for row in reader:
IDs.add(row[2])
currentfile+=1

print "Finished processing install/update logs"
print "Unique IDs found:" , len(IDs)
print "Total Time Elapsed:", timeElapsed()

currentfile = 1
for file in queryQ:
print "Processing", currentfile, "query log out of", len(queryQ)
print timeElapsed()
reader = csv.reader(open(inputdir+file), delimiter = '\t')
outputfile = csv.writer(open(outputdir+file), 'w')
for row in reader:
if row[2] in IDs:
ouputfile.writerow(row)
else:
if row[2] in foundOnceInQuery:
foundOnceInQuery.remove(row[2])
outputfile.writerow(row)
#IDremovalQ.remove(row[2])
#foundTwiceInQuery.add(row[2])

else:
foundOnceInQuery.add(row[2])
#IDremovalQ.add(row[2])


currentfile+=1

print "Finished processing query logs and writing new files"
print "# of Query log entries removed:" , len(foundOnceInQuery)
print "Total Time Elapsed:", timeElapsed()


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: proposal: add setresuid() system call to python

[Mahmoud Abdelkader]

Why don't you write a python extension module? This is a perfect opportunity
for that.

--
mahmoud mack abdelkader
http://blog.mahmoudimus.com/



On Fri, Jul 17, 2009 at 4:01 PM,

> wrote:

> Hello,
>
> Historically, I have used scripting languages like python for typical
> uses, but they tend to not fare very well at system programming; for
> close interfacing with the operating system, I'm often forced to use a
> language like C.  This is undesirable to me.
>
> I do not think this has to be the case; I see no reason why a
> scripting language can't implement more of the system call API, at the
> risk of having some OS-dependent modules.  I would actually like to
> see more network servers written in scripting languages, as they
> neatly avoid buffer overflow and integer overflow issues with no extra
> effort.
>
> One BIG roadblock to doing this is when they can't manage to drop
> permissions properly.
>
> I am suggesting that the setresuid function be added to python,
> perhaps in the OS module, because it has the clearest semantics for
> manipulating user ids.  The reason why is best described in the
> following paper:
>
> http://www.eecs.berkeley.edu/~daw/papers/setuid-usenix02.pdf
>
> One argument against this is that it is not specified by POSIX, and
> thus might be dismissed as "implementation dependent".
>
> However, as the paper above demonstrates, even though the setuid
> system call is defined by POSIX, it already has system-dependent
> behavior.  POSIX provides for at least two different behaviors of the
> setuid call, and even more if you consider that it leaves what
> constitutes "appropriate privileges" up to the OS kernel.
>
> I humbly propose that python implement all the routines necessary to
> securely drop privileges, to enable construction of network daemons
> that might need to drop privileges from root to some non-root userid
> (e.g. mail transfer agents, or POP/IMAP servers).
>
> Furthermore, where there are multiple system calls to achieve this
> effect, it should implement the ones with the clearest semantics, and
> setresuid fits that bill.  To see what an utter mess the uid-manipulation
> routines are in, I refer you once again to this paper, as the situation
> is too complicated to describe in this email:
>
> http://www.eecs.berkeley.edu/~daw/papers/setuid-usenix02.pdf
>
> Opinions?
>
> Best,
> Travis
> --
> Obama Nation | My emails do not have attachments; it's a digital signature
> that your mail program doesn't understand. |
> http://www.subspacefield.org/~travis/
> If you are a spammer, please email j...@subspacefield.org to get
> blacklisted.
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: proposal: add setresuid() system call to python

[Robert Kern]

On 2009-07-17 15:01, travis+ml-pyt...@subspacefield.org wrote:

Hello,

Historically, I have used scripting languages like python for typical
uses, but they tend to not fare very well at system programming; for
close interfacing with the operating system, I'm often forced to use a
language like C.  This is undesirable to me.


You can use ctypes for most of these use cases.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

--
http://mail.python.org/mailman/listinfo/python-list

Re: turtle dump

[Terry Reedy]

Peter Otten wrote:

Terry Reedy wrote:


I tried it. Unfortunately, OOo does not open it correctly. It just
displays the first three lines of metadate - Title, Creator, Date -- as
image text. Photoshop does read the image, and does an ok job of
conversion once anti-aliasing is turned off.


I snatched some code from the module and modified it to save a sample image:


Thank you for the example.



from turtle import *

def switchpen():
if isdown():
pu()
else:
pd()

def demo2():
"""Demo of some new features."""
speed(1)
st()
pensize(3)
setheading(towards(0, 0))
radius = distance(0, 0)/2.0
rt(90)
for _ in range(18):
switchpen()
circle(radius, 10)
write("wait a moment...")
while undobufferentries():
undo()
reset()
lt(90)
colormode(255)
laenge = 10
pencolor("green")
pensize(3)
lt(180)
for i in range(-2, 16):
if i > 0:
begin_fill()
fillcolor(255-15*i, 0, 15*i)
for _ in range(3):
fd(laenge)
lt(120)
laenge += 10
lt(15)
speed((speed()+1)%12)
end_fill()

lt(120)
pu()
fd(70)
rt(30)
pd()
color("red","yellow")
speed(0)
fill(1)


begin_fill() in 3.x


for _ in range(4):
circle(50, 90)
rt(90)
fd(30)
rt(90)
fill(0)


end_fill() in 3.x


lt(90)
pu()
fd(30)
pd()
shape("turtle")

tri = getturtle()
tri.resizemode("auto")
turtle = Turtle()
turtle.resizemode("auto")
turtle.shape("turtle")
turtle.reset()
turtle.left(90)
turtle.speed(0)
turtle.up()
turtle.goto(280, 40)
turtle.lt(30)
turtle.down()
turtle.speed(6)
turtle.color("blue","orange")
turtle.pensize(2)
tri.speed(6)
setheading(towards(turtle))
count = 1
while tri.distance(turtle) > 4:
turtle.fd(3.5)
turtle.lt(0.6)
tri.setheading(tri.towards(turtle))
tri.fd(4)
if count % 20 == 0:
turtle.stamp()
tri.stamp()
switchpen()
count += 1
tri.write("CAUGHT! ", font=("Arial", 16, "bold"), align="right")

if __name__ == "__main__":
demo2()
Screen().getcanvas().postscript(file="demo2.eps")

I could successfully insert the picture into Writer.


Whereas I still get just three lines of metadata.
The size of my demo2.eps is given as 65,628 bytes.
'Size on disk is given as a bit more but I presume that counts to the 
end of the last 4k block.



I'm on Kubuntu 9.04 with Python 2.6.2 and OpenOffice 3.0.1.


WinXP with updates, Python 3.1, and OO 3.1.0

So either tk 8.5(?) on windows is producing something different or OO3.1 
 on windows is reading differently. If your file size is different, 
could you email it so I could try to import it here?


But still, turtle is an educational tool, its main advantage is that it can 
show beginners how the image is generated.


It is also available with Python as installed. And I want to do simple 
animations (of algoritms) as well as static pictures.


If you want to generate high-quality graphics easily you need different 
primitives.


http://cairographics.org

has Python bindings, but I don't know if it's available on Windows.


Effectively not, as far as I can tell. PyCairo appears to be *nix and 
require later binaries than those available from gtk site referenced 
from cairo site.


Terry Jan Reedy

--
http://mail.python.org/mailman/listinfo/python-list

Re: turtle dump

[Terry Reedy]

alex23 wrote:


The help in iPython says the same, but also mentions that it's a
dynamically generated function, so it may not be picking up the
docstring that way. turtle.ScrolledCanvas.postscript is similarly
terse, but you can find more info in turtle.Canvas.postscript:

  Print the contents of the canvas to a postscript
  file. Valid options: colormap, colormode, file, fontmap,
  height, pageanchor, pageheight, pagewidth, pagex, pagey,
  rotate, witdh, x, y.


How, exactly, did you get that list?

tjr

--
http://mail.python.org/mailman/listinfo/python-list

Re: no return value for threading.Condition.wait(timeout)?

[Piet van Oostrum]

> Gabriel Rossetti  (GR) wrote:

>GR> Piet van Oostrum wrote:
 Gabriel Rossetti  (GR) wrote:
 
>>> 
>>> 
>GR> I have a 1-1 relation, I have a thread reading msgs from a network socket
>GR> and a method waiting for an answer to arrive (see my thread
>GR> "Threading.Condition problem"). I would like to be able to have a timeout
>GR> as to not block forever, so the idea is to check if I returned because of a
>GR> timeout or not.
 
>>> 
>>> So that case is easy I think. After the wait just check if the answer
>>> has arrived.
>>> 
>GR> The problem is other answers could arrive that are not the one I
>GR> want. Maybe I could check if I got the answer, if not then check
>GR> how much time has elapsed, and if it's grater or equal to the
>GR> timeout consider it a timeout. I still think a return value would
>GR> be better though.

The wait will not get a return value in Python. There was an issue
posted in the tracker to request this in 2005. Recently (after 4 years)
it was rejected by the BDFL. See http://bugs.python.org/issue1175933

OK, your use case might be a bit more difficult, but I still think it
can be solved without a return value. I can't tell exactly how because
of lack of information.

So (as I already mentioned in one of my previous messages) being
notified doesn't imply that everything is OK. Now suppose that your CV
is notified one microsecond before it would time out but the required
answer did not arrive. What's the difference with a real timeout except
for that microsecond?

Reimplementing Condition yourself is not to be recommended I think.
Using these synchronisation primitives is hard enough let alone
implementing them. 
-- 
Piet van Oostrum 
URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
Private email: p...@vanoostrum.org
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Writing a single list of numbers to a .m (matlab) file

[Chris Rebert]

On Fri, Jul 17, 2009 at 11:06 AM, Hanna Michelsen wrote:
> Hi,
>
> I was wondering if I could get some suggestions on how to write a single
> list of numbers to a .m file (for matlab) so that I can create a matlab
> vector out of the list of numbers from my python program. I have been using
> a csv writer to create .m files from lists of lists, but I'm not sure how to
> write just a single list to the file. (if I use the same code I've been
> using, I get an error: csv.Error sequence expected)

You didn't provide any code, but since csv is probably overkill anyway:

m_file.write("[" + " ".join(str(num) for num in numbers) + "]\n")

Cheers,
Chris
-- 
http://blog.rebertia.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: A Bug By Any Other Name ...

[MRAB]

J. Cliff Dyer wrote:

On Fri, 2009-07-17 at 20:53 +, Albert van der Horst wrote:

Because unlike in algol 68 in python whitespace is relevant,
we could get by with requiring whitespace:
x= -q   # okay
a

This is actually quite thoroughly untrue.  In python, *indentation* is
significant.  Whitespace (internal to a line) is not.

[snip]
Whitespace (internal to a line) _mostly_ is not. It's not allowed within
names, and it is needed in the second example above, before the 'and's.
--
http://mail.python.org/mailman/listinfo/python-list

Re: proposal: add setresuid() system call to python

[Jean-Paul Calderone]

On Fri, 17 Jul 2009 15:01:41 -0500, travis+ml-pyt...@subspacefield.org wrote:

Hello,

[snip]

I am suggesting that the setresuid function be added to python,
perhaps in the OS module, because it has the clearest semantics for
manipulating user ids.  The reason why is best described in the
following paper:


Yes, it'd be good for Python to expose setresuid.  The best course of
action is to file a ticket in the issue tracker.  Things will be sped
along if you also attach a patch implementing the change. :)

Jean-Paul
--
http://mail.python.org/mailman/listinfo/python-list