multiprocessing and freezing on Windows

[SK]

Is there a method for freezing a Python 2.6 app using multiprocessing
on Windows using PyInstaller or py2exe that works? It is trying to
call my executable instead of python.exe when the process starts and
passes it --multiprocessing-fork . Adding a freeze_support() to my
main doesn't help. Do I have to bundle all of Python and use
set_executable() to point to the bundled python.exe? I can't find any
workable information on the canonical way to do this.

Thanks!
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Open Source RSS Reader in Python?

[Chris Rebert]

On Wed, Jul 1, 2009 at 4:18 PM, Alex wrote:
> I am looking for an open source RSS reader (desktop, not online)
> written in Python but in vain. I am not looking for a package but a
> fully functional software.
>
> Google: python "open source" (rss OR feeds) reader
>
> Any clue ?

Straw for GTK/GNOME is written in Python, though it's somewhat inactive.
http://projects.gnome.org/straw/
http://strawreader.wordpress.com/

Cheers,
Chris
-- 
http://blog.rebertia.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Searching equivalent to C++ RAII or deterministic destructors

[Ulrich Eckhardt]

Hi!

I'm currently converting my bioware to handle Python code and I have
stumbled across a problem...

Simple scenario: I have a handle to a resource. This handle allows me to
manipulate the resource in various ways and it also represents ownership.
Now, when I put this into a class, instances to that class can be shared,
using Python's reference counting. What I'm missing is a way to
automatically release the resource, something which I would do in the
destructor in C++.

Any ideas how to solve this?

Uli

-- 
Sator Laser GmbH
Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Accessing windows structures through ctypes.

[Enrico]

"Rajat"  ha scritto nel messaggio
news:8c8b5cf2-bc77-4633-96ca-e3b908430...@z14g2000yqa.googlegroups.com...
>
> > > Using ctypes can I access the windows structures like:
> >
> > > PROCESS_INFORMATION_BLOCK, Process Environment Block(PEB),
> > > PEB_LDR_DATA, etc?
> >
> > ctypes.wintypes lists all of the Windows structures included with the
> > module.
> >
> > You should be able to use ctypes.Structure class to roll your own:
>
> Thanks Alex. As you suggested, I'm trying to implemenet the below
> structure, windows PEB, in Python:
>
> typedef struct _PEB {
> BYTE Reserved1[2];
> BYTE BeingDebugged;
> BYTE Reserved2[21];
> PPEB_LDR_DATA LoaderData;
> PRTL_USER_PROCESS_PARAMETERS ProcessParameters;
> BYTE Reserved3[520];
> PPS_POST_PROCESS_INIT_ROUTINE PostProcessInitRoutine;
> BYTE Reserved4[136];
> ULONG SessionId;
> } PEB;
>
> My equivalent Python structure is:
> class PEB(Structure):
> _fields_ = [("Reserved1", wintypes.BYTE * 2),
> ("BeingDebugged", wintypes.BYTE),
> ("Reserved2", wintypes.BYTE * 2),
> ("Reserved3", c_void_p),
> ("Ldr", pointer(PEB_LDR_DATA)),
> ("ProcessParameters", pointer
> (RTL_USER_PROCESS_PARAMETERS)),
> ("Reserved4", wintypes.BYTE * 104),
> ("Reserved5", c_void_p),
> (),
> ("Reserved6", wintypes.BYTE),
> ("Reserved7", c_void_p),
> ("SessionId", c_ulong)]
>
> I'm not sure what needs to go in the above empty tuple for
> "PPS_POST_PROCESS_INIT_ROUTINE PostProcessInitRoutine" (in Original
> PEB).
>
> Please suggest.

PostProcessInitRoutine should be a callback function or something similar.
It should be enough to define a type

PostProcessInitRoutine = WINFUNCTYPE(...parameters...)

and use this.

Regards,
Enrico


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: PEP 376

[Joachim Strömbergson]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Aloha!

Richard Brodie wrote:
> "Joachim Str�mbergson"  wrote in message 
> news:mailman.2422.1246418400.8015.python-l...@python.org...
> 
>> Even so, choosing md5 in 2009 for something that (hopefully) will be
>> used in years is a bad design decision. It creates a dependency for to
>> an algorithm that all sensible recommendations point you to move away
>> from.
> 
> Why not write the field as algorithm:value?
> 
> e.g. sha1:8590b685654367e3eba70dc00df7e45e88c21da4
> 
> Installers can fallback to using hashlib.new(), so you can plug in a new
> algorithm without changing the PEP or the installer code. 

+1

Good idea and future proof as well as being simple.

- --
Med vänlig hälsning, Yours

Joachim Strömbergson - Alltid i harmonisk svängning.

Kryptoblog - IT-säkerhet på svenska
http://www.strombergson.com/kryptoblog

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.8 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkpMaFoACgkQZoPr8HT30QHVxQCfWqKprUaAQxaxTfYJVfIp8+FC
5UIAn2cWNoMe+sdNO4BSZG7e+19qkqvl
=Id7s
-END PGP SIGNATURE-
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: getting rid of —

[Tep]

On 2 Jul., 01:56, MRAB  wrote:
> someone wrote:
> > Hello,
>
> > how can I replace '—' sign from string? Or do split at that character?
> > Getting unicode error if I try to do it:
>
> > UnicodeDecodeError: 'ascii' codec can't decode byte 0x97 in position
> > 1: ordinal not in range(128)
>
> > Thanks, Pet
>
> > script is # -*- coding: UTF-8 -*-
>
> It sounds like you're mixing bytestrings with Unicode strings. I can't
> be any more helpful because you haven't shown the code.

Oh, I'm sorry. Here it is

def cleanInput(input)
return input.replace('—', '')
-- 
http://mail.python.org/mailman/listinfo/python-list

PEP368 and pixeliterators

[Joachim Strömbergson]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Aloha!

I just read the PEP368 and really liked the proposed idea, sound like a
great battery addition to include in the std lib:

http://www.python.org/dev/peps/pep-0368/

One question/idea though: The proposed iterator will iterate over all
pixels in a line, but only one line. The example code looks like this:

# iterate over an image
for line in rgb_image:
for pixel in line:
# swap red and blue, and set green to 0
pixel.value = pixel.b, 0, pixel.r


But, wouldn't it be more Pythonic and simpler to have an iterator that
iterates over all pixels in an image? Starting with upper left corner
and moving left-right and (line by line) to lower right. This would
change the code above to:

for pixel in rgb_image:
# swap red and blue, and set green to 0
pixel.value = pixel.b, 0, pixel.r


The idea I'm having is that fundamentally the image is made up of a 2D
array of pixels, not rows of pixels. And having the iterator focus on
pixels instead make then more sense, no?

Or if possible have two iterators.


Otherwise, big thumbs up for PEP 368 as well as having PEP:s for driving
the language development process in an orderly and well documented way.

- --
Med vänlig hälsning, Yours

Joachim Strömbergson - Alltid i harmonisk svängning.

Kryptoblog - IT-säkerhet på svenska
http://www.strombergson.com/kryptoblog

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.8 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkpMcIQACgkQZoPr8HT30QFkcgCgzxb9JS2l87B/nkpf05FLDjY5
RPQAni2yPxKjCd4lM/qMBNhjp8HHg/PZ
=9gB/
-END PGP SIGNATURE-
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: string character count

[noydb]

thanks everyone for all the ideas -- simple stuff, I know for you all,
but very helpful for me.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: getting rid of —

[Tep]

On 2 Jul., 10:25, Tep  wrote:
> On 2 Jul., 01:56, MRAB  wrote:
>
> > someone wrote:
> > > Hello,
>
> > > how can I replace '—' sign from string? Or do split at that character?
> > > Getting unicode error if I try to do it:
>
> > > UnicodeDecodeError: 'ascii' codec can't decode byte 0x97 in position
> > > 1: ordinal not in range(128)
>
> > > Thanks, Pet
>
> > > script is # -*- coding: UTF-8 -*-
>
> > It sounds like you're mixing bytestrings with Unicode strings. I can't
> > be any more helpful because you haven't shown the code.
>
> Oh, I'm sorry. Here it is
>
> def cleanInput(input)
>     return input.replace('—', '')

I also need:

#input is html source code, I have problem with only this character
#input = 'foo — bar'
#return should be foo
def splitInput(input)
parts = input.split(' — ')
return parts[0]


Thanks!
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Determining if a function is a method of a class within a decorator

[Jeremiah Dodds]

On Wed, Jul 1, 2009 at 2:14 AM, David Hirschfield  wrote:

>  Unfortunately that still requires two separate decorators, when I was
> hoping there was a way to determine if I was handed a function or method
> from within the same decorator.
>
> Seems like there really isn't, so two decorators is the way to go.
> Thanks,
> -David
>
>
This is a really horrible idea, but it may work:

If you can rely on your code using the word "self", you could use the
inspect module to look at the arguments of the function being decorated, and
dispatch based on that. The idea itself feels extremely dirty though.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: A question about fill_free_list(void) function

[Peter Otten]

Pedram wrote:

> On Jul 1, 10:01 pm, Christian Heimes  wrote:
>> Pedram schrieb:
>>
>> > Hello community,
>> > I'm reading the CPython interpreter source code,
>> > first, if you have something that I should know for better reading
>> > this source code, I would much appreciate that :)
>> > second, in intobject.c file, I read the following code in
>> > fill_free_list function that I couldn't understand:
>> > while (--q > p)
>> > Py_TYPE(q) = (struct _typeobject *)(q-1);
>> > Py_TYPE(q) = NULL;
>> > 
>> > What does it mean?
>>
>> The code is abusing the ob_type member to store a single linked, NULL
>> terminated list of free integer objects. It's a tricky optimization you
>> don't have to understand in order to understand the rest of the code.
>>
>> And no, the code in ctypes.h is totally unrelated to the free list in
>> intobject.c.
>>
>> Christian
> 
> Thanks for reply.
> I totally understood the fill_free_list() function. Thanks.
> But I have problems in understanding the PyInt_FromLong(). Here's the
> code:
> 
> PyObject *
> PyInt_FromLong(long ival)
> {
> register PyIntObject *v;
> #if NSMALLNEGINTS + NSMALLPOSINTS > 0
> if (-NSMALLNEGINTS <= ival && ival < NSMALLPOSINTS) {
> v = small_ints[ival + NSMALLNEGINTS];
> Py_INCREF(v);
> #ifdef COUNT_ALLOCS
> if (ival >= 0)
> quick_int_allocs++;
> else
> quick_neg_int_allocs++;
> #endif
> return (PyObject *) v;
> }
> #endif
> if (free_list == NULL) {
> if ((free_list = fill_free_list()) == NULL)
> return NULL;
> }
> 
> v = free_list; // <-- No problem till here :)
> free_list = (PyIntObject *)Py_TYPE(v);
> PyObject_INIT(v, &PyInt_Type);
> v->ob_ival = ival;
> return (PyObject *) v;
> }
> 
> I understood that small numbers (between -5 and 256) are referenced to
> small_ints list. Others have to store in PyIntBlock, am I right? OK,
> no problem till 'v = free_list;' but can't understand the line
> 'free_list = (PyIntObject *)Py_TYPE(v);' :(. Where free_list
> referenced to? Don't this correct that free_list have to be referenced
> to another free block so next time the function called v store in this
> location?
> Sorry for my bad English. I hope you could understand me :)

Well, if I understand fill_free_list() correctly, you haven't ;) It only 
makes sense together with the allocation code in PyInt_FromLong().

Py_TYPE(v) translates to v->ob_type, and in fill_free_list() that field was 
abused(*) to hold a pointer to the previous item in the array of free 
integers, thereby temporarily turning it into a linked list.

Assuming that free_list pointed to block->objects[10] in the current 
PyIntBlock it will now point to block->objects[9].

When block->objects[0] is reached free_list will be set to NULL as block-
>objects[0]->ob_type was initialized with NULL by the line

Py_TYPE(q) = NULL; /* in fill_free_list() */

The next call of PyInt_FromLong() will then trigger the allocation of a new 
block.

Peter

(*) It may help if you think of the objects array as a

union linked {
linked *next;
PyInt_Object obj;
};


-- 
http://mail.python.org/mailman/listinfo/python-list

Need Help

[Tengiz Davitadze]

Hello. I can't find a wright mail address. If you can help me I need to get
an information about UNICODE. I am georgian and I need to write programs on
georgian language . If you can transfer this mail or send me a wright mail
about encoding or unicode information.

-- 
Tengiz Davitadze
{
programing Future
}
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: deleting certain entries in numpy array

[Sebastian Schabe]

Robert Kern schrieb:
First, convert the pos array to integers, and just the columns with 
indices in them:


  ipos = pos[:,:2].astype(int)

Now check the values in the mask corresponding to these positions:

  mask_values = mask[ipos[:,0], ipos[:,1]]

Now extract the rows from the original pos array where mask_values is 
nonzero:


  result = pos[mask_values != 0]



Great!!! That's the way I wanted.

After reading the numpy reference guide I supposed the delete function 
was not the right way, but that it's all about cerrect indexing. But I 
didn't really know how.


So thanks a lot, also to Ben.

>
> You will want to ask numpy questions on the numpy mailing list.
>
>   http://www.scipy.org/Mailing_Lists
>

I ever thought news-groups are the right way for questions like this. 
And the mailing list page just confuses me, but I'm trying to get used 
to it.



Sebastian
--
http://mail.python.org/mailman/listinfo/python-list

Re: Searching equivalent to C++ RAII or deterministic destructors

[Bearophile]

Ulrich Eckhardt:
> a way to automatically release the resource, something
> which I would do in the destructor in C++.

Is this helpful?
http://effbot.org/pyref/with.htm

Bye,
bearophile
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Multi thread reading a file

[Gabriel Genellina]

En Wed, 01 Jul 2009 12:49:31 -0300, Scott David Daniels  
 escribió:



Gabriel Genellina wrote:

...
def convert(in_queue, out_queue):
  while True:
row = in_queue.get()
if row is None: break
# ... convert row
out_queue.put(converted_line)


These loops work well with the two-argument version of iter,
which is easy to forget, but quite useful to have in your bag
of tricks:

 def convert(in_queue, out_queue):
 for row in iter(in_queue.get, None):
 # ... convert row
 out_queue.put(converted_line)


Yep, I always forget about that variant of iter() -- very handy!

--
Gabriel Genellina

--
http://mail.python.org/mailman/listinfo/python-list

Re: Searching equivalent to C++ RAII or deterministic destructors

[Ulrich Eckhardt]

Bearophile wrote:
> Ulrich Eckhardt:
>> a way to automatically release the resource, something
>> which I would do in the destructor in C++.
> 
> Is this helpful?
> http://effbot.org/pyref/with.htm

Yes, it aims in the same direction. However, I'm not sure this applies to my
case. The point is that the resource handle is not just used locally in a
restricted scope but it is allocated and stored. The 'with' is something
that makes sense in the context of mutex locking, where you have a
well-defined critical section. What I need is something similar to open(),
which returs a file. When the last reference to that object goes out of
scope, the underlying file object is closed.

Uli

-- 
Sator Laser GmbH
Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Searching equivalent to C++ RAII or deterministic destructors

[Peter Otten]

Ulrich Eckhardt wrote:

> Bearophile wrote:
>> Ulrich Eckhardt:
>>> a way to automatically release the resource, something
>>> which I would do in the destructor in C++.
>> 
>> Is this helpful?
>> http://effbot.org/pyref/with.htm
> 
> Yes, it aims in the same direction. However, I'm not sure this applies to
> my case. The point is that the resource handle is not just used locally in
> a restricted scope but it is allocated and stored. The 'with' is something
> that makes sense in the context of mutex locking, where you have a
> well-defined critical section. 

Isn't that exactly what RAII does?

> What I need is something similar to open(),
> which returs a file. When the last reference to that object goes out of
> scope, the underlying file object is closed.

You can go ahead and implement a __del__() method. It will often work in 
CPython, but you get no guarantees, especially when you have reference 
cycles and with other Python implementations that don't use refcounting.

Peter

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: A question about fill_free_list(void) function

[Pedram]

On Jul 2, 1:11 pm, Peter Otten <__pete...@web.de> wrote:
> Pedram wrote:
> > On Jul 1, 10:01 pm, Christian Heimes  wrote:
> >> Pedram schrieb:
>
> >> > Hello community,
> >> > I'm reading the CPython interpreter source code,
> >> > first, if you have something that I should know for better reading
> >> > this source code, I would much appreciate that :)
> >> > second, in intobject.c file, I read the following code in
> >> > fill_free_list function that I couldn't understand:
> >> > while (--q > p)
> >> > Py_TYPE(q) = (struct _typeobject *)(q-1);
> >> > Py_TYPE(q) = NULL;
> >> > 
> >> > What does it mean?
>
> >> The code is abusing the ob_type member to store a single linked, NULL
> >> terminated list of free integer objects. It's a tricky optimization you
> >> don't have to understand in order to understand the rest of the code.
>
> >> And no, the code in ctypes.h is totally unrelated to the free list in
> >> intobject.c.
>
> >> Christian
>
> > Thanks for reply.
> > I totally understood the fill_free_list() function. Thanks.
> > But I have problems in understanding the PyInt_FromLong(). Here's the
> > code:
>
> > PyObject *
> > PyInt_FromLong(long ival)
> > {
> > register PyIntObject *v;
> > #if NSMALLNEGINTS + NSMALLPOSINTS > 0
> > if (-NSMALLNEGINTS <= ival && ival < NSMALLPOSINTS) {
> > v = small_ints[ival + NSMALLNEGINTS];
> > Py_INCREF(v);
> > #ifdef COUNT_ALLOCS
> > if (ival >= 0)
> > quick_int_allocs++;
> > else
> > quick_neg_int_allocs++;
> > #endif
> > return (PyObject *) v;
> > }
> > #endif
> > if (free_list == NULL) {
> > if ((free_list = fill_free_list()) == NULL)
> > return NULL;
> > }
>
> > v = free_list; // <-- No problem till here :)
> > free_list = (PyIntObject *)Py_TYPE(v);
> > PyObject_INIT(v, &PyInt_Type);
> > v->ob_ival = ival;
> > return (PyObject *) v;
> > }
>
> > I understood that small numbers (between -5 and 256) are referenced to
> > small_ints list. Others have to store in PyIntBlock, am I right? OK,
> > no problem till 'v = free_list;' but can't understand the line
> > 'free_list = (PyIntObject *)Py_TYPE(v);' :(. Where free_list
> > referenced to? Don't this correct that free_list have to be referenced
> > to another free block so next time the function called v store in this
> > location?
> > Sorry for my bad English. I hope you could understand me :)
>
> Well, if I understand fill_free_list() correctly, you haven't ;) It only
> makes sense together with the allocation code in PyInt_FromLong().
>
> Py_TYPE(v) translates to v->ob_type, and in fill_free_list() that field was
> abused(*) to hold a pointer to the previous item in the array of free
> integers, thereby temporarily turning it into a linked list.
>
> Assuming that free_list pointed to block->objects[10] in the current
> PyIntBlock it will now point to block->objects[9].
>
> When block->objects[0] is reached free_list will be set to NULL as block-
>
> >objects[0]->ob_type was initialized with NULL by the line
>
> Py_TYPE(q) = NULL; /* in fill_free_list() */
>
> The next call of PyInt_FromLong() will then trigger the allocation of a new
> block.
>
> Peter
>
> (*) It may help if you think of the objects array as a
>
> union linked {
>     linked *next;
>     PyInt_Object obj;
>
>
>
>
>
> };
>
>

Oh, I got it! What a wonderful implementation! :o
Thanks so much Peter. You made my day :) I didn't read the code
carefully.

>
> ;)
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Why re.match()?

[kj]

In  Steven D'Aprano 
 writes:

>On Thu, 02 Jul 2009 03:49:57 +, kj wrote:

>> In  Duncan Booth
>>  writes:
>>>So, for example:
>> 
>> re.compile("c").match("abcdef", 2)
>>><_sre.SRE_Match object at 0x02C09B90>
>> re.compile("^c").search("abcdef", 2)
>>
>>
>> I find this unconvincing; with re.search alone one could simply do:
>> 
> re.compile("^c").search("abcdef"[2:])
>> <_sre.SRE_Match object at 0x75918>
>> 
>> No need for re.match(), at least as far as your example shows.

>Your source string "abcdef" is tiny. Consider the case where the source 
>string is 4GB of data. You want to duplicate the whole lot, minus two 
>characters. Not so easy now.

I'm sure that it is possible to find cases in which the *current*
implementation of re.search() would be inefficient, but that's
because this implementation is perverse, which, I guess, is ultimately
the point of my original post.  Why privilege the special case of
a start-of-string anchor?  What if you wanted to apply an end-anchored
pattern to some prefix of your 4GB string?  Why not have a special
re method for that?  And another for every possible special case?

If the concern is efficiency for such cases, then simply implement
optional offset and length parameters for re.search(), to specify
any arbitrary substring to apply the search to.  To have a special-case
re.match() method in addition to a general re.search() method is
antithetical to language minimalism, and plain-old bizarre.  Maybe
there's a really good reason for it, but it has not been mentioned
yet.

kj
-- 
http://mail.python.org/mailman/listinfo/python-list

performance problem with time.strptime()

[Nils Rüttershoff]

Hi everyone,

In my free time I translate scripts from open source projects or write
my own, to train my python skills. ATM I convert the aplogmerge.pl from
awstats. It merges multiple apache logfiles and sort the output by the
timestamps of each line.  My first version of this script hasn't a good
performance, so I started profiling. It turned out that the script spend
much time for converting the timestamps of the line into a struct_time
object. Here a code example (Python 2.6.2 on Ubuntu 7.10):

Rec = 
re.compile(r"^\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}\s-\s\d+\s\[(\d{2}/\w+/\d{4}:\d{2}:\d{2}:\d{2})\s\+\d{4}\].*")
Line = '1.2.3.4 - 4459 [02/Jul/2009:01:50:26 +0200] "GET /foo HTTP/1.0" 200 - 
"-" "www.example.org" "-" "-" "-"'

def strptime():
m = Rec.match(Line)
if m:
date_string = m.group(1)
# date_string example: '02/Jul/2009:01:50:26'
return time.strptime(date_string, "%d/%b/%Y:%H:%M:%S")

with timeit this functions takes approximate 125 sec and 29.004.081
function calls (I've configured timeit with 1.000.000 runs). A look at
the output of cProfile told me that more than the half time is spent in
locale.py:

  102   11.7120.000   19.5920.000 locale.py:316(normalize)

  1023.6390.000   23.2310.000 locale.py:382(_parse_localename)

  1025.1620.000   30.2710.000 locale.py:481(getlocale)



I studied the time documentation and thought that I had to set TZ in os
environ:

os.environ['TZ'] = 'Europe/Berlin'

time.set()


but that had no effect. I don't know why time.strptime() looks every
time for my locale. Maybe it's a problem with my OS locale... However.
I've introduced a work around, which works perfectly for my problem. For
time comparison I could use any sort of time representation and so I
convert to epoch:

# needed to translate month str to dig repr

Shortmonth = {'Jan' : '01',

  'Feb' : '02',

  'Mar' : '03',

  'Apr' : '04',

  'May' : '05',

  'Jun' : '06',

  'Jul' : '07',

  'Aug' : '08',

  'Sep' : '09',

  'Oct' : '10',

  'Nov' : '11',

  'Dec' : '12'}

Rec = 
re.compile(r"^\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}\s-\s\d+\s\[(?P\d{2})/(?P\w+)/(?P\d{4}):(?P\d{2}):(?P\d{2}):(?P\d{2})\s\+\d{4}\].*")

Line = '1.2.3.4 - 4459 [02/Jul/2009:01:50:26 +0200] "GET /foo HTTP/1.0" 200 - 
"-" "www.example.org" "-" "-" "-"'

def epoch():

m = Rec.match(Line)

if m:

result = m.groupdict()

date_tuple = (result["year"], Shortmonth[result["month"]], 
result["day"], result["hour"], result["min"], result["sec"], -1, -1, -1)

date_tuple = map(int,date_tuple)

return time.mktime(date_tuple)


with this workaround I had a speed up to 4 times; it tooks only 31 sec
with only 5.000.009 function calls. Maybe this helps some of you, who
had the similar problems with time conversion

...But one big question remains: Why time.strptime() checks everytime
the locale? had I missed something or could I have a problem with my OS
locale?

With python 3.1 there is no difference, unless that time.strptime() took
approx 12 sec longer... :(

regards, Nils



Here a complete test script:

#!/opt/python/2.6.2/bin/python

import time

import timeit

import cProfile

import re

# needed to tranlate month str to dig repr

Shortmonth = {'Jan' : '01',

  'Feb' : '02',

  'Mar' : '03',

  'Apr' : '04',

  'May' : '05',

  'Jun' : '06',

  'Jul' : '07',

  'Aug' : '08',

  'Sep' : '09',

  'Oct' : '10',

  'Nov' : '11',

  'Dec' : '12'}

Rec1 = 
re.compile(r"^\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}\s-\s\d+\s\[(?P\d{2})/(?P\w+)/(?P\d{4}):(?P\d{2}):(?P\d{2}):(?P\d{2})\s\+\d{4}\].*")

Rec2 = 
re.compile(r"^\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}\s-\s\d+\s\[(\d{2}/\w+/\d{4}:\d{2}:\d{2}:\d{2})\s\+\d{4}\].*")

Line = '1.2.3.4 - 4459 [02/Jul/2009:01:50:26 +0200] "GET /foo HTTP/1.0" 200 - 
"-" "www.example.org" "-" "-" "-"'

def epoch():

m = Rec1.match(Line)

if m:

result = m.groupdict()

date_tuple = (result["year"], Shortmonth[result["month"]], 
result["day"], result["hour"], result["min"], result["sec"], -1, -1, -1)

date_tuple = map(int,date_tuple)

return time.mktime(date_tuple)

def strptime():

m = Rec2.match(Line)

if m:

date_string = m.group(1)

return time.strptime(date_string, "%d/%b/%Y:%H:%M:%S")

if __name__ == "__main__":

t1 = timeit.Timer("epoch()","from __main__ import epoch")

t2 = timeit.Timer("strptime()", "from __main__ import strptime")

cProfile.run("t1.timeit();print")

print ""

cProfile.run("t2.timeit();print")


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Need Help

[Simon Brunning]

2009/7/2 Tengiz Davitadze :
> Hello. I can't find a wright mail address. If you can help me I need to get
> an information about UNICODE. I am georgian and I need to write programs on
> georgian language . If you can transfer this mail or send me a wright mail
> about encoding or unicode information.

Our chief link is
.
 and
. Our two links are
 and
. And
. Our *three* links are
,
, and
. And
. Our *four*...no...
*Amongst* our links Amongst our linkry are such elements as
,
 I'll come in
again.

-- 
Cheers,
Simon B.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: performance problem with time.strptime()

[Casey Webster]

On Jul 2, 7:30 am, Nils Rüttershoff  wrote:

> Rec = 
> re.compile(r"^\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}\s-\s\d+\s\[(\d{2}/\w+/\d{4}:\d{2}:\d{2}:\d{2})\s\+\d{4}\].*")
> Line = '1.2.3.4 - 4459 [02/Jul/2009:01:50:26 +0200] "GET /foo HTTP/1.0" 200 - 
> "-" "www.example.org" "-" "-" "-"'

I'm not sure how much it will help but if you are only using the regex
to get the date/time group element, it might be faster to replace the
regex with:

>>> date_string = Line.split()[3][1:-1]
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: PEP 376

[Tarek Ziadé]

2009/7/2 Joachim Strömbergson :
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Aloha!
>
> Richard Brodie wrote:
>> "Joachim Str�mbergson"  wrote in message
>> news:mailman.2422.1246418400.8015.python-l...@python.org...
>>
>>> Even so, choosing md5 in 2009 for something that (hopefully) will be
>>> used in years is a bad design decision. It creates a dependency for to
>>> an algorithm that all sensible recommendations point you to move away
>>> from.
>>
>> Why not write the field as algorithm:value?
>>
>> e.g. sha1:8590b685654367e3eba70dc00df7e45e88c21da4
>>
>> Installers can fallback to using hashlib.new(), so you can plug in a new
>> algorithm without changing the PEP or the installer code.
>
> +1
>
> Good idea and future proof as well as being simple.

The prefix is a good idea but since it's just a checksum to control
that the file hasn't changed
what's wrong with using a weak hash algorithm like md5 or now sha1 ?

If someone wants to modify a file of a distribution he can recreate
the checksum as well,
the only secured way to prevent that would be to use gpg keys but
isn't that overkill for what we need ?

e.g. making sure a file wasn't modified when distutils uninstalls a
distribution.

Tarek
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Searching equivalent to C++ RAII or deterministic destructors

[Dave Angel]

Ulrich Eckhardt wrote:

Hi!

I'm currently converting my bioware to handle Python code and I have
stumbled across a problem...

Simple scenario: I have a handle to a resource. This handle allows me to
manipulate the resource in various ways and it also represents ownership.
Now, when I put this into a class, instances to that class can be shared,
using Python's reference counting. What I'm missing is a way to
automatically release the resource, something which I would do in the
destructor in C++.

Any ideas how to solve this?

Uli

  
As someone else pointed out, 'with' is the first line of defense.  It 
makes the stuff you could already do with try/except/finally much easier 
to get right.


Look also at 'del' a command in the language which explicitly deletes an 
object.


But I'm guessing you want something that automatically deletes objects 
whenever the last reference disappears.  That's an implementation 
detail, not a language guarantee.  In particular CPython does what you 
want, by using reference counting.  That's the only Python I've used, so 
it's only hearsay when I say that other implementations, (maybe Cython 
or Jython) do not all work the same way.


In CPython, any object whose *last* reference goes away, will get 
immediately deleted.  So if you avoid circular references, it should do 
just what you want.  Orphaned circular references are caught by the 
garbage collector, which runs periodically, but is non-deterministic.


Two more caveats.  Exceptions tend to hang onto stuff in their near 
vicinity, and I can't tell you the algorithm for what happens.  But an 
explicit del in the except clause can probably handle that.


Finally, closures can hang onto stuff.  So if you have nested functions, 
or lambda functions, it's possible they're going to keep things longer 
than you realize.




--
http://mail.python.org/mailman/listinfo/python-list

Re: Why re.match()?

[Hrvoje Niksic]

kj  writes:

> For a recovering Perl-head like me it is difficult to understand
> why Python's re module offers both match and search.  Why not just
> use search with a beginning-of-string anchor?

I need re.match when parsing the whole string.  In that case I never
want to search through the string, but process the whole string with
some regulat expression, for example when tokenizing.  For example:

pos = 0
while pos != len(s):
match = TOKEN_RE.match(s, pos)
if match:
process_token(match)
pos = match.end()
else:
raise ParseError('invalid syntax at position %d' % pos)
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Searching equivalent to C++ RAII or deterministic destructors

[Christian Heimes]

Dave Angel wrote:
> Look also at 'del' a command in the language which explicitly deletes an
> object.

No, you are either explaining it the wrong way or you have been fallen
for a common misinterpretation of the del statement. The del statement
only removes the object from the current scope. This means it decreases
the reference count by one. It does *not* remove the object.

Christian

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: PEP368 and pixeliterators

[Casey Webster]

On Jul 2, 4:32 am, Joachim Strömbergson 
wrote:
> But, wouldn't it be more Pythonic and simpler to have an iterator that
> iterates over all pixels in an image? Starting with upper left corner
> and moving left-right and (line by line) to lower right. This would
> change the code above to:

Unless I'm totally misreading the PEP, the author does provide both
iterators.  Quoting the PEP:

Non-planar images offer the following additional methods:

pixels() -> iterator[pixel]

  Returns an iterator that iterates over all the pixels in the image,
  starting from the top line and scanning each line from left to
right.
  See below for a description of the pixel objects.

__iter__() -> iterator[line]

  Returns an iterator that iterates over all the lines in the image,
  from top to bottom. See below for a description of the line objects.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Searching equivalent to C++ RAII or deterministic destructors

[Peter Otten]

Dave Angel wrote:

> But I'm guessing you want something that automatically deletes objects
> whenever the last reference disappears.  That's an implementation
> detail, not a language guarantee.  In particular CPython does what you
> want, by using reference counting.  That's the only Python I've used, so
> it's only hearsay when I say that other implementations, (maybe Cython
> or Jython) do not all work the same way.

Here are some examples from Kubuntu 9.04's zoo of python implementations:

$ cat del.py
import sys
print sys.version

class A(object):
def __init__(self, x):
self.x = x
def __del__(self):
print "releasing A(%r)" % self.x

def f():
a = A("local in function")
f()

a = A("global (one ref)")

c = A("global (cycle)")
c.a = c
del c

b = A("global (no refs)")
del b

print "about to quit"

$ python del.py
2.6.2 (release26-maint, Apr 19 2009, 01:58:18)
[GCC 4.3.3]
releasing A('local in function')
releasing A('global (no refs)')
about to quit
releasing A('global (one ref)')

$ jython del.py
2.2.1
about to quit

$ ipy del.py
2.4.0 (IronPython 1.1.1 (1.1.1) on .NET 2.0.50727.42)
about to quit
releasing A('global (no refs)')
releasing A('global (cycle)')
releasing A('local in function')

Unhandled Exception: System.ArgumentException: I/O operation on closed file
  at IronPython.Runtime.PythonFile.ThrowIfClosed () [0x0]
  at IronPython.Runtime.PythonFile.Write (System.String s) [0x0]
$

IronPython sometimes segfaulted.

Peter

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: performance problem with time.strptime()

[Nils Rüttershoff]

Hi Casey
Casey Webster wrote:
> On Jul 2, 7:30 am, Nils Rüttershoff  wrote:
>
>   
>> Rec = 
>> re.compile(r"^\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}\s-\s\d+\s\[(\d{2}/\w+/\d{4}:\d{2}:\d{2}:\d{2})\s\+\d{4}\].*")
>> Line = '1.2.3.4 - 4459 [02/Jul/2009:01:50:26 +0200] "GET /foo HTTP/1.0" 200 
>> - "-" "www.example.org" "-" "-" "-"'
>> 
>
> I'm not sure how much it will help but if you are only using the regex
> to get the date/time group element, it might be faster to replace the
> regex with:
>
>   
 date_string = Line.split()[3][1:-1]
 

Indeed this would give a little speed up (by 100 iteration approx
3-4 sec). But this would be only a small piece of the cake. Although thx :)

The problem is that time.strptime() consult locale.py for each
iteration. Here the hole cProfile trace:

first with epoch and second with strptime (condensed):

 509 function calls in 33.084 CPU seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
10.0000.000   33.084   33.084 :1()
12.4172.417   33.084   33.084 :2(inner)
  1009.6480.000   30.6670.000 time_test.py:30(epoch)
10.0000.000   33.084   33.084 timeit.py:177(timeit)
  1003.7110.0003.7110.000 {built-in method groupdict}
  1004.3180.0004.3180.000 {built-in method match}
10.0000.0000.0000.000 {gc.disable}
10.0000.0000.0000.000 {gc.enable}
10.0000.0000.0000.000 {gc.isenabled}
  1007.7640.0007.7640.000 {map}
10.0000.0000.0000.000 {method 'disable' of
'_lsprof.Profiler' objects}
  1005.2250.0005.2250.000 {time.mktime}
20.0000.0000.0000.000 {time.time}



 2909 function calls in 124.449 CPU seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
10.0000.000  124.449  124.449 :1()
12.2442.244  124.449  124.449 :2(inner)
  1003.5000.000   33.5590.000 _strptime.py:27(_getlang)
  100   41.8140.000  100.7540.000 _strptime.py:295(_strptime)
  1004.0100.000  104.7640.000
_strptime.py:453(_strptime_time)
  100   11.6470.000   19.5290.000 locale.py:316(normalize)
  1003.6380.000   23.1670.000
locale.py:382(_parse_localename)
  1005.1200.000   30.0590.000 locale.py:481(getlocale)
  1007.2420.000  122.2050.000 time_test.py:37(strptime)
10.0000.000  124.449  124.449 timeit.py:177(timeit)
  1001.7710.0001.7710.000 {_locale.setlocale}
  1001.7350.0001.7350.000 {built-in method __enter__}
  1001.6260.0001.6260.000 {built-in method end}
  1003.8540.0003.8540.000 {built-in method groupdict}
  1001.6460.0001.6460.000 {built-in method group}
  2008.4090.0008.4090.000 {built-in method match}
10.0000.0000.0000.000 {gc.disable}
10.0000.0000.0000.000 {gc.enable}
10.0000.0000.0000.000 {gc.isenabled}
  2002.9420.0002.9420.000 {len}
10.0000.0000.0000.000 {method 'disable' of
'_lsprof.Profiler' objects}
  3004.5520.0004.5520.000 {method 'get' of 'dict'
objects}
  1002.0720.0002.0720.000 {method 'index' of 'list'
objects}
  1001.5170.0001.5170.000 {method 'iterkeys' of
'dict' objects}
  2003.1130.0003.1130.000 {method 'lower' of 'str'
objects}
  2003.2330.0003.2330.000 {method 'replace' of 'str'
objects}
  2002.9530.0002.9530.000 {method 'toordinal' of
'datetime.date' objects}
  1001.4760.0001.4760.000 {method 'weekday' of
'datetime.date' objects}
  1004.3320.000  109.0970.000 {time.strptime}
20.0000.0000.0000.000 {time.time}

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: performance problem with time.strptime()

[Dave Angel]

Nils Rüttershoff wrote:

Hi everyone,

In my free time I translate scripts from open source projects or write
my own, to train my python skills. ATM I convert the aplogmerge.pl from
awstats. It merges multiple apache logfiles and sort the output by the
timestamps of each line.  My first version of this script hasn't a good
performance, so I started profiling. It turned out that the script spend
much time for converting the timestamps of the line into a struct_time
object. Here a code example (Python 2.6.2 on Ubuntu 7.10):

Rec = 
re.compile(r"^\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}\s-\s\d+\s\[(\d{2}/\w+/\d{4}:\d{2}:\d{2}:\d{2})\s\+\d{4}\].*")
Line = '1.2.3.4 - 4459 [02/Jul/2009:01:50:26 +0200] "GET /foo HTTP/1.0" 200 - "-" "www.example.org" 
"-" "-" "-"'

def strptime():
m = Rec.match(Line)
if m:
date_string = m.group(1)
# date_string example: '02/Jul/2009:01:50:26'
return time.strptime(date_string, "%d/%b/%Y:%H:%M:%S")

with timeit this functions takes approximate 125 sec and 29.004.081
function calls (I've configured timeit with 1.000.000 runs). A look at
the output of cProfile told me that more than the half time is spent in
locale.py:

  102   11.7120.000   19.5920.000 locale.py:316(normalize)

  1023.6390.000   23.2310.000 locale.py:382(_parse_localename)

  1025.1620.000   30.2710.000 locale.py:481(getlocale)



I studied the time documentation and thought that I had to set TZ in os
environ:

os.environ['TZ'] = 'Europe/Berlin'

time.set()


but that had no effect. I don't know why time.strptime() looks every
time for my locale. Maybe it's a problem with my OS locale... However.
I've introduced a work around, which works perfectly for my problem. For
time comparison I could use any sort of time representation and so I
convert to epoch:

# needed to translate month str to dig repr

Shortmonth = {'Jan' : '01',

  'Feb' : '02',

  'Mar' : '03',

  'Apr' : '04',

  'May' : '05',

  'Jun' : '06',

  'Jul' : '07',

  'Aug' : '08',

  'Sep' : '09',

  'Oct' : '10',

  'Nov' : '11',

  'Dec' : '12'}

Rec = 
re.compile(r"^\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}\s-\s\d+\s\[(?P\d{2})/(?P\w+)/(?P\d{4}):(?P\d{2}):(?P\d{2}):(?P\d{2})\s\+\d{4}\].*")

Line = '1.2.3.4 - 4459 [02/Jul/2009:01:50:26 +0200] "GET /foo HTTP/1.0" 200 - "-" "www.example.org" 
"-" "-" "-"'

def epoch():

m = Rec.match(Line)

if m:

result = m.groupdict()

date_tuple = (result["year"], Shortmonth[result["month"]], result["day"], result["hour"], 
result["min"], result["sec"], -1, -1, -1)

date_tuple = map(int,date_tuple)

return time.mktime(date_tuple)


with this workaround I had a speed up to 4 times; it tooks only 31 sec
with only 5.000.009 function calls. Maybe this helps some of you, who
had the similar problems with time conversion

...But one big question remains: Why time.strptime() checks everytime
the locale? had I missed something or could I have a problem with my OS
locale?

With python 3.1 there is no difference, unless that time.strptime() took
approx 12 sec longer... :(

regards, Nils



Here a complete test script:

#!/opt/python/2.6.2/bin/python

import time

import timeit

import cProfile

import re

# needed to tranlate month str to dig repr

Shortmonth = {'Jan' : '01',

  'Feb' : '02',

  'Mar' : '03',

  'Apr' : '04',

  'May' : '05',

  'Jun' : '06',

  'Jul' : '07',

  'Aug' : '08',

  'Sep' : '09',

  'Oct' : '10',

  'Nov' : '11',

  'Dec' : '12'}

Rec1 = 
re.compile(r"^\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}\s-\s\d+\s\[(?P\d{2})/(?P\w+)/(?P\d{4}):(?P\d{2}):(?P\d{2}):(?P\d{2})\s\+\d{4}\].*")

Rec2 = 
re.compile(r"^\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}\s-\s\d+\s\[(\d{2}/\w+/\d{4}:\d{2}:\d{2}:\d{2})\s\+\d{4}\].*")

Line = '1.2.3.4 - 4459 [02/Jul/2009:01:50:26 +0200] "GET /foo HTTP/1.0" 200 - "-" "www.example.org" 
"-" "-" "-"'

def epoch():

m = Rec1.match(Line)

if m:

result = m.groupdict()

date_tuple = (result["year"], Shortmonth[result["month"]], result["day"], result["hour"], 
result["min"], result["sec"], -1, -1, -1)

date_tuple = map(int,date_tuple)

return time.mktime(date_tuple)

def strptime():

m = Rec2.match(Line)

if m:

date_string = m.group(1)

return time.strptime(date_string, "%d/%b/%Y:%H:%M:%S")

if __name__ == "__main__":

t1 = timeit.Timer("epoch()","from __main__ import epoch")

t2 = timeit.Timer("strptime()", "from __main__ import strptime")

cProfile.run("t1.timeit();print")

print ""

cProfile.run("t2.timeit();print")



  
As you say, if you don't actually need the datetime fields, why waste 
time generating them.  You gained a lot more than the time spent in 
local

Re: PEP 376

[Joachim Strömbergson]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Aloha!

Tarek Ziadé wrote:
> The prefix is a good idea but since it's just a checksum to control
> that the file hasn't changed
> what's wrong with using a weak hash algorithm like md5 or now sha1 ?

Because it creates a dependency to an old algorithm that should be
deprecated. Also using MD5, even for a thing like this might make people
belive that it is an ok algorithm to use - "Hey, it is used by the
default install in Python, so it must be ok, right?"

If we flip the argument around: Why would you want to use MD5 instead of
SHA-256? For the specific use case the performance will not (should not)
be an issue.

As I wrote a few mails ago, it is time to move forward from MD5 and
designing something in 2009 that will be around for many years that uses
MD5 is (IMHO) a bad design decision.

> If someone wants to modify a file of a distribution he can recreate
> the checksum as well,
> the only secured way to prevent that would be to use gpg keys but
> isn't that overkill for what we need ?

Actually, adding this type of security would IMHO be a good idea.

- --
Med vänlig hälsning, Yours

Joachim Strömbergson - Alltid i harmonisk svängning.

Kryptoblog - IT-säkerhet på svenska
http://www.strombergson.com/kryptoblog

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.8 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkpMsuYACgkQZoPr8HT30QELagCghfYyHyK5jnkS8DlaQ2ZX4KR8
W+YAniWSvWRvm47/xGu0thTaYioETY94
=2x3X
-END PGP SIGNATURE-
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: PEP368 and pixeliterators

[Joachim Strömbergson]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Aloha!

Casey Webster wrote:
> Unless I'm totally misreading the PEP, the author does provide both
> iterators.  Quoting the PEP:
> 
> Non-planar images offer the following additional methods:
> 
> pixels() -> iterator[pixel]
> 
>   Returns an iterator that iterates over all the pixels in the image,
>   starting from the top line and scanning each line from left to
> right.
>   See below for a description of the pixel objects.

Ah! I knew there was a reason for me wearing glasses. ;-)

Go PEP368!

- --
Med vänlig hälsning, Yours

Joachim Strömbergson - Alltid i harmonisk svängning.

Kryptoblog - IT-säkerhet på svenska
http://www.strombergson.com/kryptoblog

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.8 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkpMszQACgkQZoPr8HT30QGmVACfezmlZ+K93exuIEoUDJyzNzgM
3dcAn1x4Ghu9AlMKEme2og4sK4TwhXaR
=bNbl
-END PGP SIGNATURE-
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Re: Searching equivalent to C++ RAII or deterministic destructors

[Dave Angel]

Christian Heimes wrote:

Dave Angel wrote:
  

Look also at 'del' a command in the language which explicitly deletes an
object.



No, you are either explaining it the wrong way or you have been fallen
for a common misinterpretation of the del statement. The del statement
only removes the object from the current scope. This means it decreases
the reference count by one. It does *not* remove the object.

Christian


  
You're right of course.  What I was trying to say was it deletes the 
reference to the object.  Unlike obj = None, del obj removes the 
reference (attribute) entirely. Although I don't know what it should be 
called if it's a local variable.  Perhaps it "unbinds" the name.


DaveA

--
http://mail.python.org/mailman/listinfo/python-list

Re: Trying to use sets for random selection, but the pop() method returns items in order

[Mario Garcia]

This could be better:

>>> import random
>>> population = range(10)
>>> choice = random.choice(population)
>>> population.remove(choice)
>>> print population
>>> print population
[0, 1, 2, 3, 4, 5, 6, 8, 9]

That was my idea with the previous pop(), remove from the population a
certain number of elements at random.
In the docs pop is defined as:
 Remove and return an arbitrary element from the set.
My mistake: arbitrary is not the same as random :(

Mario


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: pep 8 constants

[Lie Ryan]

Eric S. Johansson wrote:
> Steven D'Aprano wrote:
>> That assumes that every word is all caps. In practice, for real-life 
>> Python code, I've tripled the vocal load of perhaps one percent of your 
>> utterances, which cuts your productivity by 2%.
>>
>> If you have 1 words in you per day, and one percent get wrapped with 
>> a leading and trailing "capslock", you have:
>>
>> 9900 regular words plus 100 constants
>>
>> versus
>>
>> 9700 regular words plus 100 constants plus 200 by capslock.
>>
>> Your productivity goes down by 200 words out of 10,000, or two percent.
> 
> oh I wish it was so simple. I would absolutely you if it was that simple the
> real world. Somehow, I fear that you wouldn't understand unless you sat next 
> to
> me and watch me try to write code to  a variety of standards.
> 
> In practice, my experience says that with a really good  speech driven
> environment for programming, IT, vocal load probably by a quarter to a half 
> and
> be more compliant with current fashion for naming.

I've tried (Windows') speech recognition some time before, and I know
it's one heck of a mess trying to type anything with it. I have a fully
working hand, so I don't actually need a voice recognition, but I
remember trying to write a single (not code) paragraph solely by
recognition. It took me over an hour correcting all those mistakes and
it really starts to gets into the nerves after the first sentence. I did
a little victory dance, after doing the first paragraph.

I know they gets better with time as they learn our speech pattern and
we learn how to pronounce the way the voice recognition expects. But one
thing I can't imagine is using voice recognition for writing code
(heck... that'll be a nightmare inside a nightmare)

> Heck, have you ever noticed how most Python smart editors can't even
> indent properly according to local contexts. Emacs is the only one and
> even that one sometimes fails
 I use kwrite for editing, and I can't say I've ever noticed a failure.
>>> okay, it fails from a speech recognition user perspective. Put the
>>> cursor on the line and hit the tab key. Insert spaces/tabs in the line
>>> of text. Not good for speech recognition. If  mess of indentation and
>>> hit the tab key, it doesn't automatically indent to the right location.
>>> I'll be damned if I'm in the sit there speaking
>>> "tabkeytabkeytabkeytabkeytabkey" because the error should take care of
>>> it for me and save my voice. Like I said, Emacs does it correctly and
>>> nobody else does as far as I can tell.
>> I don't understand what you're describing.
>>
>> If you set kwrite to use "Python style" indent mode, then starting a new 
>> line will automatically indent to either the current indent level, or one 
>> extra indent level if the line ends with a colon. You shouldn't need to 
>> indent forward more than one level at a time when writing Python code, 
>> although you will need to dedent multiple levels on occasion.
> 
> I recognize your confusion. It's very common when describing this problem. If 
> I
> had some way of making a movie of what  and describing, obviously be more 
> clear.
> I'll see if I can make a movie with my camera that demonstrates the problem.  
> It
> probably won't happen fast but, what I tried your editor, it failed   with a
> trivial effort.  I think I just hit the tab key multiple times in different
> places on the line. In all cases it did the wrong thing. For me, the right 
> thing
> is to force the line to the right indentation based on the previous line to
> matter how may times you get the tab key or where you are on that line when 
> you
> hit the tab key. I could be in the middle of the line, the end of line, start 
> a
> line, the start of the string and if I hit a tab key, that line should go to
> exactly the right place stayed there no matter how me more times I typed a 
> tab key.
> 
> But this might be what you're getting confused. What I'm really asking for is
> the ability to say "force indent" and have the line indent exactly correctly
> based on context. It doesn't have to be the tab key. The tab key is the method
> of convenience.

Maybe we need an editor that could sort of understand something like:
"constant red"
and apply the appropriate code-sensitive convention transformation (in
python it'll be RED)
"class foo inheriting object"
and typed "class Foo(object):\n\t"
"indent", "dedent"
put a tab or delete a tab (or softtab)

but for all those, we're going to need a specialized text editor or
perhaps a context-sensitive voice recognition software. The simplest
solution would be a vim plugin or emacs mode that is specially tailored
to recognize these commands and apply the appropriate transformations
depending on the contexts.

>> I've already said it's your right to reject the convention for your own 
>> code. Go right ahead. This is only a problem if you're editing other 
>> people's code which is using uppercase constants.
> 
> exactly. If I'm 

Re: pep 8 constants

[Lie Ryan]

Eric S. Johansson wrote:

> I've been working with speech recognition for 15 years. I've written something
> on the order of 10,000 lines of Python code both as open source and private
> projects. I've tried it least two dozen editors and they all fail miserably
> because they're focused on keyboard use (but understandable) I get good
> recognition accuracy because I train the system and then I let it train me.
>> For coding, you might want to investigate a tool like Dasher[1] which
>> offers an alternate form of input.  It allows for custom
>> vocabularies/keymaps if you need, as well as more precise specification
>> of a full keyboard (caps vs. mixed-case, specific punctuation
>> characters, etc).  The predictive entry should be smart enough to pick
>> up previously entered constants/terms saving you entry speed.  It can
>> also be driven by a wide variety of pointing devices (mouse, trackball,
>> touchpad, head-tracker, gyro-input, etc).
> 
> I've tried it, it's quite promising but my hands term are sufficiently that I
> can't target accurately. This is a problem for me with mice as well. Maybe 
> these
> tiny little spit dot icons and webpages drive me insane because it takes me 
> two
> or three tries to put the right spot.

You might want to try eye-ball tracking device. Dashers are designed for
that as well. You won't need to use your hand to type with it.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Timeout when connecting to sybase DBS

[eranlevi]

On Jul 2, 1:08 am, s...@pobox.com wrote:
>     Gil> There's no such group as python-sybase :-(
>
> http://sourceforge.net/mailarchive/forum.php?forum_name=python-sybase...
>
> S

Thanks :-)
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Timeout when connecting to sybase DBS

[eranlevi]

On Jul 2, 1:07 am, s...@pobox.com wrote:
>     Gil> Are you saying, that when you trying to connect to a sybase DBS
>     Gil> server and the DBS or the server is down, you get an error after a
>     Gil> few seconds and not after a few minutes?
>
> Yes, though thankfully our server tends to almost always be up.
>
>     Gil> Which python version are you using?
>
> We're still running 2.4.5 at work and have a slightly hacked very old
> version of the python-sybase package (0.36).
>
> --
> Skip Montanaro - s...@pobox.com -http://www.smontanaro.net/
>     when i wake up with a heart rate below 40, i head right for the espresso
>     machine. -- chaos @ forums.usms.org

That is really strange :-(
I'll ask in the python-sybase group

Thanks
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: PEP368 and pixeliterators

[Nobody]

On Thu, 02 Jul 2009 10:32:04 +0200, Joachim Strömbergson wrote:

> I just read the PEP368 and really liked the proposed idea, sound like a
> great battery addition to include in the std lib:
> 
> http://www.python.org/dev/peps/pep-0368/

Unfortunately, it's too simplistic, meaning that most of the existing
image formats used by the various GUI toolkits (and OpenGL) will need to
be retained.

Obvious omissions include support for arbitrary numbers of bits per
component (e.g. R5G5B5A1), component, pixel and row strides (padding),
mixing packed and planar components (e.g. packed RGB with a separate alpha
plane), and storage orientation (top-to-bottom or bottom-to-top).

Ideally, an Image should be able to use any sane storage format, so that
you don't have to keep converting the Image to an application-specific
format.

NumPy's arrays would probably be a good model here. Essentially, each
component would have a base pointer plus horizontal and vertical strides
measured in bits.

The strides wouldn't necessarily be the same for all components (allowing
for an R8G8B8 plane with a 3-byte stride plus a distinct A8 plane with a
1-byte stride), vertical stride wouldn't necessarily be greater than
horizontal stride (allowing transposition), strides could be negative
(allowing mirroring), and could be less than a byte (allowing e.g. a 1-bpp
alpha channel, planar 4-bpp data, 12-bpp greyscale, etc).

AFAICT, this would allow all common image storage formats to be
represented. The most common exception is 8-bit RGB data using a 6x6x6
colour cube, but that's not common enough to be worth the added complexity.

The interface should also be flexible enough to allow an Image to be a
"proxy" for a non-local image (e.g. one held in video RAM or in the X
server). You wouldn't be able to get a buffer for the image data, but you
should at least be able to obtain the dimensions and format of such
images, and ideally copy the data (or rectangular subregions) to/from a
local image.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Idioms and Anti-Idioms Question

[J. Cliff Dyer]

On Wed, 2009-07-01 at 17:19 +1200, Lawrence D'Oliveiro wrote:
> In message , J. Cliff 
> Dyer wrote:
> 
> > If the lines got separated, a leading + could disappear into its line
> > without any errors showing up.  A trailing + would raise a syntax error.
> 
> Unless, of course, it was moved onto the previous line as part of whatever 
> caused the separation of the lines. How would you guard against that?
> 

Can you give an example of what you mean?

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Spam? Re: whizBase vs. Python

[Nils Rüttershoff]

NurAzije wrote:
> On Jun 29, 11:04 am, Tim Harig  wrote:
>   
>> On 2009-06-29, NurAzije  wrote:
>>
>> 
>>> Hi,
>>> I am working on a study and I need expert opinion, I did not work with
>>> Python before, can anyone help me with a comparison betweenWhizBase
>>> (www.whizbase.com) and Python please.
>>>   
>> Given posts 
>> like:http://groups.google.com/group/Server-side-programing/browse_thread/t...
>> is this just thinly veiled spam?  The domain and the posting IP address are
>> both out of Sarajevo, Bosnia.
>>
>> Obviously you alreadly know your product is nothing but a database
>> macro processor.  Python is a dynamic programming language and therefore
>> far more capable.
>>
>> Your product is proprietary with less capability while Python is free and
>> powerful.
>> 
>
> Hi Tim,
> I am not a spammer, WhizBase is from Sarajevo and I am from Sarajevo,
> for that it is interesting for me.
>
> regards,
> Ashraf Gheith
>   
Hi Ashraf,

hm... but it seems you know WhizBase very well, here a blog
(http://whizbase-scripting-language.blogspot.com/) of you with very
detailed information about WhizBase. Why you are asking nearly any
programming list/group about the difference?:

http://www.highdots.com/forums/javascript-discussion-multi-lingual/whizbase-vs-javascript-281242.html
http://groups.google.com/group/alt.comp.lang.learn.c-c++/browse_thread/thread/a7a47adb904319ad
http://groups.google.com/group/comp.lang.lisp/browse_thread/thread/9dfa76ad94d75168/01a999b7b8638621?lnk=raot
http://www.mail-archive.com/dotnetdevelopm...@googlegroups.com/msg07381.html

Hm here the main blog:
http://whizbase.blogspot.com/

There is a second contributer Faik Djikic and Whizbase is from a company
called "Djikic Software Development"..

What's the Topic of your study? Whizbase compared to other languages?
Why Whizbase? Is your study public and if so were we could read it? ;)

Regards, Nils
-- 
http://mail.python.org/mailman/listinfo/python-list

logging of strings with broken encoding

[Thomas Guettler]

Hi,

I have bug in my code, which results in the same error has this one:

https://bugs.launchpad.net/bzr/+bug/295653
{{{
Traceback (most recent call last):
  File "/usr/lib/python2.6/logging/__init__.py", line 765, in emit
self.stream.write(fs % msg.encode("UTF-8"))
  ..
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 8: ordinal 
not in range(128)
}}}

I run Python 2.6. In SVN the code is the same (StreamHandler ... def emit...):
http://svn.python.org/view/python/branches/release26-maint/Lib/logging/__init__.py?revision=72507&view=markup

I think msg.encode("UTF-8", 'backslashreplace') would be better here.

What do you think?

Should I fill a bugreport?

  Thomas


-- 
Thomas Guettler, http://www.thomas-guettler.de/
E-Mail: guettli (*) thomas-guettler + de
-- 
http://mail.python.org/mailman/listinfo/python-list

multiprocessing: pool with blocking queue

[masher]

Hi,

I am trying to implement a multiprocessing pool that assigns tasks
from a blocking queue. My situation is a pretty classic producer/
consumer conundrum, where the producer can produce much faster than
the consumers can consume. The wrinkle in the story is that the
producer produces objects that consume large amounts of memory, so I
would like the queue to block when it reaches, say, twice the number
of tasks as there are processes in the pool, so that I don't devour my
RAM. I have been able to implement this one way, but I am somewhat
displeased with the result. Code speaks louder than words, so here is
a brief example of the technique I am using, followed by an
explanation of why I am unhappy with it:

===
#!/usr/bin/env python3

import time
import random
from multiprocessing import Pool, Queue, M

def procfunc(queue):
time.sleep(random.random() * 2)
return queue.get()*2

def printrange(n):
for i in range(n):
print("generated " + str(i))
yield i

if __name__ == "__main__":
sm = Manager()
pool = Pool()
queuelen = len(pool._pool) * 2
queue = sm.Queue(queuelen)
for i in printrange(100):
queue.put(i)
pool.apply_async(procfunc, (queue,), callback=print)
pool.close()
pool.join()
===

The reason I am unhappy with this trick is that if you examine the
source code of pool.py, you will note that the class Pool already uses
an internal queue, "_taskqueue" from which the tasks are assigned to
processes in the pool.

Particularly:
def __init__(self, processes=None, initializer=None, initargs=()):
self._setup_queues()
self._taskqueue = queue.Queue()
...snip

It seems to me that if I could only subclass and do
  queuelen = len(pool._pool) * 2
  self._taskqueue = queue.Queue(queuelen)

later in the constructor, once the pool length has been established, I
would have a much more elegant, transparent solution to the problem.

Unfortunately, the design of the Pool class is such that actually
implementing this solution would be very hackish and inelegant. If
only, say, _setup_queues() were called after the _taskqueue
assignment, then I could override it.

My questions, then, is: Is there a more elegant/pythonic way of doing
what I am trying to do with the current Pool class?

If the verdict is no, I'll be happy to file a bug report.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: logging of strings with broken encoding

[David Smith]

Thomas Guettler wrote:
> Hi,
> 
> I have bug in my code, which results in the same error has this one:
> 
> https://bugs.launchpad.net/bzr/+bug/295653
> {{{
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/logging/__init__.py", line 765, in emit
> self.stream.write(fs % msg.encode("UTF-8"))
>   ..
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 8: 
> ordinal not in range(128)
> }}}
> 
> I run Python 2.6. In SVN the code is the same (StreamHandler ... def emit...):
> http://svn.python.org/view/python/branches/release26-maint/Lib/logging/__init__.py?revision=72507&view=markup
> 
> I think msg.encode("UTF-8", 'backslashreplace') would be better here.
> 
> What do you think?
> 
> Should I fill a bugreport?
> 
>   Thomas
> 
> 

I think you have to decode it first using the strings original encoding
whether that be cp1252 or mac-roman or any of the other 8-bit encodings.
 Once that's done, you can encode in UTF-8

--David
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: logging of strings with broken encoding

[Peter Otten]

Thomas Guettler wrote:

> I have bug in my code, which results in the same error has this one:
> 
> https://bugs.launchpad.net/bzr/+bug/295653
> {{{
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/logging/__init__.py", line 765, in emit
> self.stream.write(fs % msg.encode("UTF-8"))
>   ..
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 8:
> ordinal not in range(128) }}}
> 
> I run Python 2.6. In SVN the code is the same (StreamHandler ... def
> emit...):
> http://svn.python.org/view/python/branches/release26-
maint/Lib/logging/__init__.py?revision=72507&view=markup
> 
> I think msg.encode("UTF-8", 'backslashreplace') would be better here.
> 
> What do you think?

That won't help. It's a *decoding* error. You are feeding it a non-ascii 
byte string.

Peter

-- 
http://mail.python.org/mailman/listinfo/python-list

MySQLdb and ordering of column names in list returned by keys() w/ a DictCursor

[Wells Oliver]

Is there some kind of mysterious logic to how the the columns are ordered
when executing the following:

sql = "SELECT player_id, SUM(K) AS K, SUM(IP) AS IP, SUM(ER) AS ER, SUM(HR)
AS HR, SUM(H) AS H, SUM(BB) AS BB, Teams.league FROM Pitching INNER JOIN
Teams ON Pitching.team = Teams.team_id WHERE Date BETWEEN '%s' AND '%s'
GROUP BY player_id" % (start, date)
cursor.execute(sql)

for row in cursor.fetchall():
print row.keys()

What I get is:

['league', 'BB', 'HR', 'IP', 'K', 'H', 'player_id', 'ER']

Neither alphabetical nor the order in which they were specified in the query
nor... any seeming order I can suss out. Any ideas? Thanks!

(cursor being a MySQLdb.cursors.DictCursor object.)

-- 
Wells Oliver
we...@submute.net
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: MySQLdb and ordering of column names in list returned by keys() w/ a DictCursor

[Tim Chase]

sql = "SELECT player_id, SUM(K) AS K, SUM(IP) AS IP, SUM(ER) AS ER, SUM(HR)
AS HR, SUM(H) AS H, SUM(BB) AS BB, Teams.league FROM Pitching INNER JOIN
Teams ON Pitching.team = Teams.team_id WHERE Date BETWEEN '%s' AND '%s'
GROUP BY player_id" % (start, date)
cursor.execute(sql)

for row in cursor.fetchall():
print row.keys()

What I get is:

['league', 'BB', 'HR', 'IP', 'K', 'H', 'player_id', 'ER']

Neither alphabetical nor the order in which they were specified in the query
nor... any seeming order I can suss out. Any ideas? Thanks!

(cursor being a MySQLdb.cursors.DictCursor object.)


My guess is you're experiencing the fact that dicts are unordered 
by nature which allows it to return in any order it likes 
(usually per the internal representation/storage).


-tkc



--
http://mail.python.org/mailman/listinfo/python-list

Re: logging of strings with broken encoding

[Lie Ryan]

Thomas Guettler wrote:
> Hi,
> 
> I have bug in my code, which results in the same error has this one:
> 
> https://bugs.launchpad.net/bzr/+bug/295653
> {{{
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/logging/__init__.py", line 765, in emit
> self.stream.write(fs % msg.encode("UTF-8"))
>   ..
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 8: 
> ordinal not in range(128)
> }}}

What's the encoding of self.stream? Is it sys.stdout/sys.stderr or a
file object?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: logging of strings with broken encoding

[Thomas Guettler]

My quick fix is this:

class MyFormatter(logging.Formatter):
def format(self, record):
msg=logging.Formatter.format(self, record)
if isinstance(msg, str):
msg=msg.decode('utf8', 'replace')
return msg

But I still think handling of non-ascii byte strings should be better.
A broken logging message is better than none.

And, if there is a UnicodeError, handleError() should not send the message
to sys.stderr, but it should use emit() of the current handler.

In my case sys.stderr gets discarded. Its very hard to debug, if you don't
see any logging messages.

  Thomas

Thomas Guettler schrieb:
> Hi,
> 
> I have bug in my code, which results in the same error has this one:
>  ...

-- 
Thomas Guettler, http://www.thomas-guettler.de/
E-Mail: guettli (*) thomas-guettler + de
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: MySQLdb and ordering of column names in list returned by keys() w/ a DictCursor

[Wells Oliver]

Will this order at least be the same for that same query every time the
script is executed?

On Thu, Jul 2, 2009 at 10:48 AM, Tim Chase wrote:

> sql = "SELECT player_id, SUM(K) AS K, SUM(IP) AS IP, SUM(ER) AS ER, SUM(HR)
>> AS HR, SUM(H) AS H, SUM(BB) AS BB, Teams.league FROM Pitching INNER JOIN
>> Teams ON Pitching.team = Teams.team_id WHERE Date BETWEEN '%s' AND '%s'
>> GROUP BY player_id" % (start, date)
>> cursor.execute(sql)
>>
>> for row in cursor.fetchall():
>>print row.keys()
>>
>> What I get is:
>>
>> ['league', 'BB', 'HR', 'IP', 'K', 'H', 'player_id', 'ER']
>>
>> Neither alphabetical nor the order in which they were specified in the
>> query
>> nor... any seeming order I can suss out. Any ideas? Thanks!
>>
>> (cursor being a MySQLdb.cursors.DictCursor object.)
>>
>
> My guess is you're experiencing the fact that dicts are unordered by nature
> which allows it to return in any order it likes (usually per the internal
> representation/storage).
>
> -tkc
>
>
>
>


-- 
Wells Oliver
we...@submute.net
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: multiprocessing: pool with blocking queue

[J Kenneth King]

masher  writes:

> My questions, then, is: Is there a more elegant/pythonic way of doing
> what I am trying to do with the current Pool class?

Forgive me, I may not fully understand what you are trying to do here
(I've never really used multiprocessing all that much)...

But couldn't you just assign your own Queue object to the Pool instance?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: deleting certain entries in numpy array

[Robert Kern]

On 2009-07-02 04:40, Sebastian Schabe wrote:

Robert Kern schrieb:



 > You will want to ask numpy questions on the numpy mailing list.
 >
 > http://www.scipy.org/Mailing_Lists
 >

I ever thought news-groups are the right way for questions like this.
And the mailing list page just confuses me, but I'm trying to get used
to it.


You may want to try GMane, then:

  http://dir.gmane.org/gmane.comp.python.numeric.general

It's how I read this list, for instance.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

--
http://mail.python.org/mailman/listinfo/python-list

Re: multiprocessing: pool with blocking queue

[masher]

On Jul 2, 12:06 pm, J Kenneth King  wrote:
> masher  writes:
> > My questions, then, is: Is there a more elegant/pythonic way of doing
> > what I am trying to do with the current Pool class?
>
> Forgive me, I may not fully understand what you are trying to do here
> (I've never really used multiprocessing all that much)...
>
> But couldn't you just assign your own Queue object to the Pool instance?

That's basically my question. It does not appear as though there is
any straightforward way of doing this because of the design of Pool's
__init__ method, which passes _taskqueue to several functions. Hence,
even if I were to reassign _taskqueue after __init__, that wouldn't
change anything.
-- 
http://mail.python.org/mailman/listinfo/python-list

Dict ordering [was: Re: MySQLdb and ordering of column names in list returned by keys() w/ a DictCursor]

[Tim Wintle]

On Thu, 2009-07-02 at 10:32 -0500, Wells Oliver wrote:
> for row in cursor.fetchall():
> print row.keys()
> 
> What I get is:
> 
> ['league', 'BB', 'HR', 'IP', 'K', 'H', 'player_id', 'ER']
> 
> Neither alphabetical nor the order in which they were specified in the
> query nor... any seeming order I can suss out. Any ideas? Thanks!

keys in a dict are not guaranteed to be in any specific order. It's not
specific to the MySQLdb cursor.

IIRC it's related to the hashes of the keys (but obviously not in
strictly increasing order). You definitely shouldn't rely on the
ordering (hence the need for an OrderedDict.

e.g. 

>>> a = ['league', 'BB', 'HR', 'IP', 'K', 'H', 'player_id', 'ER']
>>> for s in a:
...   print hash(s)
... 
-3369365635700083487
8448050754076166
9216055368083080
9344056137084361
9600028874
9216027721
1844482854915224472
8832053061079775

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: MySQLdb and ordering of column names in list returned by keys() w/ a DictCursor

[Tim Chase]

Will this order at least be the same for that same query every time the
script is executed?


I wouldn't count on it.  The order is only defined for the one 
iteration (result of the keys() call).  If the order matters, I'd 
suggest a double-dispatch with a non-dict (regular/default) query 
result, something like


  header_map = dict(
(desc[0], i)
for i,desc in enumerate(cursor.description)
)
  for row in cursor.fetchall():
print row[header_map['league']]

If you have a lot of fields, I often use a closure to simplify 
the coding:


  for row in cursor.fetchall():
item = lambda s: row[headermap[s]]
print item('league')
print item('BB')
print item('player_id')

That way, the row stays in order, if you need it that way:

  league, bb, hr, ip, k, h, id, er = row

-tkc



--
http://mail.python.org/mailman/listinfo/python-list

regex question on .findall and \b

[Ethan Furman]

Greetings!

My closest to successfull attempt:

Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]
Type "copyright", "credits" or "license" for more information.

IPython 0.9.1 -- An enhanced Interactive Python.

  In [161]: re.findall('\d+','this is test a3 attempt 79')
  Out[161]: ['3', '79']

What I really want in just the 79, as a3 is not a decimal number, but 
when I add the \b word boundaries I get:


  In [162]: re.findall('\b\d+\b','this is test a3 attempt 79')
  Out[162]: []

What am I missing?

~Ethan~
--
http://mail.python.org/mailman/listinfo/python-list

Re: logging of strings with broken encoding

[Stefan Behnel]

Thomas Guettler wrote:
> My quick fix is this:
> 
> class MyFormatter(logging.Formatter):
> def format(self, record):
> msg=logging.Formatter.format(self, record)
> if isinstance(msg, str):
> msg=msg.decode('utf8', 'replace')
> return msg
> 
> But I still think handling of non-ascii byte strings should be better.
> A broken logging message is better than none.

Erm, may I note that this is not a problem in the logging library but in
the code that uses it? How should the logging library know what you meant
by passing that byte string in the first place? And where is the difference
between accidentally passing a byte string and accidentally passing another
non-printable object? Handling this "better" may simply hide the bugs in
your code, I don't find that's any "better" at all.

Anyway, this has been fixed in Py3.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: regex question on .findall and \b

[Tim Chase]

Ethan Furman wrote:

Greetings!

My closest to successfull attempt:

Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]
Type "copyright", "credits" or "license" for more information.

IPython 0.9.1 -- An enhanced Interactive Python.

   In [161]: re.findall('\d+','this is test a3 attempt 79')
   Out[161]: ['3', '79']

What I really want in just the 79, as a3 is not a decimal number, but 
when I add the \b word boundaries I get:


   In [162]: re.findall('\b\d+\b','this is test a3 attempt 79')
   Out[162]: []

What am I missing?


The sneaky detail that the regexp should be in a raw string 
(always a good practice), not a cooked string:


  r'\b\d+\b'

The "\d" isn't a valid character-expansion, so python leaves it 
alone.  However, I believe the "\b" is a control character, so 
your actual string ends up something like:


  >>> print repr('\b\d+\b')
  '\x08\\d+\x08'
  >>> print repr(r'\b\d+\b')
  '\\b\\d+\\b'

the first of which doesn't match your target string, as you might 
imagine.


-tkc



--
http://mail.python.org/mailman/listinfo/python-list

Re: logging of strings with broken encoding

[Lie Ryan]

Thomas Guettler wrote:
> My quick fix is this:
> 
> class MyFormatter(logging.Formatter):
> def format(self, record):
> msg=logging.Formatter.format(self, record)
> if isinstance(msg, str):
> msg=msg.decode('utf8', 'replace')
> return msg
> 
> But I still think handling of non-ascii byte strings should be better.
> A broken logging message is better than none.
> 

The problem is, python 2.x assumed the default encoding of `ascii`
whenever you don't explicitly mention the encoding, and your code
apparently broke with that assumption. I haven't looked at your code,
but others have suggested that you've fed the logging module with
non-ascii byte strings. The logging module can only work with 1) unicode
string, 2) ascii-encoded byte string

If you want a quick fix, you may be able to get away with repr()-ing
your log texts. A proper fix, however, is to pass a unicode string to
the logging module instead.

>>> logging.warn('ы') # or logging.warn('\xd1\x8b')
Traceback (most recent call last):
  File "/usr/lib64/python2.6/logging/__init__.py", line 773, in emit
stream.write(fs % msg.encode("UTF-8"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd1 in position 13:
ordinal not in range(128)
>>> logging.warn(repr('ы'))
WARNING:root:'\xd1\x8b'
>>> logging.warn(u'ы')
WARNING:root:ы
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: invoking a method from two superclasses

[Lie Ryan]

Carl Banks wrote:
> On Jun 30, 6:23 pm, Carl Banks  wrote:
>> On Jun 30, 5:34 pm, Mitchell L Model  wrote:
>>
>>> Allow me to add to my previous question that certainly the superclass
>>> methods can be called explicitly without resorting to super(), e.g.:
>>> class C(A, B):
>>> def __init__(self):
>>> A.__init__(self)
>>> B.__init__(self)
>>> My question is really whether there is any way of getting around the
>>> explicit class names by using super()
>> Yes there is: just make sure that all subclasses also call super.
> 
> And by subclasses I mean base classes, of course.  ugh
> 
> 
> Carl Banks

super() is short for two things:
1) superharmful
2) superuseful
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: dealloc function in python extend c module

[Philip Semanchuk]

On Jul 2, 2009, at 2:11 AM, Shen, Yu-Teh wrote:

I create my extend type something like http://www.python.org/doc/current/extending/newtypes.html 
.

And my type has a member which is a pointer point to my allocate
memory ( no ref count).
ex:
---
typedef struct {
   PyObject_HEAD
   /* Type-specific fields go here. */
   mytype *p;
} noddy_NoddyObject;

And i write dealloc function like this:
---
static void
Noddy_dealloc(Noddy* self)
{
   delete p;
   self->ob_type->tp_free((PyObject*)self);
}

And I found it's strange that it didn't call dealloc when there is no
reference.

ex:
a = Noddy()
b = a
del a
del b
# i think there is no ref to the object, and it should call dealloc
but it didn't call!!!

could anyone tell me what happened?



Hi Shen,
I'm no expert on Python memory management, but since no once else has  
answered your question I'll tell you what I *think* is happening.


Python doesn't delete objects as soon as they're dereferenced. It  
merely marks them as safe for garbage collection (GC). If GC never  
happens (and it might not in a small test program), your dealloc  
function won't run.


Now some more knowledgeable person will probably correct me. =)

You can control the garbage collector manually with Python's gc  
module. You will probably never want to control the garbage collector  
manually in ordinary programs, but it can be useful for testing.



Hope this helps
Philip

--
http://mail.python.org/mailman/listinfo/python-list

question of style

[Simon Forman]

Hey I was hoping to get your opinions on a sort of minor stylistic
point.
These two snippets of code are functionally identical. Which would you
use and why?
The first one is easier [for me anyway] to read and understand, but
slightly less efficient, while the second is [marginally] harder to
follow but more efficient.

## First snippet

if self.higher is self.lower is None: return
if self.lower is None: return self.higher
if self.higher is None: return self.lower

## Second snippet

if self.higher is None:
if self.lower is None:
return
return self.lower
if self.lower is None:
return self.higher

What do you think?

(One minor point: in the first snippet, the "is None" in the first
line is superfluous in the context in which it will be used, the only
time "self.lower is self.higher" will be true is when they are both
None.)
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: regex question on .findall and \b

[Sjoerd Mullender]

On 2009-07-02 18:38, Ethan Furman wrote:

Greetings!

My closest to successfull attempt:

Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit
(Intel)]
Type "copyright", "credits" or "license" for more information.

IPython 0.9.1 -- An enhanced Interactive Python.

In [161]: re.findall('\d+','this is test a3 attempt 79')
Out[161]: ['3', '79']

What I really want in just the 79, as a3 is not a decimal number, but
when I add the \b word boundaries I get:

In [162]: re.findall('\b\d+\b','this is test a3 attempt 79')
Out[162]: []

What am I missing?

~Ethan~


Try this:
>>> re.findall(r'\b\d+\b','this is test a3 attempt 79')
['79']

The \b is a backspace, by using raw strings you get an actual backslash 
and b.


--
Sjoerd Mullender
--
http://mail.python.org/mailman/listinfo/python-list

Re: Direct interaction with subprocess - the curse of blocking I/O

[Pascal Chambon]

Thank you all for the comments


you might want something like Expect.


Yes "Expect" deals with such things, unfortunately it's posix only (due to the 
PTY module requirement...); whereas I'd like to find generic ways (i.e at least 
windows/linux/mac recipes)


The latter is inherently tricky (which is why C's popen() lets you connect
to stdin or stdout but not both). You have to use either multiple threads,
select/poll, or non-blocking I/O.

If the child's output is to the console, it should presumably be the
former, i.e. piping stdin but allowing the child to inherit stdout, in
which case, where's the problem? Or are you piping its stdout via Python
for the hell of it
It's well a too-way piping that I'd like to get ; but actually, even  a 
single-way piping is error-prone : if I try to send data to the child's 
stdin, and this child never wants to receive that data (for whatever 
reason), the parent thread will be forever stuck.
I can use multiple threads, but it doesn't fully solve the problem, 
because having even a single thread stuck might prevent the process from 
terminating, on some operating systems... I'd need a way to deblock I/O 
blocking threads (but even closing the pipe isn't sure to do it properly 
- the closing might be delayed by the I/O operations wating).



>I would guess the architectural differences are so great that an attempt
>to "do something simple" is going to involve architecture-specific code.
>and I personally wouldn't have it any other way.  Simulating a shell
>with hooks on its I/O should be so complicated that a "script kiddie"
>has trouble writing a Trojan.

I'm not sure I understand the security hole there - if a script kiddie 
manages to make you execute his program, he doesn't need complex I/O 
redirections to harm you, simply destroying files randomly on your hard 
drive should fit his needs shouldn't it. :?




> I met the issue : select() works only on windows ...
  


No it doesn't. It works only on sockets on Windows, on Unix/Linux it works 
with all file descriptors .



Indeed, I mixed up my words there... >_<


>If you are willing to have a wxPython dependency, wx.Execute handles 
non-blocking i/o with processes on all supported platforms more 
discussion of python's blocking i/o issues here:
>  
http://groups.google.com/group/comp.lang.python/browse_thread/thread/a037349e18f99630/60886b8beb55cfbc?q=#60886b8beb55cfbc


Very interesting link, that's exactly the problem... and it seems no 
obvious solution comes out of it, except, of course, going down to the 
OS' internals...
File support is really weak in python I feel (stdio is a little 
outdated...), it'd need a complete blocking/non-blocking, 
locking/non-locking stream API straight in the stdlib...
I'm already working on locking classes, I'll explore the opportunity for 
a higher level "file" object too whenever possible (a new filesystem api 
is in progress here at europython2009, it might be the occasion).



regards,
Pascal



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: PEP 376

[Lie Ryan]

Joachim Strömbergson wrote:
> Aloha!
> 
> Tarek Ziadé wrote:
>> The prefix is a good idea but since it's just a checksum to control
>> that the file hasn't changed
>> what's wrong with using a weak hash algorithm like md5 or now sha1 ?
> 
> Because it creates a dependency to an old algorithm that should be
> deprecated. Also using MD5, even for a thing like this might make people
> belive that it is an ok algorithm to use - "Hey, it is used by the
> default install in Python, so it must be ok, right?"
> 
> If we flip the argument around: Why would you want to use MD5 instead of
> SHA-256? For the specific use case the performance will not (should not)
> be an issue.
> 
> As I wrote a few mails ago, it is time to move forward from MD5 and
> designing something in 2009 that will be around for many years that uses
> MD5 is (IMHO) a bad design decision.
> 
>> If someone wants to modify a file of a distribution he can recreate
>> the checksum as well,
>> the only secured way to prevent that would be to use gpg keys but
>> isn't that overkill for what we need ?
> 
> Actually, adding this type of security would IMHO be a good idea.
> 

Now, are we actually talking about security or checksum?

It has been known for years that MD5 is weak, weak, weak. Not just in
the recent years. But it doesn't matter since MD5 was never designed for
security, MD5 was designed to protect against random bits corruption. If
you want security, look at least to GPG. For data protection against
intentional, malicious forging, definitely MD5 is the wrong choice. But
when you just want a simple checksum to ensure that a faulty router
somewhere in the internet backbone doesn't destroying your data, MD5 is
a fine algorithm.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: question of style

[Duncan Booth]

Simon Forman  wrote:

> Hey I was hoping to get your opinions on a sort of minor stylistic
> point.
> These two snippets of code are functionally identical. Which would you
> use and why?
> The first one is easier [for me anyway] to read and understand, but
> slightly less efficient, while the second is [marginally] harder to
> follow but more efficient.
> 
> ## First snippet
> 
> if self.higher is self.lower is None: return
> if self.lower is None: return self.higher
> if self.higher is None: return self.lower
> 
> ## Second snippet
> 
> if self.higher is None:
> if self.lower is None:
> return
> return self.lower
> if self.lower is None:
> return self.higher
> 
> What do you think?
> 
> (One minor point: in the first snippet, the "is None" in the first
> line is superfluous in the context in which it will be used, the only
> time "self.lower is self.higher" will be true is when they are both
> None.)
> 

I'd write the first one as:

if self.lower is None:
return self.higher
if self.higher is None:
return self.lower

because the entire first 'if' statement is redundant.

As a matter of style however I wouldn't use the shorthand to run two 'is' 
comparisons together, I'd write that out in full if it was actually needed 
here.

Likewise in the second one:

if self.lower is None:
return
return self.lower

is obviously the same as:

return self.lower

so apart from reversing the order of the comparisons once you've dropped 
the redundant test it is the same as the first one.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: MySQLdb and ordering of column names in list returned by keys() w/ a DictCursor

[Petr Messner]

2009/7/2 Tim Chase :
>> Will this order at least be the same for that same query every time the
>> script is executed?
>
> I wouldn't count on it.  The order is only defined for the one iteration
> (result of the keys() call).  If the order matters, I'd suggest a
> double-dispatch with a non-dict (regular/default) query result, something
...

Dictionaries are usually designed to have the best performance when
accessing certain key, so it is not expected that dictionary items
will be in any particular order; this applies not only for Python. But
(for Python dict) you can be sure that results of items(), keys(),
values(), iteritems(), iterkeys(), and itervalues() will correspond if
called with no intervening modifications to the dictionary.

If such a functionality is needed, there is a collections.OrderedDict
that was introduced in Python 3.1; there is also an implementation for
Python 2.4 or later: http://code.activestate.com/recipes/576693/ (this
links is from "What’s New In Python 3.1")

PM
-- 
http://mail.python.org/mailman/listinfo/python-list

Config files with different types

[Zach Hobesh]

Hi all,

I've written a function that reads a specifically formatted text file
and spits out a dictionary.  Here's an example:

config.txt:

Destination = C:/Destination
Overwrite = True


Here's my function that takes 1 argument (text file)

the_file = open(textfile,'r')
linelist = the_file.read().split('\n')
the_file.close()
configs = {}
for line in linelist:
   try:
  key,value = line.split('=')
  key.strip()
  value.strip()
  key.lower()
  value.lower()
  configs[key] = value

   except ValueError:
  break

so I call this on my config file, and then I can refer back to any
config in my script like this:

shutil.move(your_file,configs['destination'])

which I like because it's very clear and readable.

So this works great for simple text config files.  Here's how I want
to improve it:

I want to be able to look at the value and determine what type it
SHOULD be.  Right now, configs['overwrite'] = 'true' (a string) when
it might be more useful as a boolean.  Is there a quick way to do
this?  I'd also like to able to read '1' as an in, '1.0' as a float,
etc...

I remember once I saw a script that took a string and tried int(),
float() wrapped in a try except, but I was wondering if there was a
more direct way.

Thanks in advance,

Zach
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Suppressing Implicit Chained Exceptions (Python 3.0)

[David Bolen]

"andrew cooke"  writes:

> However, when printed via format_exc(), this new exception still has the
> old exception attached via the mechanism described at
> http://www.python.org/dev/peps/pep-3134/ (this is Python 3.0).

If you're in control of the format_exc() call, I think the new chain
keyword parameter can disable this and restore the old behavior.

If you're not in control of the traceback display, I'm not sure there's
an easy way to prevent it, given that displaying chained exceptions is
the default mode.

-- David
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: question of style

[Simon Forman]

On Jul 2, 1:44 pm, Duncan Booth  wrote:
> Simon Forman  wrote:
> > Hey I was hoping to get your opinions on a sort of minor stylistic
> > point.
> > These two snippets of code are functionally identical. Which would you
> > use and why?
> > The first one is easier [for me anyway] to read and understand, but
> > slightly less efficient, while the second is [marginally] harder to
> > follow but more efficient.
>
> > ## First snippet
>
> > if self.higher is self.lower is None: return
> > if self.lower is None: return self.higher
> > if self.higher is None: return self.lower
>
> > ## Second snippet
>
> > if self.higher is None:
> >     if self.lower is None:
> >         return
> >     return self.lower
> > if self.lower is None:
> >     return self.higher
>
> > What do you think?
>
> > (One minor point: in the first snippet, the "is None" in the first
> > line is superfluous in the context in which it will be used, the only
> > time "self.lower is self.higher" will be true is when they are both
> > None.)
>
> I'd write the first one as:
>
>         if self.lower is None:
>                 return self.higher
>         if self.higher is None:
>             return self.lower
>
> because the entire first 'if' statement is redundant.
>
> As a matter of style however I wouldn't use the shorthand to run two 'is'
> comparisons together, I'd write that out in full if it was actually needed
> here.
>
> Likewise in the second one:
>
>     if self.lower is None:
>         return
>     return self.lower
>
> is obviously the same as:
>
>         return self.lower
>
> so apart from reversing the order of the comparisons once you've dropped
> the redundant test it is the same as the first one.

Wow. The equivalence in the second bit is obvious enough-- in
hindsight :] but I totally missed the redundancy in the first bit. I
must be getting old.

Thank you very much.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: PEP 376

[Charles Yeomans]

On Jul 2, 2009, at 1:37 PM, Lie Ryan wrote:


Joachim Strömbergson wrote:

Aloha!

Tarek Ziadé wrote:

The prefix is a good idea but since it's just a checksum to control
that the file hasn't changed
what's wrong with using a weak hash algorithm like md5 or now sha1 ?


Because it creates a dependency to an old algorithm that should be
deprecated. Also using MD5, even for a thing like this might make  
people

belive that it is an ok algorithm to use - "Hey, it is used by the
default install in Python, so it must be ok, right?"

If we flip the argument around: Why would you want to use MD5  
instead of
SHA-256? For the specific use case the performance will not (should  
not)

be an issue.

As I wrote a few mails ago, it is time to move forward from MD5 and
designing something in 2009 that will be around for many years that  
uses

MD5 is (IMHO) a bad design decision.


If someone wants to modify a file of a distribution he can recreate
the checksum as well,
the only secured way to prevent that would be to use gpg keys but
isn't that overkill for what we need ?


Actually, adding this type of security would IMHO be a good idea.



Now, are we actually talking about security or checksum?

It has been known for years that MD5 is weak, weak, weak. Not just in
the recent years. But it doesn't matter since MD5 was never designed  
for
security, MD5 was designed to protect against random bits  
corruption. If

you want security, look at least to GPG. For data protection against
intentional, malicious forging, definitely MD5 is the wrong choice.  
But

when you just want a simple checksum to ensure that a faulty router
somewhere in the internet backbone doesn't destroying your data, MD5  
is

a fine algorithm.
--


On the contrary, MD5 was intended to be a cryptographic hash function,  
not a checksum.


Charles Yeomans



--
http://mail.python.org/mailman/listinfo/python-list

Re: Config files with different types

[MRAB]

Zach Hobesh wrote:

Hi all,

I've written a function that reads a specifically formatted text file
and spits out a dictionary.  Here's an example:

config.txt:

Destination = C:/Destination
Overwrite = True


Here's my function that takes 1 argument (text file)

the_file = open(textfile,'r')
linelist = the_file.read().split('\n')


You could use .splitlines() or iterate over the file itself.


the_file.close()
configs = {}
for line in linelist:
   try:
  key,value = line.split('=')
  key.strip()
  value.strip()
  key.lower()
  value.lower()


Strings are immutable. These methods don't modify the strings, but
_return_ the result.


  configs[key] = value

   except ValueError:
  break


'break' will leave the loop. Is this intentional?


so I call this on my config file, and then I can refer back to any
config in my script like this:

shutil.move(your_file,configs['destination'])

which I like because it's very clear and readable.

So this works great for simple text config files.  Here's how I want
to improve it:

I want to be able to look at the value and determine what type it
SHOULD be.  Right now, configs['overwrite'] = 'true' (a string) when
it might be more useful as a boolean.  Is there a quick way to do
this?  I'd also like to able to read '1' as an in, '1.0' as a float,
etc...

I remember once I saw a script that took a string and tried int(),
float() wrapped in a try except, but I was wondering if there was a
more direct way.


The way you saw was the safe, and recommended, way.

When checking for Boolean you might want to ignore the case; something
like:

bool_dict = {"false": False, "true": True}
...
try:
value = bool_dict[value.strip().lower()]
except ValueError:
# Not a Boolean.
...
--
http://mail.python.org/mailman/listinfo/python-list

Re: question of style

[Kee Nethery]

the fact that you felt compelled to explain the "one minor point" in  
the first snippet tells me that the second snippet does not need that  
explanation and will be easier for someone (like you for example) to  
maintain in the future.

Second snippet would be my choice.
Kee Nethery

On Jul 2, 2009, at 10:23 AM, Simon Forman wrote:


Hey I was hoping to get your opinions on a sort of minor stylistic
point.
These two snippets of code are functionally identical. Which would you
use and why?
The first one is easier [for me anyway] to read and understand, but
slightly less efficient, while the second is [marginally] harder to
follow but more efficient.

## First snippet

if self.higher is self.lower is None: return
if self.lower is None: return self.higher
if self.higher is None: return self.lower

## Second snippet

if self.higher is None:
   if self.lower is None:
   return
   return self.lower
if self.lower is None:
   return self.higher

What do you think?

(One minor point: in the first snippet, the "is None" in the first
line is superfluous in the context in which it will be used, the only
time "self.lower is self.higher" will be true is when they are both
None.)
--
http://mail.python.org/mailman/listinfo/python-list





-
I check email roughly 2 to 3 times per business day.
Kagi main office: +1 (510) 550-1336


--
http://mail.python.org/mailman/listinfo/python-list

Re: Searching equivalent to C++ RAII or deterministic destructors

[Carl Banks]

On Jul 2, 3:12 am, Ulrich Eckhardt  wrote:
> Bearophile wrote:
> > Ulrich Eckhardt:
> >> a way to automatically release the resource, something
> >> which I would do in the destructor in C++.
>
> > Is this helpful?
> >http://effbot.org/pyref/with.htm
>
> Yes, it aims in the same direction. However, I'm not sure this applies to my
> case. The point is that the resource handle is not just used locally in a
> restricted scope but it is allocated and stored. The 'with' is something
> that makes sense in the context of mutex locking, where you have a
> well-defined critical section. What I need is something similar to open(),
> which returs a file. When the last reference to that object goes out of
> scope, the underlying file object is closed.

On CPython you can do it with a __del__ attribute.

Warning: objects with a __del__ attribute prevent reference cycle
detection, which can potentially lead to memory (and resource) leaks.
So you must be careful to avoid creating reference loops with that
object.

Note that file objects have a close method; you can explicitly close
it at any time.  Your object should follow that example, and define a
close (or release, or whatever) method.  I'd recommend making an
effort to call it and to rely on __del__ as little as possible.


Carl Banks
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: regex question on .findall and \b

[Nobody]

On Thu, 02 Jul 2009 09:38:56 -0700, Ethan Furman wrote:

> Greetings!
> 
> My closest to successfull attempt:
> 
> Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]
> Type "copyright", "credits" or "license" for more information.
> 
> IPython 0.9.1 -- An enhanced Interactive Python.
> 
>In [161]: re.findall('\d+','this is test a3 attempt 79')
>Out[161]: ['3', '79']
> 
> What I really want in just the 79, as a3 is not a decimal number, but 
> when I add the \b word boundaries I get:
> 
>In [162]: re.findall('\b\d+\b','this is test a3 attempt 79')
>Out[162]: []
> 
> What am I missing?

You need to use a raw string (r'...') to prevent \b from being interpreted
as a backspace:

re.findall(r'\b\d+\b','this is test a3 attempt 79')

\d isn't a recognised escape sequence, so it doesn't get interpreted:

> print '\b'
   ^H
> print '\d'
\d
> print r'\b'
\b

Try to get into the habit of using raw strings for regexps.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: question of style

[Terry Reedy]

Simon Forman wrote:

Hey I was hoping to get your opinions on a sort of minor stylistic
point.
These two snippets of code are functionally identical. Which would you
use and why?
The first one is easier [for me anyway] to read and understand, but
slightly less efficient, while the second is [marginally] harder to
follow but more efficient.

## First snippet

if self.higher is self.lower is None: return
if self.lower is None: return self.higher
if self.higher is None: return self.lower

## Second snippet

if self.higher is None:
if self.lower is None:
return
return self.lower
if self.lower is None:
return self.higher


What happen is neither is None?

--
http://mail.python.org/mailman/listinfo/python-list

Re: multiprocessing: pool with blocking queue

[J Kenneth King]

masher  writes:

> On Jul 2, 12:06 pm, J Kenneth King  wrote:
>> masher  writes:
>> > My questions, then, is: Is there a more elegant/pythonic way of doing
>> > what I am trying to do with the current Pool class?
>>
>> Forgive me, I may not fully understand what you are trying to do here
>> (I've never really used multiprocessing all that much)...
>>
>> But couldn't you just assign your own Queue object to the Pool instance?
>
> That's basically my question. It does not appear as though there is
> any straightforward way of doing this because of the design of Pool's
> __init__ method, which passes _taskqueue to several functions. Hence,
> even if I were to reassign _taskqueue after __init__, that wouldn't
> change anything.

I think I understand.

There are ways to modify the class before instantiating it, but even the
most clever or elegant solution will still smell funny.  I suppose this
might be worth submitting as a feature suggestion to the multiprocessing
project.

Best of luck.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Suppressing Implicit Chained Exceptions (Python 3.0)

[andrew cooke]

David Bolen wrote:
> "andrew cooke"  writes:
>
>> However, when printed via format_exc(), this new exception still has
the old exception attached via the mechanism described at
>> http://www.python.org/dev/peps/pep-3134/ (this is Python 3.0).
>
> If you're in control of the format_exc() call, I think the new chain
keyword parameter can disable this and restore the old behavior.
>
> If you're not in control of the traceback display, I'm not sure there's
an easy way to prevent it, given that displaying chained exceptions is
the default mode.

Yeah, that's the impression I got too.

Just in case it wasn't clear, I'd like the chaining not to exist at all in
this case), so that it doesn't appear in format_exc outside my control.

Thanks,
Andrew




-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Open Source RSS Reader in Python?

[member thudfoo]

On Wed, Jul 1, 2009 at 4:18 PM, Alex wrote:
> I am looking for an open source RSS reader (desktop, not online)
> written in Python but in vain. I am not looking for a package but a
> fully functional software.
>
> Google: python "open source" (rss OR feeds) reader
>
> Any clue ?
> --
> http://mail.python.org/mailman/listinfo/python-list
>

rawdog is an RSS Aggregator Without Delusions Of Grandeur. It is a
"river of news"-style aggregator: it uses feedparser to download feeds
in RSS, Atom and a variety of other formats, and (by default) produces
static HTML pages containing the newest articles in date order. For
example, rawdog's default configuration produces output that looks
like this:

http://offog.org/code/rawdog.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: question of style

[Paul Rubin]

Simon Forman  writes:
> These two snippets of code are functionally identical. Which would you
> use and why?

Both are terrible.  I can't tell what you're really trying to do.  As
Terry Reedy points out, the case where self.higher and self.lower are
both not None is not handled.  If you want to explicitly return None,
use "return None" rather than "return".  Generally, having a special
value like None to denote a missing datum was considered standard
practice a few decades ago, but these days in some circles it's
treated as something of a code smell.  You may want to look for
some other representation instead.

Most importantly, whatever code you write, put in an explanatory
comment saying what the real goal is.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: question of style

[Tim Harig]

On 2009-07-02, Duncan Booth  wrote:
> so apart from reversing the order of the comparisons once you've dropped 
> the redundant test it is the same as the first one.

I try to evaluate what you have given regardless of what Booth pointed out.
So, I will only evaluate the first line as it contains most of what I want
to illistrate.

> Simon Forman  wrote:
>> if self.higher is self.lower is None: return

I *really* don't like the fact that you don't specify a return value here
when you have specified a return value for the other conditions.  Obviously
it is possible that something will be looking at the return value for this
function (ie, it would not be a void function() in C).  I had to lookup to
make sure what Python would return in such a case.  If you want the
condition to return something then you should specify what you are
returning.  Specifing 'return None' is much clearer then relying on default
behavior.

I always prefer to see a:

condition:
operation

structure to my code.  It is the way it is almost always see in Python and
in other languages.  Putting it on one line doesn't save you anything and
it makes it easier to miss something.

I have no particular problem with running the is statements together as an
extended if statement together when it makes it clearer to do so.  The fact
that you want all of them to be the same object is clear.  I do have some
wroblem in this instance because you are referencing "is" against an
instance of a builtin object.

The fact that you used is instead of == for this particular instance
throws me a little because of the use of None.  I wouldn't think second
about it if it was a user defined object.  I have to wonder whether all
objects that are assigned the value of None are actually re-referenced
to the origional None object?  Could deep copy style operations create
two equivilant but different copys of none?  The same problems may be
encounted from any builtin objects such as number literals.

Had None not been specified I would be fine with stringing the comparison
together:

# if all three objects are the same instance, return None
if object1 is object2 is object3:
return None:

Because of this pecularity, I think that it would be clearer to break the
ifs for simularity and equality:

if object1 is object2:
if object1 == None:
return None
else:
return object1
else:
if object1 == None:
return Object2
elif object2 == None::
return Object1
else:
# what do we do if neither object == None?
raise houstonWeHaveAProblemException
-- 
http://mail.python.org/mailman/listinfo/python-list

Deadline for Toronto PyCamp Registraiton Appoaching

[Chris Calloway]

Tomorrow (July 3) by midnight will be the last opportunity for Toronto 
PyCamp registration before the late registration period ending July 10.


PyCamp is the original Python BootCamp developed by a user group for
user groups. This year PyCamp is July 13-17 at the University of
Toronto, sponsored by the Department of Physics and Scryent. For
beginners, PyCamp makes you productive so you can get your work done
quickly. PyCamp emphasizes the features which make Python a simpler and
more efficient language. Following along by example speeds your learning
process in a modern high-tech classroom. Become a self-sufficient Python
developer in just five days at PyCamp!

http://pycamp.org

--
Sincerely,

Chris Calloway
http://www.secoora.org
office: 332 Chapman Hall   phone: (919) 599-3530
mail: Campus Box #3300, UNC-CH, Chapel Hill, NC 27599



--
http://mail.python.org/mailman/listinfo/python-list

Intro to Python class, 7/21-23, Ft Worth TX

[Rich Drehoff]

We are looking for someone that can help with the subject class, Intro to
Python class, 7/21-23, Ft Worth TX.  Please let me know if you can help.
Would need your resume and best possible daily rate.

 

Best regards,

 

Rich Drehoff

TechnoTraining, Inc.

328 Office Square Lane, Ste. 202, Virginia Beach, VA 23462-3648

(757) 425-0728 x15 / (757) 425-7323 Fax

Email: rdreh...@technotraining.net

Website: www.TechnoTraining.net  

 

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: question of style

[Scott David Daniels]

Duncan Booth wrote:

Simon Forman  wrote:

...
if self.higher is self.lower is None: return
...


As a matter of style however I wouldn't use the shorthand to run two 'is' 
comparisons together, I'd write that out in full if it was actually needed 
here.


Speaking only to the style issue, when I've wanted to do something like
that, I find:

  if self.higher is None is self.lower:
  ...

more readable, by making clear they are both being compared to a
constant, rather than compared to each other.

More often, I've used code like:

  if  is not None is not :
  ...

since I am usually working on non-defaulting cases in the body.
I find the form above simpler to read than:

  if  is not None and  is not None:
  ...

I do draw the line at two, though, and with three or more I'll
paren-up a list of parallel comparisons:

  if ( is not None
and  is not None
and  is not None):
  ...

--Scott David Daniels
scott.dani...@acm.org
--
http://mail.python.org/mailman/listinfo/python-list

Re: question of style

[Paul Rubin]

Simon Forman  writes:
> ## Second snippet
> 
> if self.higher is None:
> if self.lower is None:
> return
> return self.lower
> if self.lower is None:
> return self.higher
> 
> What do you think?

I'm not sure, but my guess is that what you are really trying to write
is something like this:

   # self.higher and self.lower are each either "available" (i.e.  are
   # not None), or unavailable (are None).  Return the highest of the
   # available values.  If no value is available, return None.

   if self.higher is not None: 
   return self.higher
   elif self.lower is not None: 
   return self.lower
   else: 
   return None
-- 
http://mail.python.org/mailman/listinfo/python-list

how to spawn a process under different user

[sanket]

Hello All,

I am trying to use python's subprocess module to launch a process.
but in order to do that I have to change the user.

I am not getting any clue how to do that?
so can anyone please tell me How can I spawn a process under different
user than currently I am logging in as.

Thank you,
sanket
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: question of style

[Erik Max Francis]

Simon Forman wrote:

Hey I was hoping to get your opinions on a sort of minor stylistic
point.
These two snippets of code are functionally identical. Which would you
use and why?
The first one is easier [for me anyway] to read and understand, but
slightly less efficient, while the second is [marginally] harder to
follow but more efficient.

## First snippet

if self.higher is self.lower is None: return
if self.lower is None: return self.higher
if self.higher is None: return self.lower

## Second snippet

if self.higher is None:
if self.lower is None:
return
return self.lower
if self.lower is None:
return self.higher

What do you think?


Explicit is always better, `return None` when that is your intent. 
`return` shouts to the reader, "I want to get out of this function now, 
and no one cares about the return result."


Personally, I only use the one-liner if/else clauses if it's an 
extremely trivial condition, and even then, usually not there.  (The 
only place I use it consistently is `if __name__ == '__main__': 
main()`.)  If it's part of something more important that needs 
inspection -- as this obviously does given the questions you've had 
about it, doesn't skip space.  Newlines are free.


Even when expanding about the first test, there's no reason to do 
chained `if`s.  One `if` with an `and` works just fine.


Paul Rubin's looked to be the best overall replacement, as the logic 
here looks strangely turned inside out.  You're obviously looking for 
which one _isn't_ `None`, so write the tests that way.  It's much easier 
for everyone else (including your potential future self) to follow.


--
Erik Max Francis && m...@alcyone.com && http://www.alcyone.com/max/
 San Jose, CA, USA && 37 18 N 121 57 W && AIM/Y!M/Skype erikmaxfrancis
  They are ill discoverers that think there is no land when they can
   see nothing but sea. -- Francis Bacon
--
http://mail.python.org/mailman/listinfo/python-list

Re: question of style

[Tim Harig]

On 2009-07-02, Scott David Daniels  wrote:
> Duncan Booth wrote:
>> Simon Forman  wrote:
>> As a matter of style however I wouldn't use the shorthand to run two 'is' 
>> comparisons together, I'd write that out in full if it was actually needed 
>> here.

That part I don't really have a problem with.  The chained is statements
can actually be much clearer burying the tests into nested ifs.  Nested ifs
get the same result but they also give an immediate impression that the
control structure might be looking for more then a single test result.  I
have to walk through the ifs to see all of the potential results one is
looking for before I know what the entire control struture is doing.
Multiple is operators makes it clear that I just want everything to be the
same.  The same would not be true if the operators where not all equilance
operators ( is and == are clear as long as they are not mixed; but <, >=,
!= would be more confusing).

> Speaking only to the style issue, when I've wanted to do something like
> that, I find:
>if self.higher is None is self.lower:
> more readable, by making clear they are both being compared to a
> constant, rather than compared to each other.

By comparing them to *any* constant with 'is' you end up with the same
potential problems as to whether something with a value of None is actually
None?  Is something with the value of 3 actually the same object as 3?
What about if I make a deep copy?  Can I have several objects with the
value of three, all the same 3 object, or maybe even a combination of 1
separate 3 object with two other objects of the same 3.

Is
if self.higher == None == self.lower:
any more clear then:
if self.higher == self.lower == None:
I don't personally have a problem with either.

> More often, I've used code like:
>if  is not None is not :
> since I am usually working on non-defaulting cases in the body.
> I find the form above simpler to read than:
>if  is not None and  is not None:

This is a much more compicated expression.  With equality operators
everything is either the same or it isn't.   I can start from the first
variable and compare it to the second then to the third then ...  In the
case of the inequality, I have to start with the first and compare it to
all of the rest to make sure that none are the same.  Then I have to do the
same for each with the remaining.  This is much more difficult to do in my
head.  I requires much more thought, better left to a computer, when
evaluating the expression for myself.  Therefore, equalites are quick and
easy to keep straight whereas any inequalites are more error prone when
trying to evaluate how a section of code will react under different
circumstances.

> I do draw the line at two, though, and with three or more I'll
> paren-up a list of parallel comparisons:

As long as everything is an equality, then I don't mind comparing to the
end of an 70 column line.  Mixed types of equalities or inequalities
require too many operations mentally evaluate safely.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: System default sys.path

[Brock Pytlik]

David Lyon wrote:

On Wed, 01 Jul 2009 19:48:04 -0700, Brock Pytlik  wrote:
  
Hi, I'm trying to find a way to get the value sys.path would have on a 
particular system if python was started with an empty python path. I do 
want it to include the site specific additional paths. I know I can hack 
this information myself, 



Copy the code out from site.py...

  
Well, as far as I can tell, site does several things. It calls 
abs__file__ which, afaict, doesn't effect sys.path. Then it calls 
removeduppaths, which walks over sys.path. That means that sys.path has 
already been set by something at that point. So, how/where did sys.path 
get set then? Or is sys.path always empty there at startup and this code 
is just to handle other situations?


[snip]

Brock
--
http://mail.python.org/mailman/listinfo/python-list

Re: question of style

[Tim Harig]

On 2009-07-02, Paul Rubin  wrote:
[reformated with emphasis added]
>#self.higher and self.lower are each either "available" (i.e.  are not
>#None), or unavailable (are None).  [EMPHASIS] Return the highest of the
>#available values. [/EMPHASIS]  If no value is available, return None.

Your code doesn't do what your comment says (notice EMPHASIS tags):

>if self.higher is not None: 
>return self.higher
>elif self.lower is not None: 
>return self.lower
>else: 
>return None

If lower is 5 and higher is 3, then it returns 3 because 3 != None in the
first if.
-- 
http://mail.python.org/mailman/listinfo/python-list

PyTextMagicSMS text messaging library released

[Dawie Strauss]

TextMagic (http://www.textmagic.com) offers a convenient way to send
text messages programmatically. This package provides a simple Python
API on top of the TextMagic HTTPS API.

Project documentation and source code is hosted at
http://code.google.com/p/textmagic-sms-api-python/

The package can be dowloaded from http://pypi.python.org/pypi/PyTextMagicSMS/

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: how to spawn a process under different user

[Tim Golden]

sanket wrote:

Hello All,

I am trying to use python's subprocess module to launch a process.
but in order to do that I have to change the user.

I am not getting any clue how to do that?
so can anyone please tell me How can I spawn a process under different
user than currently I am logging in as.


What platform are you on? If you are on Windows,
there was a thread on this subject some time in
the last month. (Short answer: switching user before
launching subprocess won't work; switching within
the subprocess will...)

TJG
--
http://mail.python.org/mailman/listinfo/python-list

stringio+tarfile

[superpollo]

why the following does not work? can you help me correct (if possible)?

  1 import tarfile
  2 import StringIO
  3 sf1 = StringIO.StringIO("one\n")
  4 sf2 = StringIO.StringIO("two\n")
  5 tf = StringIO.StringIO()
  6 tar = tarfile.open(tf , "w")
  7 for name in [sf1 , sf2]:
  8 tar.add(name)
  9 print tf
 10 sf1.close()
 11 sf2.close()
 12 tf.close()
 13 tar.close()

tia
--
http://mail.python.org/mailman/listinfo/python-list

Re: Searching equivalent to C++ RAII or deterministic destructors

[Roel Schroeven]

Peter Otten schreef:
> Ulrich Eckhardt wrote:
> 
>> Bearophile wrote:
>>> Ulrich Eckhardt:
 a way to automatically release the resource, something
 which I would do in the destructor in C++.
>>> Is this helpful?
>>> http://effbot.org/pyref/with.htm
>> Yes, it aims in the same direction. However, I'm not sure this applies to
>> my case. The point is that the resource handle is not just used locally in
>> a restricted scope but it is allocated and stored. The 'with' is something
>> that makes sense in the context of mutex locking, where you have a
>> well-defined critical section. 
> 
> Isn't that exactly what RAII does?

RAII also works if the resource handle is stored, for example, in a data
member of an object. If that object is destroyed (because it goes out of
scope, or because it is deleted), the resource is automatically
destroyed too.

The way RAII works is actually the one thing from C++ that I miss in Python.

-- 
The saddest aspect of life right now is that science gathers knowledge
faster than society gathers wisdom.
  -- Isaac Asimov

Roel Schroeven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: question of style

[Paul Rubin]

Tim Harig  writes:
> If lower is 5 and higher is 3, then it returns 3 because 3 != None in the
> first if.

Sorry, the presumption was that lower <= higher, i.e. the comparison
had already been made and the invariant was enforced by the class
constructor.  The comment should have been more explicit, I guess.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: how to spawn a process under different user

[sanket]

On Jul 2, 1:58 pm, Tim Golden  wrote:
> sanket wrote:
> > Hello All,
>
> > I am trying to use python's subprocess module to launch a process.
> > but in order to do that I have to change the user.
>
> > I am not getting any clue how to do that?
> > so can anyone please tell me How can I spawn a process under different
> > user than currently I am logging in as.
>
> What platform are you on? If you are on Windows,
> there was a thread on this subject some time in
> the last month. (Short answer: switching user before
> launching subprocess won't work; switching within
> the subprocess will...)
>
> TJG

Hi TJG,
Thanks for the reply.

I am using python 2.4 on centos.

Thanks,
sanket
-- 
http://mail.python.org/mailman/listinfo/python-list

String to List Question

[Hanna Michelsen]

Hi,

I am brand new to python and I love it, but I've been having some trouble
with a file parser that I've been working on. It contains lines that start
with a name and then continue with names, nicknames and phone numbers of
people associated with that name. I need to create a list of the names of
people associated with each singular person (the first name in each line).
Each name/phone number is separated by a tab but if someone doesn't have a
nickname there are two tabs between their name and number.

I've been trying to figure out how to test for two tabs, skip over these
people and move onto the next name but I just can't figure out how that will
work in python.

Any help would be greatly appreciated!

Thanks,
Hanna
-- 
http://mail.python.org/mailman/listinfo/python-list

XML(JSON?)-over-HTTP: How to define API?

[Allen Fowler]

I have an (in-development) python system that needs to shuttle events / 
requests around over the network to other parts of itself.   It will also need 
to cooperate with a .net application running on yet a different machine.

So, naturally I figured some sort of HTTP event / RPC type of would be a good 
idea?

Are there any modules I should know about, or guidelines I could read, that 
could aid me in the design of the API?


Thank you,
:)


  

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: String to List Question

[Philip Semanchuk]

On Jul 2, 2009, at 6:05 PM, Hanna Michelsen wrote:


Hi,

I am brand new to python and I love it, but I've been having some  
trouble
with a file parser that I've been working on. It contains lines that  
start
with a name and then continue with names, nicknames and phone  
numbers of
people associated with that name. I need to create a list of the  
names of
people associated with each singular person (the first name in each  
line).
Each name/phone number is separated by a tab but if someone doesn't  
have a

nickname there are two tabs between their name and number.

I've been trying to figure out how to test for two tabs, skip over  
these
people and move onto the next name but I just can't figure out how  
that will

work in python.

Any help would be greatly appreciated!


Hi Hanna,
Are you familiar with a string's split() function? It sounds like just  
what you need.


http://docs.python.org/library/stdtypes.html#str.split

HTH
Philip

--
http://mail.python.org/mailman/listinfo/python-list

Re: XML(JSON?)-over-HTTP: How to define API?

[Allen Fowler]

> I have an (in-development) python system that needs to shuttle events / 
> requests 
> around over the network to other parts of itself.   It will also need to 
> cooperate with a .net application running on yet a different machine.
> 
> So, naturally I figured some sort of HTTP event / RPC type of would be a good 
> idea?
> 
> Are there any modules I should know about, or guidelines I could read, that 
> could aid me in the design of the API?
> 
> 


To clarify:

Each message would be <1KB of data total, and consist of some structured object 
containing strings, numbers, dates, etc.

For instance there would be an "add user" request that would contain one or 
more User objects each having a number of properties like:

- Full Name
- Username
- Password
- Email addresses (a variable length array)
- Street Address line1
- Street Address line1
- Street Address line1
- City
- State
- Zip
- Sign Up Date

 and so on.


Since I need to work with other platforms, pickle is out...  what are the 
alternatives?  XML? JSON?

How should I formally define each of the valid messages and objects?

Thank you,
:)



  

-- 
http://mail.python.org/mailman/listinfo/python-list