Strange array.array performance
Hello all,
I'm currently writing a Python <-> MATLAB interface with ctypes and
array.array class, using which I'll need to push large amounts of data
to MATLAB. Everything is working well, but there was one strange
performance-related issue that I ran into and wanted to ask about.
Here's some example code to illustrate my point (this is developed on
Windows, hence the use of clock):
---
from array import array
from time import clock
input = array('B', range(256) * 1)
# Case 1
start = clock()
data1 = array('B', input)
print format(clock() - start, '.10f')
# Case 2
start = clock()
data2 = array('B')
data2[:] = input
print format(clock() - start, '.10f')
# Case 3
start = clock()
data3 = array('B')
data3.extend(input)
print format(clock() - start, '.10f')
print input == data1 == data2 == data3
---
The output from this on my machine is as follows:
0.7080547730
0.0029827034
0.0028685943
True
That seems very wrong. In the end, all arrays have the same data, but
by specifying it in the constructor the creation process takes over
350x longer than the other two methods. Is this a bug, or is there
some valid reason for it?
In the latter case, it would be a good idea to mention this in the
documentation, since that can be a significant performance improvement
in some applications. Currently the documentation states "Otherwise,
the iterable initializer is passed to the extend() method," which
doesn't seem to be the case, based on the third example.
- Max
--
http://mail.python.org/mailman/listinfo/python-list
Re: Strange array.array performance
On Thu, Feb 19, 2009 at 2:35 PM, Robert Kern wrote:
> On 2009-02-19 12:52, Maxim Khitrov wrote:
>>
>> Hello all,
>>
>> I'm currently writing a Python<-> MATLAB interface with ctypes and
>> array.array class, using which I'll need to push large amounts of data
>> to MATLAB.
>
> Have you taken a look at mlabwrap?
>
> http://mlabwrap.sourceforge.net/
>
> At the very least, you will probably want to use numpy arrays instead of
> array.array.
>
> http://numpy.scipy.org/
I have, but numpy is not currently available for python 2.6, which is
what I need for some other features, and I'm trying to keep the
dependencies down in any case. Mlabwrap description doesn't mention if
it is thread-safe, and that's another one of my requirements.
The only feature that I'm missing with array.array is the ability to
quickly pre-allocate large chunks of memory. To do that right now I'm
using array('d', (0,) * size). It would be nice if array accepted an
int as the second argument indicating how much memory to allocate and
initialize to 0.
- Max
--
http://mail.python.org/mailman/listinfo/python-list
Re: Strange array.array performance
On Thu, Feb 19, 2009 at 2:23 PM, Gabriel Genellina
wrote:
> En Thu, 19 Feb 2009 16:52:54 -0200, Maxim Khitrov
> escribió:
>
>> input = array('B', range(256) * 1)
>>
>> # Case 1
>> start = clock()
>> data1 = array('B', input)
>> print format(clock() - start, '.10f')
>
>> That seems very wrong. In the end, all arrays have the same data, but
>> by specifying it in the constructor the creation process takes over
>> 350x longer than the other two methods. Is this a bug, or is there
>> some valid reason for it?
>
> It's a known issue: http://bugs.python.org/issue5109
I see, thanks.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Strange array.array performance
On Thu, Feb 19, 2009 at 7:01 PM, Scott David Daniels
wrote:
> Maxim Khitrov wrote:
>>
>> On Thu, Feb 19, 2009 at 2:35 PM, Robert Kern
>> wrote:
>> I have, but numpy is not currently available for python 2.6, which is
>> what I need for some other features, and I'm trying to keep the
>> dependencies down in any case
>> The only feature that I'm missing with array.array is the ability to
>> quickly pre-allocate large chunks of memory. To do that right now I'm
>> using array('d', (0,) * size). It would be nice if array accepted an
>> int as the second argument indicating how much memory to allocate and
>> initialize to 0.
>
> In the meantime, you could write a function (to ease the shift to numpy)
> and reduce your interface problem to a very small set of lines:
>def zeroes_d(n):
>'''Allocate a n-element vector of 'd' elements'''
>vector = array.array('d') # fromstring has no performance bug
>vector.fromstring(n * 8 * '\0')
>return vector
> Once numpy is up and running on 2.6, this should be easy to convert
> to a call to zeroes.
If I do decide to transition at any point, it will require much
greater modification. For example, to speed-up retrieval of data from
Matlab, which is returned to me as an mxArray structure, I allocate an
array.array for it and then use ctypes.memmove to copy data directly
into the array's buffer (address obtained through buffer_info()).
Same thing for sending data, rather than allocate a separate mxArray,
copy data, and then send, I create an empty mxArray and set its data
pointer to the array's buffer. I'm sure that there are equivalents in
numpy, but the point is that the transition, which currently would not
benefit my code in any significant way, will not be a quick change.
On the other hand, I have to thank you for the fromstring example. For
some reason, it never occurred to me that creating a string of nulls
would be much faster than a tuple of zeros. In fact, you can pass the
string to the constructor and it calls fromstring automatically. For
an array of 1 million elements, using a string to initialize is 18x
faster. :)
- Max
--
http://mail.python.org/mailman/listinfo/python-list
Re: Strange array.array performance
On Thu, Feb 19, 2009 at 7:01 PM, Scott David Daniels
wrote:
> Maxim Khitrov wrote:
>>
>> On Thu, Feb 19, 2009 at 2:35 PM, Robert Kern
>> wrote:
>> I have, but numpy is not currently available for python 2.6, which is
>> what I need for some other features, and I'm trying to keep the
>> dependencies down in any case
>> The only feature that I'm missing with array.array is the ability to
>> quickly pre-allocate large chunks of memory. To do that right now I'm
>> using array('d', (0,) * size). It would be nice if array accepted an
>> int as the second argument indicating how much memory to allocate and
>> initialize to 0.
>
> In the meantime, you could write a function (to ease the shift to numpy)
> and reduce your interface problem to a very small set of lines:
>def zeroes_d(n):
>'''Allocate a n-element vector of 'd' elements'''
>vector = array.array('d') # fromstring has no performance bug
>vector.fromstring(n * 8 * '\0')
>return vector
> Once numpy is up and running on 2.6, this should be easy to convert
> to a call to zeroes.
Here's the function that I'll be using from now on. It gives me
exactly the behavior I need, with an int initializer being treated as
array size. Still not as efficient as it could be if supported
natively by array (one malloc instead of two + memmove + extra
function call), but very good performance nevertheless:
from array import array as _array
array_null = dict((tc, '\0' * _array(tc).itemsize) for tc in 'cbBuhHiIlLfd')
def array(typecode, init):
if isinstance(init, int):
return _array(typecode, array_null[typecode] * init)
return _array(typecode, init)
- Max
--
http://mail.python.org/mailman/listinfo/python-list
Re: Strange array.array performance
On Thu, Feb 19, 2009 at 9:34 PM, David Cournapeau wrote: > On Fri, Feb 20, 2009 at 4:53 AM, Maxim Khitrov wrote: >> On Thu, Feb 19, 2009 at 2:35 PM, Robert Kern wrote: >>> On 2009-02-19 12:52, Maxim Khitrov wrote: >>>> >>>> Hello all, >>>> >>>> I'm currently writing a Python<-> MATLAB interface with ctypes and >>>> array.array class, using which I'll need to push large amounts of data >>>> to MATLAB. >>> >>> Have you taken a look at mlabwrap? >>> >>> http://mlabwrap.sourceforge.net/ >>> >>> At the very least, you will probably want to use numpy arrays instead of >>> array.array. >>> >>> http://numpy.scipy.org/ >> >> I have, but numpy is not currently available for python 2.6, which is >> what I need for some other features, and I'm trying to keep the >> dependencies down in any case. Mlabwrap description doesn't mention if >> it is thread-safe, and that's another one of my requirements. > > What do you mean by thread-safe ? Different threads calling the same > matlab engine handle ? Yes, I may have a case where one thread is still sending data, while another tries to close the connection, or two threads trying to close the connection at the same time. In both cases, I need some parts of the code to be atomic to prevent errors. - Max -- http://mail.python.org/mailman/listinfo/python-list
Re: Strange array.array performance
On Thu, Feb 19, 2009 at 9:15 PM, John Machin wrote:
> On Feb 20, 6:53 am, Maxim Khitrov wrote:
>> On Thu, Feb 19, 2009 at 2:35 PM, Robert Kern wrote:
>> > On 2009-02-19 12:52, Maxim Khitrov wrote:
>>
>> >> Hello all,
>>
>> >> I'm currently writing a Python<-> MATLAB interface with ctypes and
>> >> array.array class, using which I'll need to push large amounts of data
>> >> to MATLAB.
>>
>> > Have you taken a look at mlabwrap?
>>
>> > http://mlabwrap.sourceforge.net/
>>
>> > At the very least, you will probably want to use numpy arrays instead of
>> > array.array.
>>
>> > http://numpy.scipy.org/
>>
>> I have, but numpy is not currently available for python 2.6, which is
>> what I need for some other features, and I'm trying to keep the
>> dependencies down in any case. Mlabwrap description doesn't mention if
>> it is thread-safe, and that's another one of my requirements.
>>
>> The only feature that I'm missing with array.array is the ability to
>> quickly pre-allocate large chunks of memory. To do that right now I'm
>> using array('d', (0,) * size).
>
> It would go somewhat faster if you gave it a float instead of an int.
>
>> It would be nice if array accepted an
>> int as the second argument indicating how much memory to allocate and
>> initialize to 0.
>
> While you're waiting for that to happen, you'll have to use the
> fromstring trick, or another gimmick that is faster and is likely not
> to use an extra temp 8Mb for a 1M-element array, as I presume the
> fromstring does.
>
> [Python 2.6.1 on Windows XP SP3]
> [Processor: x86 Family 15 Model 36 Stepping 2 AuthenticAMD ~1994 Mhz]
>
> C:\junk>\python26\python -mtimeit -s"from array import array" "x=array
> ('d',(0,)*
> 100)"
> 10 loops, best of 3: 199 msec per loop
>
> C:\junk>\python26\python -mtimeit -s"from array import array" "x=array
> ('d',(0.,)*100)"
> 10 loops, best of 3: 158 msec per loop
>
> C:\junk>\python26\python -mtimeit -s"from array import array" "x=array
> ('d');x.fromstring('\0'*8*100)"
> 10 loops, best of 3: 36 msec per loop
>
> C:\junk>\python26\python -mtimeit -s"from array import array" "x=array
> ('d','\0'*8*100)"
> 10 loops, best of 3: 35.7 msec per loop
>
> C:\junk>\python26\python -mtimeit -s"from array import array" "array
> ('d',(0.,))*100"
> 10 loops, best of 3: 19.5 msec per loop
Interesting, though I'm not able to replicate that last outcome. The
string method is still the fastest on my machine. Furthermore, it
looks like the order in which you do the multiplication also matters -
(8 * size * '\0') is faster than ('\0' * 8 * size). Here is my test
and outcome:
---
from array import array
from timeit import repeat
print repeat(lambda: array('d', (0,) * 10), number = 100)
print repeat(lambda: array('d', (0.0,) * 10), number = 100)
print repeat(lambda: array('d', (0.0,)) * 10, number = 100)
print repeat(lambda: array('d', '\0' * 10 * 8), number = 100)
print repeat(lambda: array('d', '\0' * 8 * 10), number = 100)
print repeat(lambda: array('d', 8 * 10 * '\0'), number = 100)
---
[0.91048107424534941, 0.88766983642377162, 0.88312824645684618]
[0.72164595848486179, 0.72038338197219343, 0.72346024633711981]
[0.10763947529894136, 0.1047547164728595, 0.10461521722863232]
[0.05856873793382178, 0.058508825334111947, 0.058361838698573365]
[0.057632016342657799, 0.057521392119007864, 0.057227118035289237]
[0.056006643320014149, 0.056331811311153501, 0.05618743321510]
The array('d', (0.0,)) * 10 method is a good compromise between
performance and amount of memory used, so maybe I'll use that instead.
- Max
--
http://mail.python.org/mailman/listinfo/python-list
Re: Strange array.array performance
On Thu, Feb 19, 2009 at 10:06 PM, David Cournapeau wrote: > On Fri, Feb 20, 2009 at 11:43 AM, Maxim Khitrov wrote: >> >> Yes, I may have a case where one thread is still sending data, while >> another tries to close the connection, or two threads trying to close >> the connection at the same time. In both cases, I need some parts of >> the code to be atomic to prevent errors. > > That does not sound like the right approach, then. Matlab engine is > not thread safe in that sense: > > http://www.mathworks.fr/support/solutions/data/1-YR98I.html?product=ML&solution=1-YR98I > > cheers, > > David > "One option is to use MUTEXes around every call into the MATLAB Engine" - that's exactly what I'm doing. Some setup work, like creating mxArrays is done without a lock, but all eng* calls are mutually exclusive. I've been doing a lot of my own testing, and so far have seen no problems with this approach. - Max -- http://mail.python.org/mailman/listinfo/python-list
Re: Strange array.array performance
On Fri, Feb 20, 2009 at 2:42 AM, Scott David Daniels wrote: > Maxim Khitrov wrote: >> >> ... Here's the function that I'll be using from now on. It gives me >> exactly the behavior I need, with an int initializer being treated as >> array size. Still not as efficient as it could be if supported >> natively by array (one malloc instead of two + memmove + extra >> function call), but very good performance nevertheless: >> >> from array import array as _array >> array_null = dict((tc, '\0' * _array(tc).itemsize) for tc in >> 'cbBuhHiIlLfd') > > How about: > array_null = dict((tc, _array(tc, (0,)).tostring() for tc in > 'cbBuhHiIlLfd') > ... > (some ancient floating points did not use all-0 bytes for 0.0). Didn't know that, thanks. I actually got rid of the dict, since benchmarks showed access time to itemsize in the function itself is not any slower than dict access. After going through all the different speed tests yesterday the function now looks like this: from array import array as _array def array(typecode, init): if isinstance(init, int): a = _array(typecode, (0,)) if a.itemsize * init > 1048576: return a * init else: a.fromstring((init - 1) * a.tostring()) return a return _array(typecode, init) This uses the fast fromstring operation when creating an array that is less than 1MB in size. Over that, array multiplication is used, which is slower, but doesn't require the extra memory. - Max -- http://mail.python.org/mailman/listinfo/python-list
Using clock() in threading on Windows
Greetings, The threading module uses time.time in _Condition and _Thread classes to implement timeouts. On Windows, time() typically has a resolution of 15.625ms. In addition, if the system clock is changed (though ntp, for example) it would reflect that change, causing the timeout to last longer or shorter depending on which way the update went. Would it not be better to use time.clock() instead? The resolution is much better, and the value is not linked to system clock. Right now, I replace the threading._time reference with clock in some of my programs and everything works perfectly. Condition and Event timeouts are precise down to the millisecond (resolution of the sleep function), and I see no side-effects. Is it possible to make that change part of the module itself (keeping time() for linux systems), or can someone think of a reason why using clock is a bad idea? I know that it's using QueryPerformanceCounter for implementation, which has some known issues, but I still think that the advantages outweigh potential faults. - Max -- http://mail.python.org/mailman/listinfo/python-list
What functions, other than sleep(), can be interrupted by Ctrl-C?
Greetings, I'm looking for a function in the standard library or pywin32 package that will block until a certain condition is met or it is interrupted by Ctrl-C. For example, time.sleep() would have been perfect for my needs if thread.interrupt_main() could interrupt the call from another thread in the same way that Ctrl-C does. Unfortunately, that is not the case. Another thing I tried was creating a pipe with os.pipe() and issuing a read call on it. The event to exit was a single byte written to the other end of the pipe, but Ctrl-C could not interrupt the read call. The threading.Event class does not work for me, because it uses short sleep intervals for timed waits. I need the reaction to be as close to instant as possible, something like this will not do: while not : sleep(0.01) I actually replaced threading.Event with my own version that uses native Windows events, and waits on those also cannot be interrupted by Ctrl-C. I'm trying to achieve the same effect as that while loop and I don't care what the condition to exit is, but the loop needs to exit as soon as the condition is met without waiting for up to X additional milliseconds. Any ideas? - Max -- http://mail.python.org/mailman/listinfo/python-list
Re: What functions, other than sleep(), can be interrupted by Ctrl-C?
On Thu, Feb 26, 2009 at 3:47 PM, Gabriel Genellina wrote: >> I'm looking for a function in the standard library or pywin32 package >> that will block until a certain condition is met or it is interrupted >> by Ctrl-C. For example, time.sleep() would have been perfect for my >> needs if thread.interrupt_main() could interrupt the call from another >> thread in the same way that Ctrl-C does. Unfortunately, that is not >> the case. > > You may try MsgWaitForMultipleObjects - send a message to the main thread > from the other thread. > An alertable wait (like SleepEx) plus QueueUserAPC should work, I presume, > but I've never actually tried in Python. I tried using MsgWaitForMultipleObjects, but even with a wake mask of 0x Ctrl-C still did not interrupt the call. Maybe it has to do with the way pywin32 implements it. QueueUserAPC is not available in pywin32, but I suppose I can try getting to it via ctypes. - Max -- http://mail.python.org/mailman/listinfo/python-list
Re: py2exe automatic upgrades of a program while it is running, is that possible?
On Mon, Mar 2, 2009 at 9:18 PM, William Heath wrote: > Hi All, > I am using py2exe to create a windows executable. I am curious if anyone > knows a way to automatically upgrade a py2exe windows executable while it is > running. Is that possible? If so how? If it isn't possible, what is the > next best thing? Also, if it is not available using py2exe is it available > in other languages/solutions your aware of? > -Tim I don't think there is a way to do this directly. The best thing that I can think of is to have your program to use the subprocess or multiprocessing modules to launch a second instance of itself when an update is required. The parent would pass the child any necessary data so the child could continue where the parent left off, have child acknowledge this transaction, and then the parent can exit. Depending on what your program is actually doing this may not be a reasonable solution. Look into os.pipe, multiprocessing.Queue, and subprocess.Popen.communicate. If you have pywin32 extension installed, win32event module contains a number of useful functions for things like creating inter-process events. - Max -- http://mail.python.org/mailman/listinfo/python-list
Style question - defining immutable class data members
Very simple question on the preferred coding style. I frequently write classes that have some data members initialized to immutable values. For example: class Test(object): def __init__(self): self.some_value = 0 self.another_value = None Similar effect can be achieved by defining some_value and another_value for the entire class, like so: class Test(object): some_value = 0 another_value = None The advantage of doing this is that the assignments are evaluated once and thus the creation of that class is a bit faster. Access is still performed through self.some_value and self.another_value. Is there a reason to prefer the first style over the second? - Max -- http://mail.python.org/mailman/listinfo/python-list
Re: Style question - defining immutable class data members
On Sat, Mar 14, 2009 at 12:50 PM, MRAB wrote: > Maxim Khitrov wrote: >> >> Very simple question on the preferred coding style. I frequently write >> classes that have some data members initialized to immutable values. >> For example: >> >> class Test(object): >> def __init__(self): >> self.some_value = 0 >> self.another_value = None >> >> Similar effect can be achieved by defining some_value and >> another_value for the entire class, like so: >> >> class Test(object): >> some_value = 0 >> another_value = None >> >> The advantage of doing this is that the assignments are evaluated once >> and thus the creation of that class is a bit faster. Access is still >> performed through self.some_value and self.another_value. Is there a >> reason to prefer the first style over the second? >> > In the first case each instance has its own attributes, whereas in the > second case the attributes belong to the class and are thus shared by > all the instances. Which you use depends on whether you want them shared > or not. When the types are immutable, there is no difference. The second case is faster and consumes less memory (initially), but nothing else changes. I'm simply asking if it is considered poor style to define data members for the class when they are only used by instances. - Max -- http://mail.python.org/mailman/listinfo/python-list
Re: Style question - defining immutable class data members
On Sat, Mar 14, 2009 at 2:07 PM, Gary Herron wrote: > Maxim Khitrov wrote: >> >> Very simple question on the preferred coding style. I frequently write >> classes that have some data members initialized to immutable values. >> For example: >> >> class Test(object): >> def __init__(self): >> self.some_value = 0 >> self.another_value = None >> >> Similar effect can be achieved by defining some_value and >> another_value for the entire class, like so: >> >> class Test(object): >> some_value = 0 >> another_value = None >> >> The advantage of doing this is that the assignments are evaluated once >> and thus the creation of that class is a bit faster. Access is still >> performed through self.some_value and self.another_value. Is there a >> reason to prefer the first style over the second? >> >> - Max >> -- >> http://mail.python.org/mailman/listinfo/python-list >> > > Such things are often called class attributes, and the are fine. If you > look through Python's standard library, you will find many examples of > class attributes. > > However, you appear to have either a misuse or misconception of the word > "immutable' here. Whether the value you assign to a class attribute is > mutable or immutable is irrelevant. Also whether you plan on leaving the > value constant or not is also not relevant. > What does matter is this: If every instance wants access to a single value > (immutable or not), use a class attribute, otherwise use an instance > attribute. > > Gary Herron Perhaps a different example would help explain what I'm trying to do: class Case1(object): def __init__(self): self.count = 0 self.list = [] def inc(self): self.count += 1 self.list.append(self.count) def val(self): return (self.count, self.list) class Case2(object): count = 0 list = [] def inc(self): self.count += 1 self.list.append(self.count) def val(self): return (self.count, self.list) for i in xrange(10): c1 = Case1() c2 = Case2() for j in xrange(i): c1.inc() c2.inc() v1, l1 = c1.val() v2, l2 = c2.val() print v1 == v2, l1 == l2 The only difference between Case1 and Case2 classes is where the count and list attributes are defined. You will notice that for an immutable type (count), this doesn't matter. On the last line, v1 == v2 is always True. When the type is mutable (list), you must define it in __init__. This isn't about class attributes or shared instance attributes/constants. This is about a small optimization in defining per-instance variables. This optimization only applies to immutable types. - Max -- http://mail.python.org/mailman/listinfo/python-list
Re: Style question - defining immutable class data members
On Sat, Mar 14, 2009 at 4:31 PM, Gary Herron wrote: >> Perhaps a different example would help explain what I'm trying to do: >> >> class Case1(object): >> def __init__(self): >> self.count = 0 >> self.list = [] >> >> def inc(self): >> self.count += 1 >> self.list.append(self.count) >> >> def val(self): >> return (self.count, self.list) >> >> class Case2(object): >> count = 0 >> list = [] >> >> def inc(self): >> self.count += 1 >> self.list.append(self.count) >> >> def val(self): >> return (self.count, self.list) >> >> for i in xrange(10): >> c1 = Case1() >> c2 = Case2() >> >> for j in xrange(i): >> c1.inc() >> c2.inc() >> >> v1, l1 = c1.val() >> v2, l2 = c2.val() >> >> print v1 == v2, l1 == l2 >> >> The only difference between Case1 and Case2 classes is where the count >> and list attributes are defined. You will notice that for an immutable >> type (count), this doesn't matter. On the last line, v1 == v2 is >> always True. When the type is mutable (list), you must define it in >> __init__. This isn't about class attributes or shared instance >> attributes/constants. This is about a small optimization in defining >> per-instance variables. This optimization only applies to immutable >> types. >> >> - Max >> -- >> http://mail.python.org/mailman/listinfo/python-list >> > > But now you are not listening to what people are telling you. It has > *nothing* to do with the mutability/immutability of the integer and the list > your two classes create. > > The difference is this: > > For C1: You create 10 instances of C1. Each one creates its own count, > and a list variables, and manipulates them calls to inc and val. Then each > on is discarded as you go through the next pass on the outer loop. Correct, though the discarded part makes no difference. > For C2; You create 10 instances of C2, but these 10 instances each > manipulate values created once in the class itself. The values manipulated > by one instance of C2 in one pass through the loop are not affected when, on > the next pass through the loop, that instance is destroyed and another > instance is created. > So... Incorrect. Only the count is unaffected, which was the whole point of my question. > If you want a variable that records/supplies some value across *all* > instances of a class, use a class variable. (Or use a global variable -- it > would have the same effect.) > > If you want a variable whose value is unique to each instance of a class, > then make it an instance variable. > > Gary Herron I never thought that such simple question would turn into this. David Stanek gave me the answer I was looking for (thank you). You, on the other hand, are still going after the wrong issue. Once again, here's the same example using Terry's suggestions. No class instances are being destroyed until the very end. class Case1(object): def __init__(self): self.count = 0 self.list = [] def inc(self): self.count += 1 self.list.append(self.count) def val(self): return (self.count, self.list) class Case2(object): count = 0 list = [] def inc(self): self.count += 1 self.list.append(self.count) def val(self): return (self.count, self.list) c1a, c1b = Case1(), Case1() c2a, c2b = Case2(), Case2() c1a.inc(), c1b.inc() c2a.inc(), c2b.inc() print c1a.val(), c1b.val(), c2a.val(), c2b.val() And the output: (1, [1]), (1, [1]), (1, [1, 1]), (1, [1, 1]) The first element of every tuple is the same. This is the count, which is immutable. The second element is not the same for c2[a,b]. This is the list, which is mutable. My question was about immutable values, and for those the who cases are identical. In the second case, c2a and c2b begin with count referring to the same '0' object. This is where the time and space savings are made. But because that object is immutable, when += 1 operation is performed, a copy is made for each instance. At that point, I am not sharing any values between class instances. The whole point is that this is a quick(er) way of providing initial values for class instance variables when those values are immutable. When/if that initial value is changed, a copy is made. In Case1, that copy is made from the very begging in __init__. Please try to understand what the question is about before responding to it. - Max -- http://mail.python.org/mailman/listinfo/python-list
Re: Style question - defining immutable class data members
On Sat, Mar 14, 2009 at 5:38 PM, Matthew Woodcraft wrote: > Gary Herron writes: > I think this code is in poor taste: it's clear that it will confuse > people (which is what Maxim was asking about in the first place). Yes, I see that now, thanks :) - Max -- http://mail.python.org/mailman/listinfo/python-list
Re: Script for a project inside own directory
2009/3/22 Filip Gruszczyński :
> I am having a project built like this:
>
> project
> module1.py
> module2.py
> packages1/
> module3.py
>
> etc.
>
> I have script that uses objects from those modules/packages. If I keep
> this script inside project directory it's ok and it works. But I would
> like to move it to own scripts directory and from there it doesn't
> work. I think I understand why it doesn't work (the root of local
> packages and modules is there and it can't see what it above it), but
> I would like to ask if there is any workaround? I would like to keep
> all my scripts in separate dir instead of main dir. I like to keep it
> clean.
import sys
sys.path.append('')
If project directory is one level up, you can do something like this:
import os
import sys
sys.path.append(os.path.realpath('..'))
- Max
--
http://mail.python.org/mailman/listinfo/python-list
Re: How Get Name Of Working File
On Sun, Mar 22, 2009 at 10:58 AM, Christian Heimes wrote: > Victor Subervi schrieb: >> Hi; >> If I am writing a script that generates HTML, how do I grab the name of the >> actual file in which I am working? For example, let us say I am working in >> test.py. I can have the following code: >> >> import os >> dir = os.getcwd() >> >> and that will give me the working dir. But what about "test.py"? > > The module variable __file__ contains the file name of the current > Python module. Keep in mind that __file__ may be set to test.pyc or test.pyo. If you always want the .py extension, do this: from os.path import splitext file = splitext(__file__)[0] + '.py' - Max -- http://mail.python.org/mailman/listinfo/python-list
Re: Relative Imports, why the hell is it so hard?
On Mon, Mar 23, 2009 at 10:16 AM, CinnamonDonkey wrote: > Hi All, > > I'm fairly new to Python so I still have a lot to learn. But I'd like > to know how to correectly use relative imports. > > Please, please... please! don't go off on rants about why you think > relative imports should not be used. I've got 15+ years in C++ and > relative inclusion of other sections of code has never been a problem. > As far as I am concerned what I am trying to do is perfectly > reasonable and valid. > > Thank you in advance to everyone who helps solve this, because I just > don't get it. > > Example: > > \ App > | main.py > +--\subpack1 > | | __init__.py > | | module1.py > | > +--\subpack2 > | | __init__.py > | | module2.py > > > Module1 needs to access functionality in Module2. > > #module1.py > from ..subpack2 import module2 > > Seems reasonable to me... but it just does not work and I was so > liking Python. :( Relative imports are perfectly fine, in my opinion. Do you have "from __future__ import absolute_import" at the top of module1.py? Should work fine once you add that line. - Max -- http://mail.python.org/mailman/listinfo/python-list
Re: Relative Imports, why the hell is it so hard?
On Mon, Mar 23, 2009 at 11:22 AM, CinnamonDonkey wrote: > Looking at http://www.python.org/dev/peps/pep-0328/#guido-s-decision > would suggest, unless I am completely miss-understanding the example, > that '.' refers to the current level and '..' pops up a level. That is correct, but you cannot jump beyond the parent package, which is why your code isn't working. > Max, thank you for the response... I tried adding "from __future__ > import absolute_import" which made no difference. I still get exactly > the same error messages. Perhaps I should have mentioned that I am > using Python 2.5, which I understand alread supports relative imports > out of the box. I'll keep this line in for now anyway though :-) > Cheers! Sorry, I use that line to avoid conflicts with standard modules, and forgot that relative imports are already enabled. Basically, the reason your code doesn't work is because you're trying to use relative imports between two separate packages. As far as I know, this isn't possible. What I did to get your code working was move main.py one directory up and create an empty __init__.py under \App. The following code should then work: # main.py import App.subpack1.module1 if __name__ == "__main__": App.subpack1.module1.subpack1_module1_foo() # App.subpack1.module1 from ..subpack2 import module2 def subpack1_module1_foo(): print "subpack1_module1_foo()" call_subpack2_module1() def call_subpack2_module1(): module2.subpack2_module2_foo() # App.subpack2.module2 def subpack2_module2_foo(): print "subpack2_module2_foo()" - Max -- http://mail.python.org/mailman/listinfo/python-list
Re: Relative Imports, why the hell is it so hard?
On Mon, Mar 23, 2009 at 12:19 PM, CinnamonDonkey wrote: > My applogies if this is a silly question... but what makes something a > package? and does that mean that what I am trying to do is not > possible ? A package is a directory that has an __init__.py file. That file can be empty, or contain some initialization code. In your original example, subpack1 and subpack2 are packages. By adding an empty __init__.py file under \App, I made App into a package, which allowed me to execute "import App.subpack1.module1" in main.py. See the following url for additional info: http://docs.python.org/tutorial/modules.html - Max -- http://mail.python.org/mailman/listinfo/python-list
Re: Relative Imports, why the hell is it so hard?
On Tue, Mar 24, 2009 at 5:05 AM, CinnamonDonkey
wrote:
> Thanx Max - your explanation sorted it :-), and a big thank you to
> everyone else also!
>
> >From the various posts, Python considers any directory containing the
> __init__.py file to be a package. The top level package is the highest
> directory (closest to root) with a __init__.py file.
>
> Inter-package communication is not allowed unless the packages
> themselves are contained by a parent package.
>
> How does this relate to the site-packages folder? Is it a top level
> package for all installed packages?
>
> Let's say I have installed the "Trac" system which uses "Genshi", they
> are both packages. They are also both installed at the same level and
> I know "Trac" uses "Genshi" to work. \Python25\Lib\site-packages does
> not contain a __init__.py file so it is not a package (i.e. not a
> parent package to "Trac" and "Genshi") :0.
Trac does not use relative imports to access genshi. When relative
imports are not used, python goes through sys.path list to find
modules (with a small exception made when absolute_imports are not
enabled, but that should be default in 2.7). The site-packages
directory is added to sys.path, so when trac executes something like
"from genshi import some_module", python will look in site-packages,
among other directories, for a directory called "genshi" that contains
an __init__.py file.
When you execute a script, the directory of that script is
automatically added to sys.path, so with your example you could have
used absolute imports between subpack1 and subpack2, with the \App
directory performing the same function as site-packages (Gabriel's
suggestion). This is for your original version of the code when
main.py was under App.
Once you moved main.py outside of \App, running "import sybpack2"
would no longer work. You can, however, append directories to
sys.path, so by doing the following in main.py you could again allow
non-relative imports between subpack1 and subpack2:
import os
import sys
sys.path.append(os.path.realpath('App'))
- Max
--
http://mail.python.org/mailman/listinfo/python-list
Re: Unit testing frameworks
On Tue, Mar 24, 2009 at 8:06 AM, wrote:
> I am looking for a unit testing framework for Python. I am aware of
> nose, but was wondering if there are any others that will
> automatically find and run all tests under a directory hierarchy.
Have you already looked at the unittest module? Below is the code I
use for one of my current projects to load all test cases in package.
This code is sitting in __init__.py, and the test cases are in
separate files (util.py, util_threading.py, etc.). Those files can
contain as many TestCase classes as needed, all are loaded with
loadTestsFromModule. You could easily modify this code to
automatically generate the modules list if you want to.
# repo/pypaq/test/__init__.py
from unittest import TestSuite, defaultTestLoader
import logging
import sys
__all__ = ['all_tests']
modules = ['util', 'util_buffer', 'util_event', 'util_threading']
if not __debug__:
raise RuntimeError('test suite must be executed in debug mode')
all_tests = []
for name in modules:
module = __import__('pypaq.test', globals(), locals(), [name], 0)
tests = defaultTestLoader.loadTestsFromModule(getattr(module, name))
__all__.append(name)
all_tests.append(tests)
setattr(sys.modules[__name__], name, tests)
logging.getLogger().setLevel(logging.INFO)
all_tests = TestSuite(all_tests)
I then have test_pypaq.py file under repo/, with which I can execute
all_tests or only the tests from a specific module:
# repo/test_pypaq.py
from unittest import TextTestRunner
from pypaq.test import *
TextTestRunner(verbosity=2).run(all_tests)
- Max
--
http://mail.python.org/mailman/listinfo/python-list
Re: Relative Imports, why the hell is it so hard?
On Tue, Mar 24, 2009 at 8:57 PM, Istvan Albert wrote: > Does it not bother you that a module that uses relative imports cannot > be run on its own anymore? $ python --help -m mod : run library module as a script (terminates option list) $ python -m some.module.name Works perfectly fine with relative imports. - Max -- http://mail.python.org/mailman/listinfo/python-list
Re: OpenGL win32 Python
On Fri, Apr 17, 2009 at 9:27 AM, gintare statkute wrote: > Hello, > > i found an example for OpenGL in windows. > It is incredibly helpful, but how to rewrite it to be useful in Python. > > How to give address of pfd in Python?: > iFormat = ChoosePixelFormat( hDC, &pfd ); > SetPixelFormat( hDC, iFormat, &pfd ); Take a look at ctypes.byref and ctypes.pointer. Post the results if you get it to work :) - Max -- http://mail.python.org/mailman/listinfo/python-list
Re: Efficient bits manipulation in Python
On Tue, Apr 28, 2009 at 7:26 AM, Li Wang wrote: > Hi: > > I have a bit-code :'1011011', how can I reverse it to '1101101'? > > Another question is I know how to transform the string '110' into > integer 6, does anyone know how to transform integer 6 to a string > '110'? > > Thank you very much:) Assuming that you are using 2.6: a = 0b1011011 print bin(a)[:1:-1] a = 6 print bin(a)[2:] - Max -- http://mail.python.org/mailman/listinfo/python-list
Re: How to timeout when waiting for raw_input from user ?
On Fri, Dec 4, 2009 at 6:55 PM, northof40 wrote:
> On Dec 5, 12:52 pm, northof40 wrote:
>> Hi - I'm writing a *very* simple program for my kids. It asks the user
>> to give it the answer to a maths question and says "right" or "wrong"
>>
>> They now want a timed version where they would only get so long to
>> respond to the question.
>>
>> I'm thinking of some logic where a raw_input call is executed and then
>> if more than X seconds elapses before the prompt is replied to the
>> process writes a message "Sorry too slow" (or similar).
>>
>> I can't see the wood for the trees here - what's the best way to do
>> this given the rather simple environment it's needed within.
>>
>> Regards
>>
>> richard.
>
> Sorry I should said that based upon other answers I've seen to similar
> questions this needs to run on a windows machine (other answers
> suggest this is more difficult than running on *nix)
>
Simplest solution I could come up with. This is indeed much easier on
*nix (just use select.select on sys.stdin with a timeout).
---
from msvcrt import getch, kbhit, putch
from time import sleep, time
ans = ''
end = time() + 5
print('2 + 2 = ?')
while True:
while time() < end:
if kbhit():
break
else:
sleep(0.001)
else:
ans = None
break
char = getch()
if char == '\r':
print('')
break
ans += char
putch(char)
if ans is None:
print('\nSorry too slow')
else:
try:
print('right' if int(ans) == 4 else 'wrong')
except:
print('not a number')
---
- Max
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to timeout when waiting for raw_input from user ?
On Sat, Dec 5, 2009 at 9:01 AM, Rune Strand wrote: > The easiest wasy is to use the Timer object in the threading module. > > > from threading import Timer Doesn't work on Windows. - Max -- http://mail.python.org/mailman/listinfo/python-list
Re: How to timeout when waiting for raw_input from user ?
On Sat, Dec 5, 2009 at 9:11 AM, Rune Strand wrote: > On Dec 5, 3:07 pm, Maxim Khitrov wrote: >> >> Doesn't work on Windows. >> >> - Max > > Yes, it does. I've used it a lot, also in Py2Exe apps. Try the > documentation example yourself > > def hello(): > print "hello, world" > > t = Timer(30.0, hello) > t.start() # after 30 seconds, "hello, world" will be printed I'm not talking about the Timer, I'm talking about the original question. There's nothing (that I know of) you can do with a Timer on Windows to interrupt a raw_input call. - Max -- http://mail.python.org/mailman/listinfo/python-list
Inconsistent raw_input behavior after Ctrl-C
Hello all, I ran into a rather strange problem when interrupting a raw_input call with Ctrl-C. This is with python 2.6.3 on Windows 7. When the call is interrupted, one of two things happen - either a KeyboardInterrupt exception is raised or raw_input raises EOFError, and KeyboardInterrupt is raised a line or two later. Here's the example that I'm testing with: import sys import traceback print '===' excs = [] try: try: raw_input() except BaseException as exc: excs.append(sys.exc_info()) print '1', type(exc) except BaseException as exc: excs.append(sys.exc_info()) print '2', type(exc) print '---' for exc in excs: traceback.print_exception(*exc) print '===' And here are all the two different outputs that I've received at random times: === 1 --- Traceback (most recent call last): File "client.py", line 26, in raw_input() KeyboardInterrupt === === 1 2 --- Traceback (most recent call last): File "client.py", line 26, in raw_input() EOFError Traceback (most recent call last): File "client.py", line 29, in print '1', type(exc) KeyboardInterrupt === This makes no sense to me. First, where does raw_input get EOF (Ctrl-Z or F6) from? Second, why is KeyboardInterrupt raised in the middle of executing a print instruction and not at raw_input? Third, if the inner except clause finishes too soon (for example, if I comment out the print statement), then the KeyboardInterrupt is sometimes raised at the print '---' line. This makes it difficult to consistently handle a Ctrl-C event without calling something like sleep after an EOFError. I don't recall seeing this problem in Windows XP, but I'm not able to test on it right now. Is this problem related to Windows 7 in some way? - Max -- http://mail.python.org/mailman/listinfo/python-list
Re: Inconsistent raw_input behavior after Ctrl-C
On Tue, Oct 20, 2009 at 6:09 PM, Maxim Khitrov wrote: > Hello all, > > I ran into a rather strange problem when interrupting a raw_input call > with Ctrl-C. This is with python 2.6.3 on Windows 7. When the call is > interrupted, one of two things happen - either a KeyboardInterrupt > exception is raised or raw_input raises EOFError, and > KeyboardInterrupt is raised a line or two later. > > This makes no sense to me. First, where does raw_input get EOF (Ctrl-Z > or F6) from? Second, why is KeyboardInterrupt raised in the middle of > executing a print instruction and not at raw_input? Third, if the > inner except clause finishes too soon (for example, if I comment out > the print statement), then the KeyboardInterrupt is sometimes raised > at the print '---' line. This makes it difficult to consistently > handle a Ctrl-C event without calling something like sleep after an > EOFError. > > I don't recall seeing this problem in Windows XP, but I'm not able to > test on it right now. Is this problem related to Windows 7 in some > way? > > - Max > Replying to my own post... The code below seems to fix the problem, though it is obviously a hack I would rather do without. I have no idea if 50ms is enough to wait for a KeyboardInterrupt, but so far I've not encountered any inconsistent behavior. import __builtin__ import time def raw_input2(prompt=''): """ Workaround for raw_input raising EOFError and KeyboardInterrupt on Ctrl-C. """ try: return raw_input1(prompt) except EOFError as exc: # If KeyboardInterrupt is not raised in 50ms, it's a real EOF event. time.sleep(0.05) raise raw_input1 = raw_input __builtin__.raw_input = raw_input2 try: raw_input() except BaseException as exc: print type(exc) -- http://mail.python.org/mailman/listinfo/python-list
Re: comparing alternatives to py2exe
On Tue, Nov 3, 2009 at 10:58 AM, Jonathan Hartley wrote: > Hi, > > Recently I put together this incomplete comparison chart in an attempt > to choose between the different alternatives to py2exe: > > http://spreadsheets.google.com/pub?key=tZ42hjaRunvkObFq0bKxVdg&output=html > > Columns represent methods of deploying to end-users such that they > don't have to worry about installing Python, packages or other > dependencies. 'Bundle' represents manually bundling an interpreter > with your app. 'Bootstrap' represents a fanciful idea of mine to > include an installer that downloads and installs an interpreter if > necessary. This sounds fiddly, since it would have to install side-by- > side with any existing interpreters of the wrong version, without > breaking anything. Has anyone done this? Maybe there is a way to use Portable Python for this, but I have no experience with it. > The remaining columns represent the projects out there I could find > which would do the bundling for me. > > Are there major things I'm missing or misunderstanding? > > Perhaps folks on the list would care to rate (+1/-1) rows that they > find important or unimportant, or suggest additional rows that would > be important to them. Maybe an updated and complete version of this > table would help people agree on what's important, and help the > various projects to improve faster. > > Best regards, > > Jonathan Good work. Recently I played with cx_freeze and compared it to py2exe, which I've been using for a while. Here are my findings: 1. I don't think cx_freeze supports single exe. I haven't even been able to get it to append the generated library.zip file to the executable using documented options. Other things like .pyd files always seem to be separate. At the same time, singe executables generated by py2exe do not always work. I have a program that works fine on Windows XP, Vista, and 7 if it is built under XP. However, if I build the exact same program under Windows 7, it no longer works on Vista or XP. I'm sure it has something to do with SxS or other dll issues. 2. For output directory structure, you are able to specify where to put the generated executable and all of its dependencies with both py2exe and cx_freeze. You cannot do things like put python26.dll in a separate directory from the executable. Not sure if that is what you are referring to. 3. py2exe does not support Python 3 (unfortunately). 4. Although cx_freeze does support optimization (-O), it's a bit broken in that the __debug__ variable is always set to True. In other words, the code is optimized and things like assert statements are not executed, but conditional statements that check __debug__ == True are. I know that py2exe does not have this problem, no experience with other tools. 5. py2exe is capable of generating smaller executables than cx_freeze because of the base executable size (18.5 KB vs 1.35 MB). This is offset by the fact that py2exe saves many more standard library components to library.zip by default. In a quick test I just ran, both generated a package of 4.03 MB, but I can remove at least a meg from py2exe's library.zip. Rather than "distribution size", I think it makes more sense to show "overhead" above the required components (exclude minimal library.zip, python dll, and pyd files). 6. cx_freeze is as easy to use as py2exe after looking at the bundled examples. - Max -- http://mail.python.org/mailman/listinfo/python-list
Re: comparing alternatives to py2exe
On Tue, Nov 3, 2009 at 3:50 PM, iu2 wrote: > On Nov 3, 5:58 pm, Jonathan Hartley wrote: >> Hi, >> >> Recently I put together this incomplete comparison chart in an attempt >> to choose between the different alternatives to py2exe: >> >> http://spreadsheets.google.com/pub?key=tZ42hjaRunvkObFq0bKxVdg&output... >> >> Columns represent methods of deploying to end-users such that they >> don't have to worry about installing Python, packages or other >> dependencies. 'Bundle' represents manually bundling an interpreter >> with your app. 'Bootstrap' represents a fanciful idea of mine to >> include an installer that downloads and installs an interpreter if >> necessary. This sounds fiddly, since it would have to install side-by- >> side with any existing interpreters of the wrong version, without >> breaking anything. Has anyone done this? >> >> The remaining columns represent the projects out there I could find >> which would do the bundling for me. >> >> Are there major things I'm missing or misunderstanding? >> >> Perhaps folks on the list would care to rate (+1/-1) rows that they >> find important or unimportant, or suggest additional rows that would >> be important to them. Maybe an updated and complete version of this >> table would help people agree on what's important, and help the >> various projects to improve faster. >> >> Best regards, >> >> Jonathan > > Another thing that I think is of interest is whether the application > support modifying the version and description of the exe (that is, on > Windows, when you right-click on an application and choose > 'properties' you view the version number and description of the > application, it is a resource inside the exe). I think py2exe supports > it. py2exe supports this, cx_freeze doesn't. - Max -- http://mail.python.org/mailman/listinfo/python-list
Re: Is it possible to get the Physical memory address of a variable in python?
On Tue, Nov 10, 2009 at 6:32 AM, Ognjen Bezanov wrote: > Hey, > > Thanks for all the responses guys. In hindsight I probably should have > explained why on earth I'd need the physical address from an interpreted > language. > > I'm trying to see if there is any way I can make Python share data between > two hosts using DMA transfers over a firewire connection, so avoiding the > need for another layer on top such as IPv4 + Python sockets. > > Thanks to some old python bindings which I updated to python 2.6, I can read > any write to the RAM of any firewire connected host within python. Because > it uses DMA (the cpu is not involved in this at all), I can only specify a > physical address within the 4GB ram limit to read from and write to. > > Now what I've done so far is on the remote host I run python and set a > variable as so: > > a = "foo" > print a > 'foo' > > Then on the local host I run a python script that scans the entire RAM > looking for the string "foo", and replaces it with the string "oof". I have > had success with this method. Once it's done and I do "print a" on the > remote host, I get "oof" as the variable value, so in theory it can work. > > Problem is that it's slow. Scanning 3GB of RAM every time you want to do > this is not a good method. I thought that if I could get python to return > the physical address of where the value of a variable is, then I can just > jump to that address and write the data. > > From what I've been told so far, it's not possible to do this without some > OS-specific (Linux in this case) syscall. Is this correct? > > Thanks! > > > Ognjen If all you need is a memory buffer, array.array provides a buffer_info() method that tells you the current (virtual) address. For DMA, I think there are dma_map_* functions in linux that you may be able to call through ctypes. - Max -- http://mail.python.org/mailman/listinfo/python-list
Re: no return value for threading.Condition.wait(timeout)?
On Thu, Jul 16, 2009 at 5:00 PM, Carl Banks wrote:
> On Jul 16, 8:12 am, Gabriel Rossetti
> wrote:
>> Hello everyone,
>>
>> I am using threading.Condition.wait(timeout) and was surprised to see
>> that there is no return value nor an exception when wait() is used w/ a
>> timeout. How am I supposed to know if it was notified or if it timed out?
>
> That's a good question. Condition.wait seems to violate an invariant
> if it can return without having acquired the underlying lock. And if
> it does the least it could do is to return a status so you know if you
> have to skeddadle.
>
> Workaround is to use the _is_owned() method of Condition to see if you
> are the owner of the condition, but this method is undocumented.
>
> cond = Condition()
> ...
> cond.acquire()
> while not ok_to_proceed():
> cond.wait()
> if not cond._is_owned():
> # must've timed out
> raise TimeOutException
> operate()
> cond.release()
You will always be the owner of the condition whether it times out or
your thread is notified. This doesn't solve the problem. As an aside,
the implementation of wait() is rather poor in any case; the sleep
function should not be used for the purpose of waiting for an event.
Depending on what platform you are using, I can offer two solutions.
The simplest one is to subclass Condition and override the wait
method. Just copy the code as it is, but modify the end of the
function to look like this:
if not gotit:
if __debug__:
self._note("%s.wait(%s): timed out", self, timeout)
try:
self.__waiters.remove(waiter)
except ValueError:
pass
return False
else:
if __debug__:
self._note("%s.wait(%s): got it", self, timeout)
return True
The two return statements will tell you whether you timed out ("if not
gotit" condition), or were notified ("else").
The better way to solve this and the other problem that I mentioned is
to use native OS events. This is what I had to do for a recent
project. I used pywin32 extension to completely rewrite Event,
Condition, and Timer classes to use a true event-based approach. In
the process of the rewrite I also modified some functionality, such as
returning True of False from wait().
Unfortunately, I'm not able to post this code right now, though I can
try and get permission to do so a bit later. If you are writing
software for Windows, then simply take a look at win32event module.
CreateEvent, SetEvent, ResetEvent, WaitForSingleObject are all the
Windows functions that you need to reimplement threading.Event. Using
the new event class you can implement a proper Condition.wait() method
(hint: use new events rather than locks to wait for notification).
If you are on Linux or BSD, I can't help you. I'm sure that there is
some equivalent functionality that you may be able to access using
ctypes. Exactly how to do it, however, is not something that I can
help you with right now.
- Max
--
http://mail.python.org/mailman/listinfo/python-list
Re: Override a method but inherit the docstring
On Thu, Jul 16, 2009 at 9:13 PM, Jean-Paul Calderone wrote: > On Fri, 17 Jul 2009 11:01:49 +1000, Ben Finney > wrote: >> >> Howdy all, >> >> The following is a common idiom:: >> >> class FooGonk(object): >> def frobnicate(self): >> """ Frobnicate this gonk. """ >> basic_implementation(self.wobble) >> >> class BarGonk(FooGonk): >> def frobnicate(self): >> special_implementation(self.warble) >> >> The docstring for ‘FooGonk.frobnicate’ is, intentionally, perfectly >> applicable to the ‘BarGonk.frobnicate’ method also. Yet in overriding >> the method, the original docstring is not associated with it. >> >> Ideally there would be a way to specify that the docstring should be >> inherited. The best I can come up with is:: >> >> class BarGonk(FooGonk): >> def frobnicate(self): >> special_implementation(self.warble) >> frobnicate.__doc__ = FooGonk.frobnicate.__doc__ >> >> but that violates DRY (the association between BarGonk and FooGonk is >> being repeated), puts the docstring assignment awkwardly after the end >> of the method instead of at the beginning where docstrings normally go, >> and reads poorly besides. >> >> What is the most Pythonic, DRY-adherent, and preferably least-ugly >> approach to override a method, but have the same docstring on both >> methods? >> > > How about this? > > class BarGonk(FooGonk): > @inherit_docstring > def frobnicate(self): > special_implementation(self.warble) > > The implementation of "inherit_docstring" is left as an exercise for the > reader (it's not utterly trivial, I admit, as "FooGonk" will not readily > be at hand, but it is still possible). > > By the way, I don't think this is a particularly good idea. Presumably > there is a reason your implementation is special. It would probably be > better if this were reflected in the docstring somehow. Perhaps this > idea is a better one: > > class BarGonk(FooGonk): > @append_to_docstring > def frobnicate(self): > """ > This implementation takes the warble into consideration. > """ > special_implementation(self.warble) > > With the result of BarGonk.frobnicate.__doc__ being set to: > > > Frobnicate this gonk. > > This implementation takes the warble into consideration. Another way is to use a metaclass. Have its __new__ method loop through all attributes and compare those with what is already defined in bases. If you find a match, copy the __doc__ attribute. The advantage here is that it will work for all methods without any additional code, not counting the "__metaclass__ = ..." line. If you define a metaclass for the base, then no modifications are required for any subclasses. I do agree, however, that the best thing to do is to write a very short explanation for what the override is for. - Max -- http://mail.python.org/mailman/listinfo/python-list
Re: How can I get the line number ?
On Fri, Jul 24, 2009 at 2:51 PM, kk wrote:
> Hello
>
> I am writing some Python code that runs in another application(has
> wrapper functions). Due to lack of debugging I am printing out alot of
> outputs and manual messages. I want to be able to create a function
> that would let me print the current line number that is called from.
> This is not for debugging exceptions it is rather to simplify my debug
> messages, at least I can trace my debug messages.
>
> thanks
Modify the following as needed:
from inspect import currentframe, getframeinfo
def caller_info(depth=0):
"""
Get file, line number, and name of the calling function.
"""
if depth < 0:
raise ValueError('invalid stack depth')
caller = frame = currentframe()
try:
for i in xrange(-1, depth):
caller = caller.f_back
if caller is None:
return (None, None, None)
return getframeinfo(caller, 0)[:3]
finally:
del caller, frame
- Max
--
http://mail.python.org/mailman/listinfo/python-list
