Re: Interesting talk on Python vs. Ruby and how he would like Python to have just a bit more syntactic flexibility.

[Jonathan Gardner]

On Fri, Feb 19, 2010 at 11:16 PM, Lie Ryan  wrote:
>
> Now, why don't we start a PEP to make python a fully-functional language
> then?
>

Because people don't think the same way that programs are written in
functional languages.

-- 
Jonathan Gardner
jgard...@jonathangardner.net
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Capturing errors raised by other scripts ?

[Lie Ryan]

On 02/20/10 14:39, northof40 wrote:
> On Feb 20, 4:13 pm, MRAB  wrote:
>> northof40 wrote:
>>> I'm using the subroutine module to run run python script A.py from
>>> B.py (this is on windows fwiw).
>>
>>> A.py is not my script and it may raise arbitary errors before exiting.
>>> How can I determine what's happened before A.py exited ?
>>
>>> To simulate this I've got this script (which is meant to simulate
>>> A.py):
>>
>>> class customError(Exception):
>>>def __init__(self, value):
>>>self.value = value
>>>def __str__(self):
>>>return repr(self.value)
>>
>>> try:
>>>raise customError(2*2)
>>> except customError as e:
>>>print 'Custom exception occurred, value:', e.value
>>
>>> I then run my A.py like this :
>>
>> fdOut, fOut = tempfile.mkstemp(suffix='.txt', prefix='AOut-')
>> fdErr, fErr = tempfile.mkstemp(suffix='.txt', prefix='AErr-')
>> try:
>>> ... pathtojob="python.exe A.py"
>>> ... p = subprocess.Popen(pathtojob, stderr=fdErr, stdout=fdOut)
>>> ... except:
>>> ... print "bad stuff happened"
>>> ...
>>
>>> When I do this I the exception handler is not fired and the text
>>> "Custom exception occurred, value: 4" ends up in the stdout file.



>>> I'd really like it to end up in stderr because then I could say
>>> "anything in stderr ? then ignore the output and flag an error".

Not really; not all that is written to stderr signifies errors per se.
Warning and Debugging info are often written to stderr as well.

> Thanks for your reply. I take your point about print - perhaps I
> hadn't really thought that through. Unfortunately I don't have control
> of the real script that's being run so if the programmer of that
> script has used print there's nothing I can do about it.

Tracebacks is almost always written to stderr, not stdout; unless the
programmer explicitly redirect the traceback into stdout.

> Perhaps I could modify the question. If A.py terminates due an
> unhandled exception is there anyway for B.py to know that's happened
> (without A.py's cooperation ?).

A well-behaved program will exit with status code 0; if and only if it
exited cleanly. I believe a python program that exited due to unhandled
exception always return non-zero status code.

However, if the program handled an error and called exit(0) explicitly
or the execution of the program falls to the end of the module; there is
no way to distinguish it from regular clean exit.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Can't Access ANY url from python (errno 61)

[MattB]

On Feb 20, 2:02 am, Lie Ryan  wrote:
> On 02/20/10 13:32, MattB wrote:
>
>
>
> > I'm using the network in my own apartment. Not the campus's.
> > Moreover, my mac's MAC address is different from the MAC address shown
> > by my router, but as I said I'm also blocked when using my friend's
> > wireless router at his apartment.
>
> > So it must be my mac's MAC, and not the router's MAC, that's being
> > blocked, right?
>
> > But ALSO -- is it my ISP that's blocking the mac's MAC (and not the
> > school), since I can't raise ANY url's from python when I'm on
> > wireless?
>
> MAC or IP blocking can't be the reason, as the OP stated, he can use
> Firefox just fine.
>
> Can you access, say,http://www.google.comfrom urllib or mechanize?
>
> If you can't access *any website* using urllib/mechanize but you can
> with a browser and you're on a unix-based machine, you probably have the
> same problem as I used to have. Check whether you used the same hostname
> in /etc/conf.d/hostname and /etc/hosts (or wherever your distro saves
> its hostname configurations, I use Gentoo); after editing those files
> reboot (there are ways to avoid reboot, but rebooting guarantees the
> conf file is reread).
>
> Check the hostname by running this python script:
>
> import socket
> hn = socket.gethostname()
> print hn
> print socket.gethostbyname(hn) # should be 127.0.0.1

Lie,

Wow. Strangely, that script returned 192.168.1.106. However, in Snow
Leopard's airport settings, if I click on 'advanced' and then
'proxies', the default proxy for 'http' is 127.0.0.1: (and in
these settings, the 'use proxy for http' is checked).

I just tried checking the unix files you mentioned. In etc/hosts, the
following info is displayed:

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##


127.0.0.1   localhost
255.255.255.255 broadcasthost
::1 localhost
fe80::1%lo0 localhost

Also found a file called ntp-restrict.conf, containing:

# Access restrictions documented in ntp.conf(5) and
# http://support.ntp.org/bin/view/Support/AccessRestrictions
# Limit network machines to time queries only

restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery

# localhost is unrestricted
restrict 127.0.0.1
restrict -6 ::1

includefile /private/etc/ntp.conf

Not sure if these are what I'm looking for -- I'm new to unix so I may
need a bit more hand-holding here.

I appreciate your time and effort.

Matt
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Interesting talk on Python vs. Ruby and how he would like Python to have just a bit more syntactic flexibility.

[Chris Rebert]

On Fri, Feb 19, 2010 at 11:17 PM, sjdevn...@yahoo.com
 wrote:
> On Feb 20, 1:30 am, Lawrence D'Oliveiro  central.gen.new_zealand> wrote:
>> If Python doesn’t distinguish between procedures and functions, why should
>> it distinguish between statements and expressions?
>
> Because the latter are different in Python (and in Ruby

I think your Ruby assertion needs fact-checking:

irb(main):001:0> a = 7  # assignments have a value
=> 7
irb(main):002:0> puts(b = 42)  # as further proof
42
=> nil
irb(main):003:0> b
=> 42
irb(main):004:0> c = [6,4,5]
=> [6, 4, 5]
irb(main):005:0> if false
irb(main):006:1>   c.reverse!
irb(main):007:1> else
irb(main):008:1*   c.sort!
irb(main):009:1> end  # even the if-else control structure has a value
=> [4, 5, 6]
irb(main):010:0> begin # same with exception handling
irb(main):011:1*raise "a runtime error"
irb(main):012:1> rescue RuntimeError
irb(main):013:1>   "sounds bad"
irb(main):014:1> end
=> "sounds bad"
irb(main):015:0> def foo # and same with method bodies
irb(main):016:1>   99
irb(main):017:1> end
=> nil
irb(main):018:0> foo
=> 99

Quoth Wikipedia regarding Ruby (programming language):
"For practical purposes there is no distinction between expressions
and statements"

Cheers,
Chris
--
http://blog.rebertia.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Can't Access ANY url from python (errno 61)

[Shashwat Anand]

throw up your 'ifconfig' and mozilla-proxy output here. It seems you don't
know whether you are using proxy.
For mozilla proxy :
open mozilla -> cmd + "," -> Network -> Settings -> Paste everything that is
there/ may be take a snapshot and upload a link.

~l0nwlf


On Sat, Feb 20, 2010 at 2:06 PM, MattB  wrote:

> On Feb 20, 2:02 am, Lie Ryan  wrote:
> > On 02/20/10 13:32, MattB wrote:
> >
> >
> >
> > > I'm using the network in my own apartment. Not the campus's.
> > > Moreover, my mac's MAC address is different from the MAC address shown
> > > by my router, but as I said I'm also blocked when using my friend's
> > > wireless router at his apartment.
> >
> > > So it must be my mac's MAC, and not the router's MAC, that's being
> > > blocked, right?
> >
> > > But ALSO -- is it my ISP that's blocking the mac's MAC (and not the
> > > school), since I can't raise ANY url's from python when I'm on
> > > wireless?
> >
> > MAC or IP blocking can't be the reason, as the OP stated, he can use
> > Firefox just fine.
> >
> > Can you access, say,http://www.google.comfrom urllib or mechanize?
> >
> > If you can't access *any website* using urllib/mechanize but you can
> > with a browser and you're on a unix-based machine, you probably have the
> > same problem as I used to have. Check whether you used the same hostname
> > in /etc/conf.d/hostname and /etc/hosts (or wherever your distro saves
> > its hostname configurations, I use Gentoo); after editing those files
> > reboot (there are ways to avoid reboot, but rebooting guarantees the
> > conf file is reread).
> >
> > Check the hostname by running this python script:
> >
> > import socket
> > hn = socket.gethostname()
> > print hn
> > print socket.gethostbyname(hn) # should be 127.0.0.1
>
> Lie,
>
> Wow. Strangely, that script returned 192.168.1.106. However, in Snow
> Leopard's airport settings, if I click on 'advanced' and then
> 'proxies', the default proxy for 'http' is 127.0.0.1: (and in
> these settings, the 'use proxy for http' is checked).
>
> I just tried checking the unix files you mentioned. In etc/hosts, the
> following info is displayed:
>
> ##
> # Host Database
> #
> # localhost is used to configure the loopback interface
> # when the system is booting.  Do not change this entry.
> ##
>
>
> 127.0.0.1   localhost
> 255.255.255.255 broadcasthost
> ::1 localhost
> fe80::1%lo0 localhost
>
> Also found a file called ntp-restrict.conf, containing:
>
> # Access restrictions documented in ntp.conf(5) and
> # http://support.ntp.org/bin/view/Support/AccessRestrictions
> # Limit network machines to time queries only
>
> restrict default kod nomodify notrap nopeer noquery
> restrict -6 default kod nomodify notrap nopeer noquery
>
> # localhost is unrestricted
> restrict 127.0.0.1
> restrict -6 ::1
>
> includefile /private/etc/ntp.conf
>
> Not sure if these are what I'm looking for -- I'm new to unix so I may
> need a bit more hand-holding here.
>
> I appreciate your time and effort.
>
> Matt
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list

os.pipe() + os.fork()

[Sebastian Noack]

Hi,

I have a problem using os.pipe() together with os.fork(). Usually when
the writing end of the pipe is closed, the reading end gets EOF. So
subsequent attempts to read data will return an empty string. But when
you call os.fork() after you have created a pipe using os.pipe(), and
read data from the pipe in the child process, when the partent process
has already closed the writing end, the reading end does not get EOF.
Instead of the os.read() call in the child process is blocking.

I am using Linux, so I would like to know if this is a bug in Python or
just weird behaviour under Linux and if it is possible work around it.

Regards
Sebastian Noack


---

import os

def test_pipe_sync():
pipe = os.pipe()

os.write(pipe[1], 'Spam')
os.close(pipe[1])

print repr(os.read(pipe[0], 4))# 'Spam' is printed.
print repr(os.read(pipe[0], 4))# '' is printed, because of EOF is 
reached.

def test_pipe_forked():
pipe = os.pipe()
pid  = os.fork()

if pid:
os.write(pipe[1], 'Spam')
os.close(pipe[1])

os.waitpid(pid, 0)
else:
print repr(os.read(pipe[0], 4))# 'Spam' is printed.
print repr(os.read(pipe[0], 4))# Nothing is printed and os.read()
   # is blocking, eventhough the other
   # end of the pipe was closed.

if __name__ == '__main__':
test_pipe_sync()

print
print '-' * 80
print

test_pipe_forked()


signature.asc
Description: PGP signature
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Avoid converting functions to methods in a class

[Arnaud Delobelle]

On 20 Feb, 03:33, Steven D'Aprano  wrote:
> I have a convention when writing unit tests to put the target of the test
> into a class attribute, as follows:
>
> class MyTest(unittest.TestCase):
>     target = mymodule.someclass
>
>     def test_spam(self):
>         """Test that someclass has a spam attribute."""
>         self.failUnless(hasattr(self.target, 'spam'))
>
> It works well until I write a test for stand-alone functions:
>
> class AnotherTest(unittest.TestCase):
>     target = mymodule.function
>
>     def test_foo(self):
>         self.assertEquals(self.target('a', 'b'), 'foo')
>
> The problem is that target is turned into a method of my test class, not
> a standalone function, and I get errors like:
>
> TypeError: function() takes exactly 2 arguments (3 given)
>
> The solution I currently use is to drop the target attribute in this
> class, and just refer to mymodule.function in each individual test. I
> don't like this solution because it violates Once And Only Once: if the
> function changes name, I have to make many edits to the test suite rather
> than just one.
>
> Are there any better solutions?
>
> --
> Steven

Why not define target in the TestCase.setUp() method?

class AnotherTest(unittest.TestCase):

def setUp(self):
self.target = mymodule.function

def test_foo(self):
self.assertEquals(self.target('a', 'b'), 'foo')

--
Arnaud



class
--
Arnaud
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Scalable python dict {'key_is_a_string': [count, some_val]}

[Arnaud Delobelle]

On 20 Feb, 06:36, krishna  wrote:
> I have to manage a couple of dicts with huge dataset (larger than
> feasible with the memory on my system), it basically has a key which
> is a string (actually a tuple converted to a string) and a two item
> list as value, with one element in the list being a count related to
> the key. I have to at the end sort this dictionary by the count.
>
> The platform is linux. I am planning to implement it by setting a
> threshold beyond which I write the data into files (3 columns: 'key
> count some_val' ) and later merge those files (I plan to sort the
> individual files by the key column and walk through the files with one
> pointer per file and merge them; I would add up the counts when
> entries from two files match by key) and sorting using the 'sort'
> command. Thus the bottleneck is the 'sort' command.
>
> Any suggestions, comments?
>
> By the way, is there a linux command that does the merging part?
>
> Thanks,
> Krishna

Have you looked here? http://docs.python.org/library/persistence.html

--
Arnaud
-- 
http://mail.python.org/mailman/listinfo/python-list

Precision issue in python

[mukesh tiwari]

Hello everyone. I think it is  related to the precision with double
arithmetic so i posted here.I am trying with this problem (https://
www.spoj.pl/problems/CALCULAT) and the problem say that "Note : for
all test cases whose N>=100, its K<=15." I know precision of doubles
in c is 16 digits. Could some one please help me with this precision
issue.I used stirling (http://en.wikipedia.org/wiki/
Stirling's_approximation) to calculate the first k digits of N.
Thank you.

__author__="Administrator"
__date__ ="$Feb 19, 2010 3:16:47 PM$"
import math
if __name__ == "__main__":
   n=int(raw_input())
   while(n>0):
   n-=1
   raw_input()
   l=raw_input();
   m=l.split(" ")
   N,K,L=int(m[0]),int(m[1]),int(m[2])
   fwd,bkd=1,1
   s_1,s_2="",""
   if(N<=200):
for i in range(1,N+1):
fwd=fwd*i;
s=str(fwd)
s_1=s[:K]
   else:
   d_1=(N*math.log(N)-N+0.5*math.log(2*math.pi*N)+(1/12/N)-
(1/360/pow(N,3))+(1/1260/pow(N,5))-(1/1680/pow(N,7))+(1/1188/pow(N,9))-
(691/2730/12/11/pow(N,11))+(7/6/14/13/pow(N,13)))*math.log10(math.e)
   d_2=d_1-int(d_1)+K-1
   fwd=pow(10,d_2)
   #print fwd
   fwd=int(fwd)
   s_1=str(fwd)

   if(N<=500):
   for i in range(1,N+1):
bkd=bkd*i
bkd=bkd%pow(10,L)
   if(bkd==0):
s_2="0"*L
   else:
s_2=str(bkd)
   else:
   s_2="0"*L

   print s_1+" "+s_2


-- 
http://mail.python.org/mailman/listinfo/python-list

Shared web host good citizenship question

[Carl Banks]

Not specifically Python related but thought I'd ask.

If you have a low-bandwidth website running on a shared web hosting
service as a WSGI server (or FastCGI, or any other interface with its
own process), is it considered a responsible thing for the process to
exit on its own after a period of non-activity, or is that something
best left for the host to manage?

I'd guess that modern web hosts would prefer that processes stay
running, even if they go long periods without use, rather than having
to periodically restart processes.

Carl Banks
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: os.pipe() + os.fork()

[Sebastian Noack]

I have figured out that, you have to close the writing end in the child
process, which is reading from the pipe. Otherwise the underlying pipe
is not going to be closed when the parent process is closing its
writing end. This has nothing to do with Python itself. I have tried
plain C and there it is the same behaviour.

Regards
Sebastian Noack


signature.asc
Description: PGP signature
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Precision issue in python

[Mark Dickinson]

On Feb 20, 11:17 am, mukesh tiwari 
wrote:
> Hello everyone. I think it is  related to the precision with double
> arithmetic so i posted here.I am trying with this problem 
> (https://www.spoj.pl/problems/CALCULAT) and the problem say that "Note : for
> all test cases whose N>=100, its K<=15." I know precision of doubles
> in c is 16 digits. Could some one please help me with this precision
> issue.I used stirling (http://en.wikipedia.org/wiki/
> Stirling's_approximation) to calculate the first k digits of N.
> Thank you.

If I understand you correctly, you're trying to compute the first k
digits in the decimal expansion of N!, with bounds of k <= 15 and 100
<= N < 10**8.  Is that right?

Python's floats simply don't give you enough precision for this:
you'd need to find the fractional part of log10(N!) with >= 15 digits
of precision.  But the integral part needs ~ log10(log10(N!)) digits,
so you end up needing to compute log10(N!) with at least 15 +
log10(log10(N!)) ~ 15 + log10(N) + log10(log10(N)) digits (plus a few
extra digits for safety);  so that's at least 25 digits needed for N
close to 10**8.

The decimal module would get you the results you need (are you allowed
imports?).  Or you could roll your own log implementation based on
integer arithmetic.

--
Mark
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Interesting talk on Python vs. Ruby and how he would like Python to have just a bit more syntactic flexibility.

[Michael Sparks]

On Feb 18, 4:15 pm, Steve Howell  wrote:
...
>     def print_numbers()
>         [1, 2, 3, 4, 5, 6].map { |n|
>             [n * n, n * n * n]
>         }.reject { |square, cube|
>             square == 25 || cube == 64
>         }.map { |square, cube|
>             cube
>         }.each { |n|
>             puts n
>         }
>     end

This strikes me as a terrible example. For example, this is
significantly clearer:
def print_numbers()
for n in [1,2,3,4,5,6]:
square, cube = n * n, n * n * n
if square != 25 and cube != 64:
print n

I /can/ see arguments for ruby style blocks in python, but not for
this sort of thing, or lisp style quoted expressions[1]. ie I can see
situations where you have more complex code in real life where they
will definitely simplify things.

[1] This is perhaps more appropriate because '(a b c) is equivalent
to (quote a b c), and quote a b c can be viewed as close to
python's expression "lambda: a b c"

However, I can also see that in simple situations - such as the
example you post - they will have a tendency to make code
significantly less clear/direct.

I suppose, if I have a choice between something (hard being possible &
simple code looking simple) and (hard things being simpler & simple
things looking harder), I'd probably personally choose the former.
This is not because I don't like hard things being simple, but because
I think that simple things are more common and making them look harder
is a mistake.

I'm well aware that's opinion however,

Regards,


Michael.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Precision issue in python

[Shashwat Anand]

A quick solution I came out with, no stirling numbers and had tried to avoid
large integer multiplication as much as possible.

import math

for i in range(int(raw_input())):
n, k, l = [int(i) for i in raw_input().split()]
e = sum(math.log10(i) for i in range(1, n+1))
frac_e = e - math.floor(e)
a = str(10**frac_e * 10**(k - 1)).split('.')[0]

b  = 1
for i in range(n, 0, -1):
b = (b * i) % 10**l
lb = len(str(b))
if lb < l:
b = str(b) + '0'* (l - lb)

print a, b

~l0nwlf

On Sat, Feb 20, 2010 at 6:14 PM, Mark Dickinson  wrote:

> On Feb 20, 11:17 am, mukesh tiwari 
> wrote:
> > Hello everyone. I think it is  related to the precision with double
> > arithmetic so i posted here.I am trying with this problem (
> https://www.spoj.pl/problems/CALCULAT) and the problem say that "Note :
> for
> > all test cases whose N>=100, its K<=15." I know precision of doubles
> > in c is 16 digits. Could some one please help me with this precision
> > issue.I used stirling (http://en.wikipedia.org/wiki/
> > Stirling's_approximation) to calculate the first k digits of N.
> > Thank you.
>
> If I understand you correctly, you're trying to compute the first k
> digits in the decimal expansion of N!, with bounds of k <= 15 and 100
> <= N < 10**8.  Is that right?
>
> Python's floats simply don't give you enough precision for this:
> you'd need to find the fractional part of log10(N!) with >= 15 digits
> of precision.  But the integral part needs ~ log10(log10(N!)) digits,
> so you end up needing to compute log10(N!) with at least 15 +
> log10(log10(N!)) ~ 15 + log10(N) + log10(log10(N)) digits (plus a few
> extra digits for safety);  so that's at least 25 digits needed for N
> close to 10**8.
>
> The decimal module would get you the results you need (are you allowed
> imports?).  Or you could roll your own log implementation based on
> integer arithmetic.
>
> --
> Mark
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Looking for crossfold validation code

[Sandy]

Following is the code I use. I got it from web, but forgot the link.

def k_fold_cross_validation(X, K, randomise = False):
"""
Generates K (training, validation) pairs from the items in X.

Each pair is a partition of X, where validation is an iterable
of length len(X)/K. So each training iterable is of length
(K-1)*len(X)/K.

If randomise is true, a copy of X is shuffled before partitioning,
otherwise its order is preserved in training and validation.
"""
if randomise: from random import shuffle; X=list(X); shuffle(X)
for k in xrange(K):
training = [x for i, x in enumerate(X) if i % K != k]
validation = [x for i, x in enumerate(X) if i % K == k]
yield training, validation


Cheers,
dksr

On Feb 20, 1:15 am, Mark Livingstone 
wrote:
> Hello,
>
> I am doing research as part of a Uni research Scholarship into using
> data compression for classification. What I am looking for is python
> code to handle the crossfold validation side of things for me - that
> will take my testing / training corpus and create the testing /
> training files after asking me for number of folds and number of times
> (or maybe allow me to enter a random seed or offset instead of times.)
> I could then either hook my classifier into the program or use it in a
> separate step.
>
> Probably not very hard to write, but why reinvent the wheel ;-)
>
> Thanks in advance,
>
> MarkL

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Precision issue in python

[mukesh tiwari]

On Feb 20, 5:44 pm, Mark Dickinson  wrote:
> On Feb 20, 11:17 am, mukesh tiwari 
> wrote:
>
> > Hello everyone. I think it is  related to the precision with double
> > arithmetic so i posted here.I am trying with this problem 
> > (https://www.spoj.pl/problems/CALCULAT) and the problem say that "Note : for
> > all test cases whose N>=100, its K<=15." I know precision of doubles
> > in c is 16 digits. Could some one please help me with this precision
> > issue.I used stirling (http://en.wikipedia.org/wiki/
> > Stirling's_approximation) to calculate the first k digits of N.
> > Thank you.
>
> If I understand you correctly, you're trying to compute the first k
> digits in the decimal expansion of N!, with bounds of k <= 15 and 100
> <= N < 10**8.  Is that right?
>
> Python's floats simply don't give you enough precision for this:
> you'd need to find the fractional part of log10(N!) with >= 15 digits
> of precision.  But the integral part needs ~ log10(log10(N!)) digits,
> so you end up needing to compute log10(N!) with at least 15 +
> log10(log10(N!)) ~ 15 + log10(N) + log10(log10(N)) digits (plus a few
> extra digits for safety);  so that's at least 25 digits needed for N
> close to 10**8.
>
> The decimal module would get you the results you need (are you allowed
> imports?).  Or you could roll your own log implementation based on
> integer arithmetic.
>
> --
> Mark

Yes i am trying to first k digits of N!.I will try your method.
Thank you
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Few questions on SOAP

[Muhammad Alkarouri]

Thanks every one for commenting. I guess I misspoke. I meant to say
that the group is not necessarily the best for parts of this question,
so Subhabrata might not get as enthusiastic responses as in some other
lists (which i don't recollect at the moment, sorry). I didn't want to
convey the sense that his question is not welcome, rather that it
might not get a lot of answers.

Thanks Brendon and Steve for the support, and thanks Mark for
correcting my slip.

Regards,

k
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Precision issue in python

[mukesh tiwari]

On Feb 20, 8:13 pm, mukesh tiwari 
wrote:
> On Feb 20, 5:44 pm, Mark Dickinson  wrote:
>
>
>
>
>
> > On Feb 20, 11:17 am, mukesh tiwari 
> > wrote:
>
> > > Hello everyone. I think it is  related to the precision with double
> > > arithmetic so i posted here.I am trying with this problem 
> > > (https://www.spoj.pl/problems/CALCULAT) and the problem say that "Note : 
> > > for
> > > all test cases whose N>=100, its K<=15." I know precision of doubles
> > > in c is 16 digits. Could some one please help me with this precision
> > > issue.I used stirling (http://en.wikipedia.org/wiki/
> > > Stirling's_approximation) to calculate the first k digits of N.
> > > Thank you.
>
> > If I understand you correctly, you're trying to compute the first k
> > digits in the decimal expansion of N!, with bounds of k <= 15 and 100
> > <= N < 10**8.  Is that right?
>
> > Python's floats simply don't give you enough precision for this:
> > you'd need to find the fractional part of log10(N!) with >= 15 digits
> > of precision.  But the integral part needs ~ log10(log10(N!)) digits,
> > so you end up needing to compute log10(N!) with at least 15 +
> > log10(log10(N!)) ~ 15 + log10(N) + log10(log10(N)) digits (plus a few
> > extra digits for safety);  so that's at least 25 digits needed for N
> > close to 10**8.
>
> > The decimal module would get you the results you need (are you allowed
> > imports?).  Or you could roll your own log implementation based on
> > integer arithmetic.
>
> > --
> > Mark
>
> Yes i am trying to first k digits of N!.I will try your method.
> Thank you

I am out of luck.I put d_2=d_1-int(d_1)+25 instead of d_2=d_1-
int(d_1)+K-1  but i am getting WA.
"The decimal module would get you the results you need (are you
allowed
imports?).  Or you could roll your own log implementation based on
integer arithmetic. "
I don't know if is possible to import this decimal module but kindly
tell me.Also a bit about log implementation
Thank you
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Precision issue in python

[Shashwat Anand]

> I don't know if is possible to import this decimal module but kindly
> tell me.Also a bit about log implementation
>
Why don't you read about decimal module (there is log too in it) and try
writing your approach here in case it does not work? Or you insist someone
to rewrite your code using decimal module ? :-/

~l0nwlf
-- 
http://mail.python.org/mailman/listinfo/python-list

Pure virtual functions in Python?

[lallous]

Hello

How can I do something similar to pure virtual functions in C++ ?

Let us consider this:

class C1:

# Pure virtual
def cb(self, param1, param2):
"""
This is a callback

@param param1: ...
@param param2: ...
"""
raise NotImplementedError, "Implement me"

# Implementation w/o a 'cb', thus 'cb' should not be used
class C2(C1):
def __init__(self):
pass

# Implementation w/ 'cb', thus 'cb' can be used
class C3(C1):
def __init__(self):
pass

def cb(self, param1, param2):
print "i am c3 cb"

# Dispatcher function that calls 'cb' only if 'cb' is implemented in
child classes
def dispatcher(c):
if hasattr(c, 'cb'):
c.cb("Hello", "World")

dispatcher(C2())
dispatcher(C3())

What I want is the ability to have the dispatcher() not to call 'cb'
if it was not implemented in one of the child classes.

Please advise.
-- 
http://mail.python.org/mailman/listinfo/python-list

finding element by tag in xml

[sWrath swrath]

Hi

I am trying to search an element by tag and new in reading a xml file
(in python). I coded this , but it did not work

--
'''This is to detect the first element and print out all that element
by tag'''


from xml.dom.minidom import parse
from xml.etree.ElementTree import*

file1="book.xml"
tmptree=ElementTree()
tmptree.parse(file1)
items=root.getiterator()


dom = parse(file1)


#Find tag names
for node in items :
if node.tag == 'author': #Get tag
print dom.getElementsByTagName ('book') #Error 1



-'
2 Questions

1. Why can't I use dom.getElementsByTagName('book') in #Error 1? How
do i print the elements ?
  Error- AttributeError: ElementTree instance has no attribute
'getElementsByTagName'



Thanks
John
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Upgrading Py2exe App

[T]

On Feb 19, 4:32 pm, Ryan Kelly  wrote:
> On Fri, 2010-02-19 at 11:08 -0800, T wrote:
> > On Feb 18, 7:19 pm, Ryan Kelly  wrote:
> > > On Thu, 2010-02-18 at 07:46 -0800, T wrote:
> > > > I have a Python app which I converted to an EXE (all files separate;
> > > > single EXE didn't work properly) via py2exe - I plan on distributing
> > > > this and would like the ability to remotely upgrade the program (for
> > > > example, via HTTP/HTTPS).   Looks like it's not just the EXE that I
> > > > will need need to replace (DLLs, the library.zip, etc.).  What would
> > > > be the best way to go about doing this?
>
> > > I've been working on an auto-update framework for my own frozen apps,
> > > you might find it useful:
>
> > >  http://pypi.python.org/pypi/esky
>
> > > Docs are a little scarce at the moment, the next release will hopefully
> > > come with a short tutorial (as well as support for cx_freeze and maybe
> > > py2app, depending on how adventurous I'm feeling).
>
> > Thanks Ryan..this looks like it could be what I'm looking for, but I'm
> > still a bit unsure of how exactly how it works.  Do you happen to have
> > an idea approx when the next release w/ tutorial will be out?
>
> If I punt on the py2app support, I should be able to get it done in the
> next 3-4 days.  I'll send a quick email to python-list when it's ready.
>
> Here's a rough roadmap of where the project is heading:
>
>   v0.4.0:  cx_freeze support and a tutorial [the next 3-4 days]
>   v0.4.1:  py2app support [the next 2-3 weeks]
>   v0.5.0:  differential updates using bsdiff [next few months]
>
>   Cheers,
>
>      Ryan
>
> --
> Ryan Kellyhttp://www.rfk.id.au |  This message is digitally signed. Please 
> visit
> r...@rfk.id.au        |  http://www.rfk.id.au/ramblings/gpg/for details
>
>  signature.asc
> < 1KViewDownload

Excellent - I look forward to giving it a try.  Thanks again!
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Interesting talk on Python vs. Ruby and how he would like Python to have just a bit more syntactic flexibility.

[Steve Howell]

On Feb 20, 6:13 am, Michael Sparks  wrote:
> On Feb 18, 4:15 pm, Steve Howell  wrote:
> ...
>
> >     def print_numbers()
> >         [1, 2, 3, 4, 5, 6].map { |n|
> >             [n * n, n * n * n]
> >         }.reject { |square, cube|
> >             square == 25 || cube == 64
> >         }.map { |square, cube|
> >             cube
> >         }.each { |n|
> >             puts n
> >         }
> >     end
>
> This strikes me as a terrible example. For example, this is
> significantly clearer:
>     def print_numbers()
>         for n in [1,2,3,4,5,6]:
>             square, cube = n * n, n * n * n
>             if square != 25 and cube != 64:
>                 print n

This is not an exact translation.  My example prints the cubes.  It is
my fault for using "n" as the parameter in the last block.  I would
rename the parameter to cube.

>
> I /can/ see arguments for ruby style blocks in python, but not for
> this sort of thing, or lisp style quoted expressions[1]. ie I can see
> situations where you have more complex code in real life where they
> will definitely simplify things.
>
> [1] This is perhaps more appropriate because '(a b c) is equivalent
>     to (quote a b c), and quote a b c can be viewed as close to
>     python's expression "lambda: a b c"
>
> However, I can also see that in simple situations - such as the
> example you post - they will have a tendency to make code
> significantly less clear/direct.
>
> I suppose, if I have a choice between something (hard being possible &
> simple code looking simple) and (hard things being simpler & simple
> things looking harder), I'd probably personally choose the former.
> This is not because I don't like hard things being simple, but because
> I think that simple things are more common and making them look harder
> is a mistake.
>


I agree with much of what you are saying.  The example is indeed
terribly contrived.

I'm not sure I agree that there is anything unclear or undirect about
the Ruby, though.  I've been fairly immersed in Ruby code, so maybe
it's been warping my brain, but once you get over the unfamiliarity of
the syntax, you see that there's actually a rhythm to the code.

Setting aside punctuation and parameter lists, the code clearly
expresses the transformations and actions in the natural order that
you'd do them:

   LIST map
  expression
reject
  criteria
map
  expression
each
  statement

In English, for the list elements, map them to tuples of squares and
cubes, reject the oddballs, take the cube, and print it out.

 [1, 2, 3, 4, 5, 6].map { |n|
 [n * n, n * n * n]
 }.reject { |square, cube|
 square == 25 || cube == 64
 }.map { |square, cube|
 cube
 }.each { |cube|
 puts cube
 }

For such a small problem, I agree it's verbose.   But it's also
completely flat--you don't need to use an "if" statement to express
the concept of rejection.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Pure virtual functions in Python?

[Martin v. Loewis]

lallous wrote:
> Hello
> 
> How can I do something similar to pure virtual functions in C++ ?

See, for example

http://code.activestate.com/recipes/266468/

Regards,
Martin
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Pure virtual functions in Python?

[Diez B. Roggisch]

Am 20.02.10 17:12, schrieb lallous:

Hello

How can I do something similar to pure virtual functions in C++ ?

Let us consider this:

class C1:

 # Pure virtual
 def cb(self, param1, param2):
 """
 This is a callback

 @param param1: ...
 @param param2: ...
 """
 raise NotImplementedError, "Implement me"

# Implementation w/o a 'cb', thus 'cb' should not be used
class C2(C1):
 def __init__(self):
 pass

# Implementation w/ 'cb', thus 'cb' can be used
class C3(C1):
 def __init__(self):
 pass

 def cb(self, param1, param2):
 print "i am c3 cb"

# Dispatcher function that calls 'cb' only if 'cb' is implemented in
child classes
def dispatcher(c):
 if hasattr(c, 'cb'):
 c.cb("Hello", "World")

dispatcher(C2())
dispatcher(C3())

What I want is the ability to have the dispatcher() not to call 'cb'
if it was not implemented in one of the child classes.

Please advise.


There is nothing more beyond that what you already did. You can raise a 
NotImplementedError for classes that don't implement the method. That's it.


Diez
--
http://mail.python.org/mailman/listinfo/python-list

Re: Pure virtual functions in Python?

[Martin v. Loewis]

>> class C1:
>>
>>  # Pure virtual
>>  def cb(self, param1, param2):
>>  """
>>  This is a callback
>>
>>  @param param1: ...
>>  @param param2: ...
>>  """
>>  raise NotImplementedError, "Implement me"
>>
>> # Dispatcher function that calls 'cb' only if 'cb' is implemented in
>> child classes
>> def dispatcher(c):
>>  if hasattr(c, 'cb'):
>>  c.cb("Hello", "World")
>>
>> dispatcher(C2())
>> dispatcher(C3())
>>
>> What I want is the ability to have the dispatcher() not to call 'cb'
>> if it was not implemented in one of the child classes.
>>
>> Please advise.
> 
> There is nothing more beyond that what you already did. You can raise a
> NotImplementedError for classes that don't implement the method. That's it.

That's not true. Currently, the hasattr() call would report that cb is
available, when it is actually not implemented. It would be possible to
do something like

  if hasattr(c, 'cb') and not is_pure(c.cb):
  c.cb("Hello", "World")

is_pure could, for example, look at a function attribute of the
callback. You'd write something like

  @pure_virtual
  def cb(self, param1, param2):
  not_implemented

Regards,
Martin
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Pure virtual functions in Python?

[Diez B. Roggisch]

Sorry, I totally mis-read the OP, too tired. You are right of course.

Diez
--
http://mail.python.org/mailman/listinfo/python-list

Re: Pure virtual functions in Python?

[Rami Chowdhury]

On Saturday 20 February 2010 11:46:42 Diez B. Roggisch wrote:
> Am 20.02.10 17:12, schrieb lallous:
> > Hello
> >
> > How can I do something similar to pure virtual functions in C++ ?
> >
> > Let us consider this:
> >
> > class C1:
> >
> >  # Pure virtual
> >  def cb(self, param1, param2):
> >  """
> >  This is a callback
> >
> >  @param param1: ...
> >  @param param2: ...
> >  """
> >  raise NotImplementedError, "Implement me"
> >
> > # Implementation w/o a 'cb', thus 'cb' should not be used
> > class C2(C1):
> >  def __init__(self):
> >  pass
> >
> > # Implementation w/ 'cb', thus 'cb' can be used
> > class C3(C1):
> >  def __init__(self):
> >  pass
> >
> >  def cb(self, param1, param2):
> >  print "i am c3 cb"
> >
> > # Dispatcher function that calls 'cb' only if 'cb' is implemented in
> > child classes
> > def dispatcher(c):
> >  if hasattr(c, 'cb'):
> >  c.cb("Hello", "World")
> >
> > dispatcher(C2())
> > dispatcher(C3())
> >
> > What I want is the ability to have the dispatcher() not to call 'cb'
> > if it was not implemented in one of the child classes.
> >
> > Please advise.
> 
> There is nothing more beyond that what you already did. You can raise a
> NotImplementedError for classes that don't implement the method. That's it.
> 
> Diez
> 

Perhaps you could use an easier-to-ask-forgiveness-than-permission idiom?

def dispatcher(c):
try:
c.cb("Hello", "World")
except NotImplementedError:
pass




Rami Chowdhury
"Passion is inversely proportional to the amount of real information 
available." -- Benford's Law of Controversy
408-597-7068 (US) / 07875-841-046 (UK) / 01819-245544 (BD)
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Precision issue in python

[Mark Dickinson]

On Feb 20, 3:37 pm, mukesh tiwari 
wrote:
> I don't know if is possible to import this decimal module but kindly
> tell me.Also a bit about log implementation

The decimal module is part of the standard library;  I don't know what
the rules are for SPOJ, but you're already importing the math module,
so at least *some* imports are obviously permitted.  Here's a quick
demonstration of how you might use the decimal module:  you can
probably find ways to tweak it for speed and accuracy.

from decimal import Decimal as D
from decimal import getcontext

getcontext().prec = 100   # working precision = 100 digits

pi =
D('3.14159265358979323846264338327950288419716939937510582097494459'
 
'2307816406286208998628034825342117067982148086513282306647093844')
half = D('0.5')
log = D.ln

def logfac(x):
"""Approximation to log(x!), using first few terms of Stirling's
series."""

x = D(x)
return log(2*pi)/2 + (x + half)*log(x) - x + \
1/(12*x) - 1/(360*x**3) + 1/(1260*x**5)

def fac_first_digits(n, k):
"""Get first k decimal digits of n!."""

log10_nfac = logfac(n)/log(D(10))
frac = log10_nfac - int(log10_nfac)
return int(10**(frac - 1 + k))


With the above code I get, for example (with Python 2.6):

>>> fac_first_digits(12345, 15)
344364246918678
>>> from math import factorial
>>> int(str(factorial(12345))[:15])
344364246918678

For small n, you'll need more terms of Stirling's series than I've
used above.  And there's always a small chance that the intermediate
errors involved in the computation might change those first 15
digits;  with a working precision of 100 and lots of terms of
Stirling's series it's a *very* small chance, but it's there.  If you
want something 100% watertight, you'd need to  compute strict upper
and lower bounds for the true result, and increase precision until
those bounds match sufficiently far that you can be sure of the first
k digits being correct.

--
Mark
-- 
http://mail.python.org/mailman/listinfo/python-list

Building a dict from a tuple of tuples

[vsoler]

Hello everyone!

I have a tuple of tuples, coming from an Excel range, such as this:

((None, u'x', u'y'),
(u'a', 1.0, 7.0),
(u'b', None, 8.0))

I need to build a dictionary that has, as key, the row and column
header.

For example:
d={ (u'a',u'x'):1.0, (u'a',u'y'): 7.0, (u'b',u'y'):8.0 }

As you can see, if the value in the matrix is None, no key has to be
added to the dictionary.

Of course, my tuple of tuples is a lot bigger.

How can I possibly do this?

Thank you
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Can't Access ANY url from python (errno 61)

[Lie Ryan]

On 02/20/10 19:36, MattB wrote:
> On Feb 20, 2:02 am, Lie Ryan  wrote:
>> On 02/20/10 13:32, MattB wrote:
>>
>>
>>
>>> I'm using the network in my own apartment. Not the campus's.
>>> Moreover, my mac's MAC address is different from the MAC address shown
>>> by my router, but as I said I'm also blocked when using my friend's
>>> wireless router at his apartment.
>>
>>> So it must be my mac's MAC, and not the router's MAC, that's being
>>> blocked, right?
>>
>>> But ALSO -- is it my ISP that's blocking the mac's MAC (and not the
>>> school), since I can't raise ANY url's from python when I'm on
>>> wireless?
>>
>> MAC or IP blocking can't be the reason, as the OP stated, he can use
>> Firefox just fine.
>>
>> Can you access, say,http://www.google.comfrom urllib or mechanize?
>>
>> If you can't access *any website* using urllib/mechanize but you can
>> with a browser and you're on a unix-based machine, you probably have the
>> same problem as I used to have. Check whether you used the same hostname
>> in /etc/conf.d/hostname and /etc/hosts (or wherever your distro saves
>> its hostname configurations, I use Gentoo); after editing those files
>> reboot (there are ways to avoid reboot, but rebooting guarantees the
>> conf file is reread).
>>
>> Check the hostname by running this python script:
>>
>> import socket
>> hn = socket.gethostname()
>> print hn
>> print socket.gethostbyname(hn) # should be 127.0.0.1
> 
> Lie,
> 
> Wow. Strangely, that script returned 192.168.1.106. However, in Snow
> Leopard's airport settings, if I click on 'advanced' and then
> 'proxies', the default proxy for 'http' is 127.0.0.1: (and in
> these settings, the 'use proxy for http' is checked).
> 
> I just tried checking the unix files you mentioned. In etc/hosts, the
> following info is displayed:
> 
> ##
> # Host Database
> #
> # localhost is used to configure the loopback interface
> # when the system is booting.  Do not change this entry.
> ##
> 
> 
> 127.0.0.1 localhost
> 255.255.255.255   broadcasthost
> ::1 localhost
> fe80::1%lo0   localhost
> 
> 
> Not sure if these are what I'm looking for -- I'm new to unix so I may
> need a bit more hand-holding here.

Try the alternatives in this site: http://movealong.org/hostname.html


> Also found a file called ntp-restrict.conf, containing:
>
> # Access restrictions documented in ntp.conf(5) and
> # http://support.ntp.org/bin/view/Support/AccessRestrictions
> # Limit network machines to time queries only
>
> restrict default kod nomodify notrap nopeer noquery
> restrict -6 default kod nomodify notrap nopeer noquery
>
> # localhost is unrestricted
> restrict 127.0.0.1
> restrict -6 ::1
>
> includefile /private/etc/ntp.conf

Not sure if ntp-restrict.conf is a red herring here since its name
implies it's something for NTP (Network Time Protocol, used for time
synchronization)
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: the mystery of dirname()

[MRAB]

Shashwat Anand wrote:
basically I infer that : dirname = path - basename, like for path =  
'//x', basename = x, hence dirname = '//'



[snip]
Basically, os.path.dirname() should return the directory name, which
means dropping everything after the last slash, and also the last slash.
However, there's the special case where the name is in the root
directory, and you want to retain the slash(es).

When building a path (os.path.join()) you want to join the parts with a
single slash between them, but there's the special case when a part is
the root directory, and you don't want to add a slash between them, eg
os.part.join('/x', 'y') should return '/x/y', but os.part.join('/', 'x')
should return '/x'.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Precision issue in python

[Mark Dickinson]

On Sat, Feb 20, 2010 at 2:42 PM, Shashwat Anand
 wrote:
> A quick solution I came out with, no stirling numbers and had tried to avoid
> large integer multiplication as much as possible.
>
> import math
>
> for i in range(int(raw_input())):
>     n, k, l = [int(i) for i in raw_input().split()]
>     e = sum(math.log10(i) for i in range(1, n+1))
> frac_e = e - math.floor(e)

This isn't going to give you enough accuracy when n gets large (and
the original problem seems to allow n to be as large as 10**8), for
the reasons I described earlier---that is, Python floats only give you
around 16 decimal digits of precision;  your 'e' value will already
have used up some of those 16 digits before the point, leaving you
fewer than 16 digits of precision after the point, so the absolute
error in frac_e will likely be larger than 1e-15.  That's not good
enough for getting the first 15 digits of 10**frac_e accurately.  For
large n, you'll also be accumulating significant error in the sum.

> a = str(10**frac_e * 10**(k - 1)).split('.')[0]

The str(...).split('.') here doesn't do a good job of extracting the
integer part when its argument is >= 1e12, since Python produces a
result in scientific notation.  I think you're going to get strange
results when k >= 13.

--
Mark
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Building a dict from a tuple of tuples

[MRAB]

vsoler wrote:

Hello everyone!

I have a tuple of tuples, coming from an Excel range, such as this:

((None, u'x', u'y'),
(u'a', 1.0, 7.0),
(u'b', None, 8.0))

I need to build a dictionary that has, as key, the row and column
header.

For example:
d={ (u'a',u'x'):1.0, (u'a',u'y'): 7.0, (u'b',u'y'):8.0 }

As you can see, if the value in the matrix is None, no key has to be
added to the dictionary.

Of course, my tuple of tuples is a lot bigger.

How can I possibly do this?

Thank you


Does this help?

matrix = ((None, u'x', u'y'),
(u'a', 1.0, 7.0),
(u'b', None, 8.0))

for row in matrix[1 : ]:
for col, val in zip(matrix[0][1 : ], row[1 : ]):
print row[0], col, val
--
http://mail.python.org/mailman/listinfo/python-list

Re: Can't Access ANY url from python (errno 61)

[Martin P. Hellwig]

On 02/20/10 00:20, MattB wrote:



Also, based on Martin's comment, I just wanted to make you all aware
that I intend no misuse, but rather am just trying to learn, as I'm a
programming noob.


It wasn't my intention to imply that, rather the opposite, that if some 
BOFH would see your action as misuse (some admins are pretty trigger 
happy), you would expect them to contact you directly (I would).

Even though you are on a wireless there are ways to track you down.
For example I would search my log to see if I can make an association 
between a login to one of my servers and your mac address.


BTW, I always followed the philosophy that learning is much more fun if 
you can brake/blow something up. Thus on my networks all students had a 
lot of freedom to do whatever they think was appropriate but they where 
aware that every action on my network was monitored and logged.


--
mph
--
http://mail.python.org/mailman/listinfo/python-list

Re: the mystery of dirname()

[Shashwat Anand]

got it. thanks. :)


On Sat, Feb 20, 2010 at 11:19 PM, MRAB  wrote:

> Shashwat Anand wrote:
>
>> basically I infer that : dirname = path - basename, like for path =
>>  '//x', basename = x, hence dirname = '//'
>>
>>  [snip]
> Basically, os.path.dirname() should return the directory name, which
> means dropping everything after the last slash, and also the last slash.
> However, there's the special case where the name is in the root
> directory, and you want to retain the slash(es).
>
> When building a path (os.path.join()) you want to join the parts with a
> single slash between them, but there's the special case when a part is
> the root directory, and you don't want to add a slash between them, eg
> os.part.join('/x', 'y') should return '/x/y', but os.part.join('/', 'x')
> should return '/x'.
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Pure virtual functions in Python?

[Peter Otten]

lallous wrote:

> How can I do something similar to pure virtual functions in C++ ?

http://docs.python.org/library/abc.html#abc.abstractmethod

Peter
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Precision issue in python

[Shashwat Anand]

@Mark,

The str(...).split('.') here doesn't do a good job of extracting the
> integer part when its argument is >= 1e12, since Python produces a
> result in scientific notation.  I think you're going to get strange
> results when k >= 13.
>

Yeah, you were correct. I tested it for k >= 13, and there were issues i.e.
only one character is printed.
Shashwat-Anands-MacBook-Pro:Desktop l0nwlf$ python CALCULAT.py
3
100 12 12
826393051664 
100 13 13
8 0
100 15 4
9 

The logic I tried was : for
alpha = n!
log(alpha) = log(n!)
= log(n) + log(n-1) + .. log(2) + log(1)
= e [where e = log(n) +  + log(1)]

frac_e =  {e} (fractional part of e, like {1.23} = .23)
now last k digits of alpha = 10**frac_e * 10**(k - 1)

As I can see, the method is mathematically sound, and there is precision
issues. Time for code modification.

~l0nwlf
-- 
http://mail.python.org/mailman/listinfo/python-list

Compiling and running 32-bit Python on 64-bit server?

[Mikko Ohtamaa]

Hi,

Some server-side Python applications are limited by memory usage
(hint: Zope), because Python effective uses processes and not threads
for multiprocessing. This is especially true for 64-bit platforms,
since Python programs are all about references and objects and 64-bit
effectively doubles reference size.

Some benchmarks 32-bit vs. 64-bit were discussed here:

http://jstahl.org/archives/2010/01/2...ks-python-2-6/

How one could create 32-bit Python run-time enviroment, preferable
virtualenv, on 64-bit Linux (VPS), reducing memory usage? This
environment could actually beat 64-bit in performance, due to better
memory cache use.

I assume this involves having lib32 libs and compiling Python with
some magical switches.

Cheers,
Mikko
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Building a dict from a tuple of tuples

[vsoler]

On Feb 20, 7:00 pm, MRAB  wrote:
> vsoler wrote:
> > Hello everyone!
>
> > I have a tuple of tuples, coming from an Excel range, such as this:
>
> > ((None, u'x', u'y'),
> > (u'a', 1.0, 7.0),
> > (u'b', None, 8.0))
>
> > I need to build a dictionary that has, as key, the row and column
> > header.
>
> > For example:
> > d={ (u'a',u'x'):1.0, (u'a',u'y'): 7.0, (u'b',u'y'):8.0 }
>
> > As you can see, if the value in the matrix is None, no key has to be
> > added to the dictionary.
>
> > Of course, my tuple of tuples is a lot bigger.
>
> > How can I possibly do this?
>
> > Thank you
>
> Does this help?
>
> matrix = ((None, u'x', u'y'),
> (u'a', 1.0, 7.0),
> (u'b', None, 8.0))
>
> for row in matrix[1 : ]:
>      for col, val in zip(matrix[0][1 : ], row[1 : ]):
>          print row[0], col, val

and the dictionary?

it is the ultimate goal of what I am intending...

Thank you
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: os.pipe() + os.fork()

[Gary Herron]

Sebastian Noack wrote:

I have figured out that, you have to close the writing end in the child
process, which is reading from the pipe. Otherwise the underlying pipe
is not going to be closed when the parent process is closing its
writing end. This has nothing to do with Python itself. I have tried
plain C and there it is the same behaviour.

Regards
Sebastian Noack
  



Correct.  The fork creates two processes with references to the read and 
write ends of the pipe.  Both parent and child processes should close 
the ends they are not using.


Here's a thought:  Consider the subprocess module.   It can do the fork 
and any necessary pipes and can do so in an OS independent way.   It 
might make you life much easier.


Gary Herron

--
http://mail.python.org/mailman/listinfo/python-list

Re: Compiling and running 32-bit Python on 64-bit server?

[Martin v. Loewis]

> How one could create 32-bit Python run-time enviroment, preferable
> virtualenv, on 64-bit Linux (VPS), reducing memory usage?

I'd install a 32-bit Linux on the hardware, and install a bigmem kernel
if it has more than 3GB of main memory.

> I assume this involves having lib32 libs and compiling Python with
> some magical switches.

The precise set of packages that you will need depends on the specific
Linux distribution. Check whether "gcc -m32" can build a hello-world
program. Then, set CC to "gcc -m32", and follow the build instructions
in Python's README file.

Regards,
Martin
-- 
http://mail.python.org/mailman/listinfo/python-list

if not global -- then what?

[egasimus]

Hi, newbie here. I've read on using the 'global' keyword being
discouraged; then what is the preferred way to have something, for
example a class containing program settings, accessible from
everywhere, in a program spanning multiple files?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Building a dict from a tuple of tuples

[MRAB]

vsoler wrote:

On Feb 20, 7:00 pm, MRAB  wrote:

vsoler wrote:

Hello everyone!
I have a tuple of tuples, coming from an Excel range, such as this:
((None, u'x', u'y'),
(u'a', 1.0, 7.0),
(u'b', None, 8.0))
I need to build a dictionary that has, as key, the row and column
header.
For example:
d={ (u'a',u'x'):1.0, (u'a',u'y'): 7.0, (u'b',u'y'):8.0 }
As you can see, if the value in the matrix is None, no key has to be
added to the dictionary.
Of course, my tuple of tuples is a lot bigger.
How can I possibly do this?
Thank you

Does this help?

matrix = ((None, u'x', u'y'),
(u'a', 1.0, 7.0),
(u'b', None, 8.0))

for row in matrix[1 : ]:
 for col, val in zip(matrix[0][1 : ], row[1 : ]):
 print row[0], col, val


and the dictionary?

it is the ultimate goal of what I am intending...

Thank you


The difficult bit is working out how to produce the keys and values for
the dict from the tuple of tuples, and I've shown you that. The rest is
straightforward.
--
http://mail.python.org/mailman/listinfo/python-list

Re: if not global -- then what?

[MRAB]

egasimus wrote:

Hi, newbie here. I've read on using the 'global' keyword being
discouraged; then what is the preferred way to have something, for
example a class containing program settings, accessible from
everywhere, in a program spanning multiple files?


Python's 'global' keyword is global in a file (module), not global
across multiple files, so it would have helped you anyway.

If you want program settings accessible across multiple modules then put
the settings in a module and import that module in any other module that
needs to access them.
--
http://mail.python.org/mailman/listinfo/python-list

Re: if not global -- then what?

[Gary Herron]

egasimus wrote:

Hi, newbie here. I've read on using the 'global' keyword being
discouraged; then what is the preferred way to have something, for
example a class containing program settings, accessible from
everywhere, in a program spanning multiple files?
  


Define your "global" in a module (a .py file) and import that wherever 
you need it:


mod.py:
gvar = 123   # A Global (like) value


main.py:
import mod
... mod.gvar ...# To access the value
... mod.gvar = 456 ...  #To change the value


Using the "global" keyword has nothing to do with this.  And in fact 
Python does not even have global variables in the sense you may be thinking.


If you wish to provide a function in mode.py that sets the value then 
this would fail


mod.py:
gvar=123
def SetGvar(v):
 gvar = v

because the gvar inside the function is local to that function.   This 
is the proper place to use the "global" keyword. 


mode.py:
gvar = 123
def SetGvar(v):
   global gvar   # Makes gvar refer to the (global) module level gvar
   gvar = v

Gary Herron

--
http://mail.python.org/mailman/listinfo/python-list

Re: if not global -- then what?

[Krister Svanlund]

On Sat, Feb 20, 2010 at 8:25 PM, egasimus  wrote:
> Hi, newbie here. I've read on using the 'global' keyword being
> discouraged; then what is the preferred way to have something, for
> example a class containing program settings, accessible from
> everywhere, in a program spanning multiple files?
> --
> http://mail.python.org/mailman/listinfo/python-list
>

There is probably a smarter way but I would recommend passing a
settings object around.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Pure virtual functions in Python?

[I V]

On Sat, 20 Feb 2010 08:12:01 -0800, lallous wrote:
> How can I do something similar to pure virtual functions in C++ ?

>From what you want, it seems like you want cb() to not be called if it 
isn't implemented in the derived class; this isn't really what pure 
virtual functions in C++ do - pure virtual functions enforce, at compile 
time, that the derived class implements the method.

If you have a situation when you want to either call a derived class's 
version of cb(), or do nothing, can you not just have an implementation 
of cb() in the base class that does nothing, i.e.

class C1(object):
def cb(self, param1, param2):
pass
-- 
http://mail.python.org/mailman/listinfo/python-list

How would I do a continuous write over a pipe in the following code...

[chad]

Given the following

#!/usr/bin/python

import subprocess as s

broadcast = s.Popen("echo test | wall", shell=True,stdout=s.PIPE)

out = broadcast.stdout
while 1:
out
broadcast.wait()

broadcast.stdout.close()


The code only executes once. What I want to do is be able to
continuously write over the pipe once it is open. I could put
s.Popen() inside the while loop, but that seems a bit too messy. So is
there some way to just open the pipe once, and once it is open, just
continuously write over it vs just opening and closing the pipe every
time?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Building a dict from a tuple of tuples

[vsoler]

On Feb 20, 8:54 pm, MRAB  wrote:
> vsoler wrote:
> > On Feb 20, 7:00 pm, MRAB  wrote:
> >> vsoler wrote:
> >>> Hello everyone!
> >>> I have a tuple of tuples, coming from an Excel range, such as this:
> >>> ((None, u'x', u'y'),
> >>> (u'a', 1.0, 7.0),
> >>> (u'b', None, 8.0))
> >>> I need to build a dictionary that has, as key, the row and column
> >>> header.
> >>> For example:
> >>> d={ (u'a',u'x'):1.0, (u'a',u'y'): 7.0, (u'b',u'y'):8.0 }
> >>> As you can see, if the value in the matrix is None, no key has to be
> >>> added to the dictionary.
> >>> Of course, my tuple of tuples is a lot bigger.
> >>> How can I possibly do this?
> >>> Thank you
> >> Does this help?
>
> >> matrix = ((None, u'x', u'y'),
> >> (u'a', 1.0, 7.0),
> >> (u'b', None, 8.0))
>
> >> for row in matrix[1 : ]:
> >>      for col, val in zip(matrix[0][1 : ], row[1 : ]):
> >>          print row[0], col, val
>
> > and the dictionary?
>
> > it is the ultimate goal of what I am intending...
>
> > Thank you
>
> The difficult bit is working out how to produce the keys and values for
> the dict from the tuple of tuples, and I've shown you that. The rest is
> straightforward.

I'll try. Thank you very much MRAB
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: finding element by tag in xml

[Jim]

On Feb 20, 11:27 am, sWrath swrath  wrote:
> 2 Questions
>
> 1. Why can't I use dom.getElementsByTagName('book') in #Error 1? How
> do i print the elements ?
>   Error- AttributeError: ElementTree instance has no attribute
> 'getElementsByTagName'

I only see one question here.

I think the error is asserting that your instance of the ET class does
not have such a fcn.  I also do not see it in the ET documentation.
Perhaps you are looking for the XPath support at 
http://effbot.org/zone/element-xpath.htm
?  Or perhaps you want to make your document dom representation via a
different library?

Jim
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: finding element by tag in xml

[Dj Gilcrease]

On Sat, Feb 20, 2010 at 9:27 AM, sWrath swrath  wrote:
> from xml.dom.minidom import parse
> from xml.etree.ElementTree import*
>
> file1="book.xml"
> tmptree=ElementTree()
> tmptree.parse(file1)
> items=root.getiterator()
>
>
> dom = parse(file1)
>
>
> #Find tag names
> for node in items :
>    if node.tag == 'author': #Get tag
>        print dom.getElementsByTagName ('book') #Error 1

You are mixing two different types of xml parsers, either user minidom

for node in dom.getElementsByTagName('author'):
print node.getElementsByTagName('book')

or use etree

for el in items:
if el.tag == 'author':
print el.find('book')

There are a few differences you need to note, the getElementsByTagName
always returns a list even if there is only 1 element and find only
returns the first element found no matter what. If you want to print
all books associated with an author you would need to do something
like

for author in tmptree.getiterator('author'):
for book in author.getiterator('book'):
print book
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Pure virtual functions in Python?

[Arnaud Delobelle]

lallous  writes:

> Hello
>
> How can I do something similar to pure virtual functions in C++ ?
>
> Let us consider this:
>
> class C1:
>
> # Pure virtual
> def cb(self, param1, param2):
> """
> This is a callback
>
> @param param1: ...
> @param param2: ...
> """
> raise NotImplementedError, "Implement me"

Why define it if it is virtual?

> # Implementation w/o a 'cb', thus 'cb' should not be used
> class C2(C1):
> def __init__(self):
> pass
>
> # Implementation w/ 'cb', thus 'cb' can be used
> class C3(C1):
> def __init__(self):
> pass
>
> def cb(self, param1, param2):
> print "i am c3 cb"
>
> # Dispatcher function that calls 'cb' only if 'cb' is implemented in
> child classes
> def dispatcher(c):
> if hasattr(c, 'cb'):
> c.cb("Hello", "World")
>
> dispatcher(C2())
> dispatcher(C3())
>
> What I want is the ability to have the dispatcher() not to call 'cb'
> if it was not implemented in one of the child classes.

If you don't define cb in the parent class, it'll work.

-- 
Arnaud
-- 
http://mail.python.org/mailman/listinfo/python-list

Reading a large bz2 textfile exits early

[Norman Rieß]

Hello,

i am trying to read a large bz2 compressed textfile using the bz2 module.
The file is 1717362770 lines long and 8GB large.
Using this code

source_file = bz2.BZ2File(file, "r")
for line in source_file:
print line.strip()

print "Exiting"
print "I used file: " + file

the loop exits cleanly after 4311 lines in midline and the prints are
executed.
This happened on two different boxes runnig different brands of linux.
Is there something i miss or should be done differently?

Thank you.

Regards,
Norman

-- 
http://mail.python.org/mailman/listinfo/python-list

/usr/bin/ld: cannot find -lz on Cent OS - Python 2.4

[V8 NUT]

Been trying to fix this issue for over 6 hours now.
It's doin my head in, any one know whats going on here.

==START==
python setup.py build
running build
running build_py
copying MySQLdb/release.py -> build/lib.linux-x86_64-2.4/MySQLdb
running build_ext
building '_mysql' extension
gcc -pthread -shared build/temp.linux-x86_64-2.4/_mysql.o -L/usr/lib64/
mysql -L/usr/lib64 -lmysqlclient_r -lz -lpthread -lcrypt -lnsl -lm -
lpthread -lssl -lcrypto -o build/lib.linux-x86_64-2.4/_mysql.so
/usr/bin/ld: skipping incompatible /usr/lib/libz.so when searching for
-lz
/usr/bin/ld: skipping incompatible /usr/lib/libz.a when searching for -
lz
/usr/bin/ld: cannot find -lz
collect2: ld returned 1 exit status
error: command 'gcc' failed with exit status 1

==END==

Am trying to get MySQL-python-1.2.3c1 installed on a CentOS 5 box
running Python 2.4
I do not want to upgrade my Python version.
I have tried:

1. yum install mysql-python
2. yum install librsync-devel
3. yum install python-devel

With all return no errors and a notification that there's nothing to
install or update.

Please help.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: calculating a string equation

[Astan Chee]

Arnaud Delobelle wrote:

Astan Chee  writes:

  

Hi,
I have some variables in my script that looks like this:
vars = {'var_a':'10','var_b':'4'}
eqat = "(var_a/2.0) <= var_b"
result = "(var_a+var_b)/7"
What I'm trying to do is to plug in var_a and var_b's values from vars
into eqat and see if eqat returns true or false as well as getting the
value of result if these variables were "plugged in". How do I do
this?
I'm also expecting eqat and result to contain various python
mathematical operators like **, and compounded ()'s.
I'm not sure how to convert the equation; if I have to make a bunch of
if-statements or if there is a python function that already does
something like this.



Yes: eval()

  

vars = {'var_a':10 ,'var_b':4}
eqat = "(var_a/2.0) <= var_b"
result = "(var_a+var_b)/7"
eval(eqat, vars)


False

Hi,
I now have a slight problem with this. This doesnt seem to work:
>>> vars = {'var a':10 ,'var b':4}
>>> eqat = "(var a/2.0) <= var b"
>>> eval(eqat, vars)

Traceback (most recent call last):
 File "", line 1, in 
   eval(eqat, vars)
 File "", line 1
   (var a/2.0) <= var b
^
SyntaxError: invalid syntax

So eval can't take spaces? is there a way to go around this?
Thanks again for any suggestions
Cheers
Astan

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: calculating a string equation

[Chris Rebert]

On Sat, Feb 20, 2010 at 3:07 PM, Astan Chee  wrote:
> Arnaud Delobelle wrote:
> Astan Chee  writes:
> Hi,
> I have some variables in my script that looks like this:
> vars = {'var_a':'10','var_b':'4'}
> eqat = "(var_a/2.0) <= var_b"
> result = "(var_a+var_b)/7"
> What I'm trying to do is to plug in var_a and var_b's values from vars
> into eqat and see if eqat returns true or false as well as getting the
> value of result if these variables were "plugged in". How do I do
> this?
> I'm also expecting eqat and result to contain various python
> mathematical operators like **, and compounded ()'s.
> I'm not sure how to convert the equation; if I have to make a bunch of
> if-statements or if there is a python function that already does
> something like this.
>
>
> Yes: eval()
>
>
>
> vars = {'var_a':10 ,'var_b':4}
> eqat = "(var_a/2.0) <= var_b"
> result = "(var_a+var_b)/7"
> eval(eqat, vars)
>
>
> False
>
> Hi,
> I now have a slight problem with this. This doesnt seem to work:
 vars = {'var a':10 ,'var b':4}
 eqat = "(var a/2.0) <= var b"
 eval(eqat, vars)
>
> Traceback (most recent call last):
>   File "", line 1, in 
>     eval(eqat, vars)
>   File "", line 1
>     (var a/2.0) <= var b
>  ^
> SyntaxError: invalid syntax
>
> So eval can't take spaces? is there a way to go around this?

eval() requires the string be valid Python code and conform to
Python's syntax. Obviously your string isn't valid Python.
You can either require beforehand that variables not be prefixed with
"var " as they are currently, or you can get rid of the "var"s
yourself:

vars = {'a':10 ,'b':4}
equat = equat.replace('var ','')
eval(equat, vars)

but this could cause problems if you have a variable whose names ends
with "var"; using a regular expression for the removal would fix that.

Cheers,
Chris
--
http://blog.rebertia.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Is there a way to continue after an exception ?

[Stef Mientki]

hello,

I would like my program to continue on the next line after an uncaught 
exception,

is that possible ?

thanks
Stef Mientki
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Is there a way to continue after an exception ?

[Krister Svanlund]

On Sun, Feb 21, 2010 at 12:52 AM, Stef Mientki  wrote:
> hello,
>
> I would like my program to continue on the next line after an uncaught
> exception,
> is that possible ?
>
> thanks
> Stef Mientki
>

Yes, you catch the exception and do nothing.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: MODULE FOR I, P FRAME

[Tim Roberts]

DANNY  wrote:
>
>If I want to have a MPEG-4/10 coded video and stream it through the
>network and than have the same video on the client side, what should I
>use and of course I don't want to have raw MPEG data, because than I
>couldn't extract the frames to manipulate them.

If you want to manipulate the frames (as bitmaps), then you have little
choice but to decode the MPEG as you receive it, manipulate the bitmaps,
and re-encode it back to MPEG.

That's going to take a fair amount of time...
-- 
Tim Roberts, t...@probo.com
Providenza & Boekelheide, Inc.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: if not global -- then what?

[Steven D'Aprano]

On Sat, 20 Feb 2010 11:25:46 -0800, egasimus wrote:

> Hi, newbie here. I've read on using the 'global' keyword being
> discouraged; then what is the preferred way to have something, for
> example a class containing program settings, accessible from everywhere,
> in a program spanning multiple files?

Such a thing is no different from keeping many global settings. It's not 
the global KEYWORD that is discouraged, but the over-use of global 
variables itself.

See, for example:

http://c2.com/cgi/wiki?GlobalVariablesAreBad
http://weblogs.asp.net/wallen/archive/2003/05/08/6750.aspx


In a nutshell: shared variables introduce tight coupling between parts of 
your code that should be independent. Whether they are truly global, or 
passed around in a Settings object, makes little or no difference. In 
both cases, the risks and problems are identical.

Let me give a fanciful example: suppose your refrigerator, oven and 
shower all shared a single global temperature setting instead of having 
three local temperature settings. This would be a recipe for disaster: 
every operation to any of the three would need to carefully save the 
current temperature setting, and there's the constant risk of scalding 
showers, ruined foods, and undercooked meals.

Having three different global temperature settings helps a bit, but that 
just expands the number of variables that everything has access too. Why 
do the television and the garage door need access to any temperature 
setting, let alone all three?

Python globals aren't as bad as in some other languages, because "global" 
means global to a single module, not global to the entire program, but if 
you create a Settings object and import it from module to module, you 
break the encapsulation, *and* you then have the problem of needing to 
make sure it is initialised properly.

http://archive.eiffel.com/doc/manuals/technology/bmarticles/joop/globals.html

Global constants also aren't as bad, because the coupling is weaker. 
Unfortunately Python doesn't have constants, except by convention, so 
there's always the risk that some badly-behaved function might decide to 
redefine (say) math.pi to 3.15.

Unfortunately, you probably can't get rid of globals altogether, but the 
tricks are (1) use as few of them as possible, and (2) make them local as 
quickly as possible.

For example, you might have something like this:

default_header = "My special header"  # A global

for option, argument in command_line:
if option == "--header":
default_header = argument

class Printer:
def print_page(self, text, header=None):
if header is None:
header = default_header
print header
print text


I would limit the scope of default_header so that it was local to the 
printer class:


for option, argument in command_line:
if option == "--header":
Printer.default_header = argument

class Printer:
default_header = "My special header"

def print_page(self, text, header=None):
if header is None:
header = self.default_header
print header
print text

You might not be able to completely eliminate either globals, or a 
Settings object, but you should be able to make it much, much smaller.

In practice, to make testing easier, I'd split the command line 
processing into two steps: a function that reads and processes the 
command line into a single local settings object, then a separate 
function that applies those settings to the various classes. The function 
that reads the command line doesn't need to know where the settings 
eventually end up, and the function that applies the settings doesn't 
need to know where they come from. All the coupling between settings and 
classes is in one place, the second function, instead of scattered all 
over your code.



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Can't Access ANY url from python (errno 61)

[Steven D'Aprano]

On Sat, 20 Feb 2010 18:28:16 +, Martin P. Hellwig wrote:

> On 02/20/10 00:20, MattB wrote:
> 
>>
>> Also, based on Martin's comment, I just wanted to make you all aware
>> that I intend no misuse, but rather am just trying to learn, as I'm a
>> programming noob.
> 
> It wasn't my intention to imply that, rather the opposite, that if some
> BOFH would see your action as misuse (some admins are pretty trigger
> happy), you would expect them to contact you directly (I would). Even
> though you are on a wireless there are ways to track you down. For
> example I would search my log to see if I can make an association
> between a login to one of my servers and your mac address.

This may be what you would do, but in my experience, it is more likely 
that the admin would be far too busy and/or lazy to try to track you down 
unless (1) they wanted to give you a warning prior to expulsion, or (2) 
they needed to know who you were prior to laying charges.

In my experience they're far more likely to just hit the problem with a 
hammer by blocking you as much as is technically possible.


> BTW, I always followed the philosophy that learning is much more fun if
> you can brake/blow something up. Thus on my networks all students had a
> lot of freedom to do whatever they think was appropriate but they where
> aware that every action on my network was monitored and logged.

Unfortunately this is the exception rather than the rule, I think.


-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: speed question, reading csv using takewhile() and dropwhile()

[Vincent Davis]

Thanks for the help, this is considerably faster and easier to read (see
below). I changed it to avoid the "break" and I think it makes it easy to
understand. I am checking the conditions each time slows it but it is worth
it to me at this time.
Thanks again
Vincent

def read_data_file(filename):
reader = csv.reader(open(filename, "U"),delimiter='\t')

data = []
mask = []
outliers = []
modified = []

data_append = data.append
mask_append = mask.append
outliers_append = outliers.append
modified_append = modified.append

maskcount = 0
outliercount = 0
modifiedcount = 0

for row in reader:
if '[MASKS]' in row:
maskcount += 1
if '[OUTLIERS]' in row:
outliercount += 1
if '[MODIFIED]' in row:
modifiedcount += 1
if not any((maskcount, outliercount, modifiedcount, not row)):
data_append(row)
elif not any((outliercount, modifiedcount, not row)):
mask_append(row)
elif not any((modifiedcount, not row)):
outliers_append(row)
else:
if row: modified_append(row)

data = data[1:]
mask = mask[3:]
outliers = outliers[3:]
modified = modified[3:]
return [data, mask, outliers, modified]

*Vincent Davis
720-301-3003 *
vinc...@vincentdavis.net
 my blog  |
LinkedIn


On Fri, Feb 19, 2010 at 4:36 PM, Jonathan Gardner <
jgard...@jonathangardner.net> wrote:

>
> On Fri, Feb 19, 2010 at 1:58 PM, Vincent Davis 
> wrote:
>
>> In reference to the several comments about "[x for x in read] is basically
>> a copy of the entire list. This isn't necessary." or list(read). I had
>> thought I had a problem with having iterators in the takewhile() statement.
>> I thought I testes and it didn't work. It seems I was wrong. It clearly
>> works. I'll make this change and see if it is any better.
>>
>> I actually don't plan to read them all in at once, only as needed, but I
>> do need the whole file in an array to perform some mathematics on them and
>> compare different files. So my interest was in making it faster to open them
>> as needed. I guess part of it is that they are about 5mb so I guess it might
>> be disk speed in part.nks
>>
>>
>
> Record your numbers in an array and then work your magic on them later.
> Don't store the entire file in memory, though.
>
> --
> Jonathan Gardner
> jgard...@jonathangardner.net
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Is there a way to continue after an exception ?

[Lie Ryan]

> On Sun, Feb 21, 2010 at 12:52 AM, Stef Mientki  wrote:
>> hello,
>>
>> I would like my program to continue on the next line after an uncaught
>> exception,
>> is that possible ?
>>
>> thanks
>> Stef Mientki
>>

That reminds me of VB's "On Error Resume Next"
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: The Disappearing Program?

[rzed]

"W. eWatson"  wrote in
news:hlls5c$89...@news.eternal-september.org: 

> I've successfully compiled several small python programs on Win
> XP into executables using py2exe. A program goes from a name like
> snowball.py to snowball. A dir in the command prompt window finds
> snowball.py but not snowball. If I type in snowball, it executes.
> What's up with that? 

There is a ludicrous setting in Windows explorer that hides the 
extensions for known file types, such as .exe. If you see "snowball" 
in Win Explorer, that's probably the way your system is set. If you 
click on Tools . Folder Options . View you should see a checkbox for 
the "Hide extensions for known file types" option. Uncheck it, and the 
disappearing program will return to view.

This is a separate issue for why snowball.py executes when you enter 
"snowball" at the command line.

-- 
rzed
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Is there a way to continue after an exception ?

[Daniel Fetchinson]

> I would like my program to continue on the next line after an uncaught
> exception,
> is that possible ?

try:
# here is your error
except:
pass

# this will get executed no matter what

See http://docs.python.org/tutorial/errors.html

HTH,
Daniel



-- 
Psss, psss, put it down! - http://www.cafepress.com/putitdown
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: The future of "frozen" types as the number of CPU cores increases

[Tim Roberts]

Chris Rebert  wrote:

>On Thu, Feb 18, 2010 at 11:58 AM, John Nagle  wrote:
>>
>>   Python isn't ready for this.  Not with the GIL.
>
>Is any language, save perhaps Erlang, really ready for it?

F# is.  I only wish the syntax was a little less Perl-like.  Too many
special characters.
-- 
Tim Roberts, t...@probo.com
Providenza & Boekelheide, Inc.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: The Disappearing Program?

[MRAB]

rzed wrote:

"W. eWatson"  wrote in
news:hlls5c$89...@news.eternal-september.org: 


I've successfully compiled several small python programs on Win
XP into executables using py2exe. A program goes from a name like
snowball.py to snowball. A dir in the command prompt window finds
snowball.py but not snowball. If I type in snowball, it executes.
What's up with that? 


There is a ludicrous setting in Windows explorer that hides the 
extensions for known file types, such as .exe. If you see "snowball" 
in Win Explorer, that's probably the way your system is set. If you 
click on Tools . Folder Options . View you should see a checkbox for 
the "Hide extensions for known file types" option. Uncheck it, and the 
disappearing program will return to view.



The problem with that setting is that an .exe file could masquerade as
some other type, eg "foo.txt.exe" would appear as "foo.txt", so you
might have thought that you were safe opening it because it's just text.

This is a separate issue for why snowball.py executes when you enter 
"snowball" at the command line.




--
http://mail.python.org/mailman/listinfo/python-list

Efficient way to break up a list into two pieces

[marwie]

Hello,

I recently read about augmented assignments and that (with l1, l2
being lists)

l1.extend(l2)

is more efficient than

l1 = l1 + l2

because unnecessary copy operations can be avoided. Now my question is
if there's a similar thing for breaking a list into two parts. Let's
say I want to remove from l1 everything from and including position 10
and store it in l2. Then I can write

l2 = l1[10:]
del l1[10:]

But since I'm assigning a slice the elements will be copied.
Basically, I'm looking for something like l1.pop(10,len(l1)) which
returns and removes a whole chunk of data. Is there such a thing (and
if not, why not?)

Cheers,
Martin.
-- 
http://mail.python.org/mailman/listinfo/python-list

Not sure why this is filling my sys memory

[Vincent Davis]

Code is below, The files are about 5mb and 230,000 rows. When I have 43
files of them and when I get to the 35th (reading it in) my system gets so
slow that it is nearly functionless. I am on a mac and activity monitor
shows that python is using 2.99GB of memory (of 4GB). (python 2.6 64bit).
The getsizeof() returns 6424 bytes for the alldata . So I am not sure what
is happening.
Any ideas
Thanks

*Vincent Davis
720-301-3003 *
vinc...@vincentdavis.net
 my blog  |
LinkedIn
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Efficient way to break up a list into two pieces

[Jonathan Gardner]

On Sat, Feb 20, 2010 at 4:55 PM, marwie  wrote:
> Hello,
>
> I recently read about augmented assignments and that (with l1, l2
> being lists)
>
>    l1.extend(l2)
>
> is more efficient than
>
>    l1 = l1 + l2
>
> because unnecessary copy operations can be avoided. Now my question is
> if there's a similar thing for breaking a list into two parts. Let's
> say I want to remove from l1 everything from and including position 10
> and store it in l2. Then I can write
>
>    l2 = l1[10:]
>    del l1[10:]
>
> But since I'm assigning a slice the elements will be copied.
> Basically, I'm looking for something like l1.pop(10,len(l1)) which
> returns and removes a whole chunk of data. Is there such a thing (and
> if not, why not?)
>

The idiom is:

>>> l1, l2 = l1[:10], l1[10:]

Don't know if it's optimized or not. If it's not, it could probably
be. This is a really common idiom.

-- 
Jonathan Gardner
jgard...@jonathangardner.net
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Not sure why this is filling my sys memory

[Vincent Davis]

> Code is below, The files are about 5mb and 230,000 rows. When I have 43
> files of them and when I get to the 35th (reading it in) my system gets so
> slow that it is nearly functionless. I am on a mac and activity monitor
> shows that python is using 2.99GB of memory (of 4GB). (python 2.6 64bit).
> The getsizeof() returns 6424 bytes for the alldata . So I am not sure what
> is happening.
> Any ideas
> Thanks
>
> import csv, os, glob
import sys

def read_data_file(filename):
reader = csv.reader(open(filename, "U"),delimiter='\t')

data = []
mask = []
outliers = []
modified = []

data_append = data.append
mask_append = mask.append
outliers_append = outliers.append
modified_append = modified.append

maskcount = 0
outliercount = 0
modifiedcount = 0

for row in reader:
if '[MASKS]' in row:
maskcount += 1
if '[OUTLIERS]' in row:
outliercount += 1
if '[MODIFIED]' in row:
modifiedcount += 1
if not any((maskcount, outliercount, modifiedcount, not row)):
data_append(row)
elif not any((outliercount, modifiedcount, not row)):
mask_append(row)
elif not any((modifiedcount, not row)):
outliers_append(row)
else:
if row: modified_append(row)

data = data[1:]
mask = mask[3:]
outliers = outliers[3:]
modified = modified[3:]

return [data, mask, outliers, modified]


def ImportDataFrom(folder):
print 'Importing files from: ', folder
alldata = dict()
infolder = glob.glob( os.path.join(folder, '*.txt') )
numfiles = len(infolder)
print 'Importing ' + str(numfiles) + ' files from: ', folder
for infile in infolder:
print "Loading into memory: " + os.path.split(infile)[1]
fname = os.path.split(infile)[1]

filedata = dict(zip([fname + '_data', fname + '_mask', fname +
'_outliers', fname+'_modified'], read_data_file(infile)))
print fname + ' has ' + str(len(filedata[fname + '_data'])) + ' rows
of data'
print fname + ' has ' + str(len(filedata[fname + '_mask'])) + ' rows
of masked data'
print fname + ' has ' + str(len(filedata[fname + '_outliers'])) + '
rows of outliers'
print fname + ' has ' + str(len(filedata[fname +'_modified'])) + '
modified rows of data'
print str(sys.getsizeof(filedata)) +'bytes'' of memory used for '+
fname
print ' '
alldata.update(filedata)
print str(len(alldata)/4) + ' files of ' + str(numfiles) + ' using '
+ str(sys.getsizeof(alldata)) + ' bytes of memory'
return alldata


ImportDataFrom("/Users/vmd/Dropbox/dna/data/rawdata")
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Efficient way to break up a list into two pieces

[Shashwat Anand]

You can always implement your own data-structures or simply a function if
you need so. A language consist of building blocks and not buildings.

On Sun, Feb 21, 2010 at 6:36 AM, Jonathan Gardner <
jgard...@jonathangardner.net> wrote:

> On Sat, Feb 20, 2010 at 4:55 PM, marwie  wrote:
> > Hello,
> >
> > I recently read about augmented assignments and that (with l1, l2
> > being lists)
> >
> >l1.extend(l2)
> >
> > is more efficient than
> >
> >l1 = l1 + l2
> >
> > because unnecessary copy operations can be avoided. Now my question is
> > if there's a similar thing for breaking a list into two parts. Let's
> > say I want to remove from l1 everything from and including position 10
> > and store it in l2. Then I can write
> >
> >l2 = l1[10:]
> >del l1[10:]
> >
> > But since I'm assigning a slice the elements will be copied.
> > Basically, I'm looking for something like l1.pop(10,len(l1)) which
> > returns and removes a whole chunk of data. Is there such a thing (and
> > if not, why not?)
> >
>
> The idiom is:
>
> >>> l1, l2 = l1[:10], l1[10:]
>
> Don't know if it's optimized or not. If it's not, it could probably
> be. This is a really common idiom.
>
> --
> Jonathan Gardner
> jgard...@jonathangardner.net
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: speed question, reading csv using takewhile() and dropwhile()

[Jonathan Gardner]

On Sat, Feb 20, 2010 at 4:21 PM, Vincent Davis wrote:

> Thanks for the help, this is considerably faster and easier to read (see
> below). I changed it to avoid the "break" and I think it makes it easy to
> understand. I am checking the conditions each time slows it but it is worth
> it to me at this time.
>
>
It seems you are beginning to understand that programmer time is more
valuable than machine time. Congratulations.



> def read_data_file(filename):
> reader = csv.reader(open(filename, "U"),delimiter='\t')
>
> data = []
> mask = []
> outliers = []
> modified = []
>
> data_append = data.append
> mask_append = mask.append
> outliers_append = outliers.append
> modified_append = modified.append
>
>

I know some people do this to speed things up. Really, I don't think it's
necessary or wise to do so.


> maskcount = 0
> outliercount = 0
> modifiedcount = 0
>
> for row in reader:
> if '[MASKS]' in row:
> maskcount += 1
> if '[OUTLIERS]' in row:
> outliercount += 1
> if '[MODIFIED]' in row:
> modifiedcount += 1
> if not any((maskcount, outliercount, modifiedcount, not row)):
> data_append(row)
> elif not any((outliercount, modifiedcount, not row)):
> mask_append(row)
> elif not any((modifiedcount, not row)):
> outliers_append(row)
> else:
> if row: modified_append(row)
>
>

Just playing with the logic here:

1. Notice that if "not row" is True, nothing happens? Pull it out
explicitly.

2. Notice how it switches from mode to mode? Program it more explicitly.

Here's my suggestion:

def parse_masks(reader):
for row in reader:
if not row: continue
elif '[OUTLIERS]' in row: parse_outliers(reader)
elif '[MODIFIED]' in row: parse_modified(reader)
   masks.append(row)

def parse_outliers(reader):
for row in reader:
if not row: continue
elif '[MODIFIED]' in row: parse_modified(reader)
   outliers.append(row)

def parse_modified(reader):
for row in reader:
if not row: continue
   modified.append(row)

for row in reader:
if not row: continue
elif '[MASKS]' in row: parse_masks(reader)
elif '[OUTLIERS]' in row: parse_outliers(reader)
elif '[MODIFIED]' in row: parse_modified(reader)
else: data.append(row)

Since there is global state involved, you may want to save yourself some
trouble in the future and put the above in a class where separate parsers
can be kept separate.

It looks like your program is turning into a regular old parser. Any format
that is a little more than trivial to parse will need a real parser like the
above.

-- 
Jonathan Gardner
jgard...@jonathangardner.net
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: speed question, reading csv using takewhile() and dropwhile()

[Vincent Davis]

Thanks again for the comment, not sure I will implement all of it but I will
separate the "if not row" The files have some extraneous blank rows in the
middle that I need to be sure not to import as blank rows.
I am actually having trouble with this filling my sys memory, I posted a
separate question "Why is this filling my sys memory" or something like that
is the subject.
I might be that my 1yr old son has been trying to help for the last hour. It
is very distracting.

  *Vincent Davis
720-301-3003 *
vinc...@vincentdavis.net
 my blog  |
LinkedIn


On Sat, Feb 20, 2010 at 6:18 PM, Jonathan Gardner <
jgard...@jonathangardner.net> wrote:

> On Sat, Feb 20, 2010 at 4:21 PM, Vincent Davis 
> wrote:
>
>> Thanks for the help, this is considerably faster and easier to read (see
>> below). I changed it to avoid the "break" and I think it makes it easy to
>> understand. I am checking the conditions each time slows it but it is worth
>> it to me at this time.
>>
>>
> It seems you are beginning to understand that programmer time is more
> valuable than machine time. Congratulations.
>
>
>
>> def read_data_file(filename):
>> reader = csv.reader(open(filename, "U"),delimiter='\t')
>>
>> data = []
>> mask = []
>> outliers = []
>> modified = []
>>
>> data_append = data.append
>> mask_append = mask.append
>> outliers_append = outliers.append
>> modified_append = modified.append
>>
>>
>
> I know some people do this to speed things up. Really, I don't think it's
> necessary or wise to do so.
>
>
>> maskcount = 0
>> outliercount = 0
>> modifiedcount = 0
>>
>> for row in reader:
>> if '[MASKS]' in row:
>> maskcount += 1
>> if '[OUTLIERS]' in row:
>> outliercount += 1
>> if '[MODIFIED]' in row:
>> modifiedcount += 1
>>  if not any((maskcount, outliercount, modifiedcount, not row)):
>> data_append(row)
>> elif not any((outliercount, modifiedcount, not row)):
>> mask_append(row)
>> elif not any((modifiedcount, not row)):
>> outliers_append(row)
>> else:
>> if row: modified_append(row)
>>
>>
>
> Just playing with the logic here:
>
> 1. Notice that if "not row" is True, nothing happens? Pull it out
> explicitly.
>
> 2. Notice how it switches from mode to mode? Program it more explicitly.
>
> Here's my suggestion:
>
> def parse_masks(reader):
> for row in reader:
> if not row: continue
> elif '[OUTLIERS]' in row: parse_outliers(reader)
> elif '[MODIFIED]' in row: parse_modified(reader)
>masks.append(row)
>
> def parse_outliers(reader):
> for row in reader:
> if not row: continue
> elif '[MODIFIED]' in row: parse_modified(reader)
>outliers.append(row)
>
> def parse_modified(reader):
> for row in reader:
> if not row: continue
>modified.append(row)
>
> for row in reader:
> if not row: continue
> elif '[MASKS]' in row: parse_masks(reader)
> elif '[OUTLIERS]' in row: parse_outliers(reader)
> elif '[MODIFIED]' in row: parse_modified(reader)
> else: data.append(row)
>
> Since there is global state involved, you may want to save yourself some
> trouble in the future and put the above in a class where separate parsers
> can be kept separate.
>
> It looks like your program is turning into a regular old parser. Any format
> that is a little more than trivial to parse will need a real parser like the
> above.
>
> --
> Jonathan Gardner
> jgard...@jonathangardner.net
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: if not global -- then what?

[Jonathan Gardner]

On Sat, Feb 20, 2010 at 11:25 AM, egasimus  wrote:
> Hi, newbie here. I've read on using the 'global' keyword being
> discouraged; then what is the preferred way to have something, for
> example a class containing program settings, accessible from
> everywhere, in a program spanning multiple files?

One of the powerful concepts to come out of Lisp was "dynamic scope".
This is the ideal solution for configuration and logging.

"Lexical scope" is where the value of the variables are found in the
file it was defined in. This is what Python does, and it's what you're
used to in most other languages.

"Dynamic scope" is where the value of the variables are found by
looking up the call stack. This is similar to perl's "local" keyword.
It's also similar to environment variables, if you consider programs
that run programs that run programs as similar to functions. (If
you're familiar with neither, then that's fine.)

This is useful for a program where you know it needs configuration or
logging, but you don't want to have to specify it as part of the
program. Whoever calls the program needs to configure it and setup
their logger before calling it.

Unfortunately, there is no built-in way to deal with dynamic variables
in Python. But that doesn't mean you can't emulate it explicitly and
clearly!

In order to emulate it, you simply pass the configuration and such
down to the functions you call. Each function or class instance needs
to pass it further along. So, rather than:

  def foo():
  bar()

  def bar()
 ...

you need to have:

  def foo(config):
 bar(config)

  def bar(config):
 ...

Obviously, only certain functions care for the config and logging
setup. Others don't. And obviously, you can modify the value on the
way down. Just be sure to make a copy or you'll change the entire
stack's value for that configuration.

There are a couple of systems that do something close to this. I don't
know of any, except SQLAlchemy, that does this exactly right. WSGI is
also a system that does this right, although I don't know that many
people realize it.

Usually, what people end up doing is setting the config values to some
global in some module. Then everyone is expected to look in that
module for the config. This ends up being messy, particularly for
testing where you'd like to mess with the config without messing with
THE config. You can tell it's messy because people end up writing code
like this:

  old_value = get_config(...)
  set_config(... some new value ...)
  ... do your stuff ...
  set_config(...back to old_value...)

This kind of code is hard to get right. (What happens if there is an
exception before you set the old config back? That's right, you need a
try-finally block.)

In terms of "global", you should only really use "global" when you are
need to assign to a lexically scoped variable that is shared among
other functions. For instance:

def foo():
i = 0
def inc(): global i; i+=1
def dec(): global i; i-=1
def get(): return i
return (inc, dec, get)

This really isn't that common, although it is useful. Note that the
above might be better organized into a class instance.

Good luck.

-- 
Jonathan Gardner
jgard...@jonathangardner.net
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Efficient way to break up a list into two pieces

[Steven D'Aprano]

On Sat, 20 Feb 2010 16:55:10 -0800, marwie wrote:

> Hello,
> 
> I recently read about augmented assignments and that (with l1, l2 being
> lists)
> 
> l1.extend(l2)
> 
> is more efficient than
> 
> l1 = l1 + l2
> 
> because unnecessary copy operations can be avoided. Now my question is
> if there's a similar thing for breaking a list into two parts. Let's say
> I want to remove from l1 everything from and including position 10 and
> store it in l2. Then I can write
> 
> l2 = l1[10:]
> del l1[10:]
> 
> But since I'm assigning a slice the elements will be copied. 

Yes, list slicing can't make any sort of assumptions about what you're 
going to do next, so when you take a slice, it has to copy the data.


> Basically,
> I'm looking for something like l1.pop(10,len(l1)) which returns and
> removes a whole chunk of data. Is there such a thing (and if not, why
> not?)

Such a list pop would have to copy the data too. How else could it work?

What exactly is the problem you are trying to solve?

If you are unhappy that copy-and-remove a slice from a list takes two 
lines instead of one ("won't somebody think of the shortage of 
newlines!!!" *wink*), then just write a helper function and call that:

def copy_and_remove(alist, slice):
tmp = alist[slice]
del alist[slice]
return tmp

l2 = copy_and_remove(l1, slice(10, None))


If you are unhappy that copying slices from lists copies data, well 
that's just the way things are. Don't make the mistake of trying to 
optimize your application before you actually know what bits need 
optimizing.

Python lists are arrays of pointers to objects, so copying a slice is 
fast: it doesn't have to copy the objects, just pointers. Deleting from 
the end of the list is also quick, because you don't have to move memory, 
just clear some pointers and change the length field.

Splitting such an array without copying data is, essentially, impossible. 
Python lists aren't linked lists.

But as I said, most of the time copying slices doesn't matter: the time 
taken is unlikely to be a serious factor in practice. If it is (and 
you've profiled your code, right? you're not just guessing what you think 
is fast and slow?) then there may be ways to optimize your code to avoid 
needing to copy slices, but we'd need to know what you're doing with the 
data.


-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: /usr/bin/ld: cannot find -lz on Cent OS - Python 2.4

[Jonathan Gardner]

On Sat, Feb 20, 2010 at 3:03 PM, V8 NUT  wrote:


> /usr/bin/ld: skipping incompatible /usr/lib/libz.so when searching for
> -lz
> /usr/bin/ld: skipping incompatible /usr/lib/libz.a when searching for -
> lz
> /usr/bin/ld: cannot find -lz

This is your problem.

>
> Am trying to get MySQL-python-1.2.3c1 installed on a CentOS 5 box
> running Python 2.4
> I do not want to upgrade my Python version.
> I have tried:
>
> 1. yum install mysql-python
> 2. yum install librsync-devel
> 3. yum install python-devel
>
> With all return no errors and a notification that there's nothing to
> install or update.
>

I think you need libz's -devel module. Not sure what libz is or what
its name is in the yum repository. Might take a search or maybe an rpm
-qif /usr/lib/libz.so

The problem could be that mysql-python expects to see a different
version of libz than what's installed on your box. If so, you'll need
to see if you can find the right version.

The MySQL-python people should have more help for you.


-- 
Jonathan Gardner
jgard...@jonathangardner.net
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: datelib pythonification

[alex goretoy]

hello all,
since I posted this last time, I've added a new function dates_diff and
modified the dates_dict function to set timedelta values returned by
dates_diff in the returned dict

def dates_dict(self,*targs,**dargs):
"""
dates_dict() - takes params same as prefs()
returns dict key/value pair of formatted/calculated dates
key is a date, value is timedelta enddate-startdate

"""
self.prefs(*targs,**dargs)
d={}
try:
if self.format!="":
d[self.startdate.strftime(self.format)]=self.dates_diff()
else:
d[self.startdate]=self.dates_diff()
except ValueError:
d["%s"%self.startdate]=self.dates_diff()
while self.startdate-- 
http://mail.python.org/mailman/listinfo/python-list

Re: speed question, reading csv using takewhile() and dropwhile()

[Jonathan Gardner]

On Sat, Feb 20, 2010 at 5:32 PM, Vincent Davis  wrote:
>
> Thanks again for the comment, not sure I will implement all of it but I will 
> separate the "if not row" The files have some extraneous blank rows in the 
> middle that I need to be sure not to import as blank rows.
> I am actually having trouble with this filling my sys memory, I posted a 
> separate question "Why is this filling my sys memory" or something like that 
> is the subject.
> I might be that my 1yr old son has been trying to help for the last hour. It 
> is very distracting.
>

It may be that the csvfile is reading the entire file in, rather than
line-by-line.

--
Jonathan Gardner
jgard...@jonathangardner.net
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Not sure why this is filling my sys memory

[Jonathan Gardner]

On Sat, Feb 20, 2010 at 5:07 PM, Vincent Davis  wrote:
>> Code is below, The files are about 5mb and 230,000 rows. When I have 43
>> files of them and when I get to the 35th (reading it in) my system gets so
>> slow that it is nearly functionless. I am on a mac and activity monitor
>> shows that python is using 2.99GB of memory (of 4GB). (python 2.6 64bit).
>> The getsizeof() returns 6424 bytes for the alldata . So I am not sure what
>> is happening.

With this kind of data set, you should start looking at BDBs or
PostgreSQL to hold your data. While processing files this large is
possible, it isn't easy. Your time is better spent letting the DB
figure out how to arrange your data for you.

-- 
Jonathan Gardner
jgard...@jonathangardner.net
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Efficient way to break up a list into two pieces

[Steven D'Aprano]

On Sat, 20 Feb 2010 17:06:36 -0800, Jonathan Gardner wrote:

> On Sat, Feb 20, 2010 at 4:55 PM, marwie  wrote:
[...]
>>    l2 = l1[10:]
>>    del l1[10:]
>>
>> But since I'm assigning a slice the elements will be copied. Basically,
>> I'm looking for something like l1.pop(10,len(l1)) which returns and
>> removes a whole chunk of data. Is there such a thing (and if not, why
>> not?)
>>
>>
> The idiom is:
> 
 l1, l2 = l1[:10], l1[10:]

No, that has different behaviour to what the Original Poster wants, AND 
it does two copies instead of one:

(1) copy l1[:10] and l1[10:]
(2) assign the names l1 and l2 to them
(3) if, and only if, there are no other references to the original l1, it 
gets deleted (garbage collected).


What the OP is doing is quite different:

(1) copy l1[:10]
(2) assign the name l2 to it
(3) resize l1 in place to the first 10 items.


What the OP wants is:

(1) assign the name l2 to l1[:10] without copying
(2) resize l1 in place to the first 10 items without affecting l2.


-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

netcdf4-python

[deadpickle]

I'm trying to use the python module "netcdf4-python" to read a netcdf
file. So far I'm trying to do the basics and just open the script:

from netCDF4 import Dataset
rootgrp = Dataset('20060402-201025.netcdf', 'r',
format='NETCDF3_CLASSIC')
print rootgrp.file_format
rootgrp.close()

when I do this I get the exit code error (when I run in Scite):

>pythonw -u "netcdf_test.py"
>Exit code: -1073741819
 or Python stops responding (in windows cmd).

I'm not sure what is wrong so if anyone as any ideas I would gladly
send you the netcdf file to try. Thanks.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Not sure why this is filling my sys memory

[Vincent Davis]

On Sat, Feb 20, 2010 at 6:44 PM, Jonathan Gardner <
> jgard...@jonathangardner.net> wrote:

With this kind of data set, you should start looking at BDBs or

PostgreSQL to hold your data. While processing files this large is

possible, it isn't easy. Your time is better spent letting the DB

figure out how to arrange your data for you.


I really do need all of it in at time, It is dna microarray data. Sure there
are 230,00 rows but only 4 columns of small numbers. Would it help to make
them float() ? I need to at some point. I know in numpy there is a way to
set the type for the whole array "astype()" I think.
What I don't get is that it show the size of the dict with all the data to
have only 6424 bytes. What is using up all the memory?

  *Vincent Davis
720-301-3003 *
vinc...@vincentdavis.net
 my blog  |
LinkedIn


On Sat, Feb 20, 2010 at 6:44 PM, Jonathan Gardner <
jgard...@jonathangardner.net> wrote:

> On Sat, Feb 20, 2010 at 5:07 PM, Vincent Davis 
> wrote:
> >> Code is below, The files are about 5mb and 230,000 rows. When I have 43
> >> files of them and when I get to the 35th (reading it in) my system gets
> so
> >> slow that it is nearly functionless. I am on a mac and activity monitor
> >> shows that python is using 2.99GB of memory (of 4GB). (python 2.6
> 64bit).
> >> The getsizeof() returns 6424 bytes for the alldata . So I am not sure
> what
> >> is happening.
>
> With this kind of data set, you should start looking at BDBs or
> PostgreSQL to hold your data. While processing files this large is
> possible, it isn't easy. Your time is better spent letting the DB
> figure out how to arrange your data for you.
>
> --
> Jonathan Gardner
> jgard...@jonathangardner.net
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: if not global -- then what?

[Steven D'Aprano]

On Sat, 20 Feb 2010 17:34:15 -0800, Jonathan Gardner wrote:

> In terms of "global", you should only really use "global" when you are
> need to assign to a lexically scoped variable that is shared among other
> functions. For instance:
> 
> def foo():
> i = 0
> def inc(): global i; i+=1
> def dec(): global i; i-=1
> def get(): return i
> return (inc, dec, get)

That doesn't do what you think it does:


>>> def foo():
... i = 0
... def inc(): global i; i+=1
... def dec(): global i; i-=1
... def get(): return i
... return (inc, dec, get)
...
>>> inc = foo()[0]
>>> inc()
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 3, in inc
NameError: global name 'i' is not defined


The problem is that i is not global. Inside the inc and dec functions, 
you need to declare i nonlocal, not global, and that only works in Python 
3 or better.



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Efficient way to break up a list into two pieces

[marwie]

On 21 Feb., 02:30, Steven D'Aprano  wrote:
> Python lists are arrays of pointers to objects, so copying a slice is
> fast: it doesn't have to copy the objects, just pointers. Deleting from
> the end of the list is also quick, because you don't have to move memory,
> just clear some pointers and change the length field.
>
> Splitting such an array without copying data is, essentially, impossible.
> Python lists aren't linked lists.

Well, to split a C array I would simply set l2 to point to l1[10] and
then
change the length of l1 (which I store somewhere else). No copying of
elements
needed. I would have assumed that python can do something like this
with its
internal arrays of pointers, too.

Anyway, this was more a question about coding style. I use
l1.extend(l2) or
l1 += l2 rather than l1 = l1 + l2 because it's as readable and
possibly faster.
I was simply wondering if something similar exists for splitting
lists.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: The future of "frozen" types as the number of CPU cores increases

[sjdevn...@yahoo.com]

On Feb 18, 2:58 pm, John Nagle  wrote:
>     Multiple processes are not the answer.  That means loading multiple
> copies of the same code into different areas of memory.  The cache
> miss rate goes up accordingly.

A decent OS will use copy-on-write with forked processes, which should
carry through to the cache for the code.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Is there a way to continue after an exception ?

[Lie Ryan]

On 02/21/10 12:02, Stef Mientki wrote:
> On 21-02-2010 01:21, Lie Ryan wrote:
>>> On Sun, Feb 21, 2010 at 12:52 AM, Stef Mientki
 wrote:
>>>
 hello,

 I would like my program to continue on the next line after an uncaught
 exception,
 is that possible ?

 thanks
 Stef Mientki


>> That reminds me of VB's "On Error Resume Next"
>>
> I think that's what I'm after ...

First, read this:
http://www.developerfusion.com/code/4325/on-error-resume-next-considered-harmful/

> I already redirected sys.excepthook to my own function,
> but now I need a way to get to continue the code on the next line.
> Is that possible ?

No, not in python. You can (ab)use generators' yield to resume
execution, but not in the general case:

def on_error_resume_next(func):
def _func(*args, **kwargs):
gen = func(*args, **kwargs)
resp = next(gen)
while isinstance(resp, Exception):
print 'an error happened, ignoring...'
resp = next(gen)
return resp
return _func

@on_error_resume_next
def add_ten_error_if_zero(args):
if args == 0:
# raise Exception()
yield Exception()
# return args + 10
yield args + 10

print add_ten_error_if_zero(0)
print add_ten_error_if_zero(10)




A slightly better approach is to retry calling the function again, but
as you can see, it's not appropriate for certain cases:

def retry_on_error(func):
def _func(*args, **kwargs):
while True:
try:
ret = func(*args, **kwargs)
except Exception:
print 'An error happened, retrying...'
else:
return ret
return _func

@retry_on_error
def add_ten_error_if_zero(args):
if args == 0:
raise Exception()
return args + 10

print add_ten_error_if_zero(0)
print add_ten_error_if_zero(10)



A much better approach is to use callbacks, the callbacks determines
whether to raise an exception or continue execution:

def handler(e):
if datetime.datetime.now() >= datetime.datetime(2012, 12, 21):
raise Exception('The world has ended')
# else: ignore, it's fine

def add_ten_error_if_zero(args, handler):
if args == 0:
handler(args)
return args + 10

print add_ten_error_if_zero(0, handler)
print add_ten_error_if_zero(10, handler)
print add_ten_error_if_zero(0, lambda e: None) # always succeeds



Ignoring arbitrary error is against the The Zen of Python "Errors should
never pass silently."; not that it is ever a good idea to ignore
arbitrary error, when an exception happens often the function is in an
indeterminate state, and continuing blindly could easily cause havocs.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Is there a way to continue after an exception ?

[sstein...@gmail.com]

On Feb 20, 2010, at 9:17 PM, Lie Ryan wrote:

> On 02/21/10 12:02, Stef Mientki wrote:
>> On 21-02-2010 01:21, Lie Ryan wrote:
 On Sun, Feb 21, 2010 at 12:52 AM, Stef Mientki
>  wrote:
 
> hello,
> 
> I would like my program to continue on the next line after an uncaught
> exception,
> is that possible ?
> 
> thanks
> Stef Mientki
> 
> 
>>> That reminds me of VB's "On Error Resume Next"
>>> 
>> I think that's what I'm after ...
> 
> First, read this:
> http://www.developerfusion.com/code/4325/on-error-resume-next-considered-harmful/

The link goes to an "Oh dear. Gremlins at work!" page.  

They're probably using On Error Resume Next in their VBScript code and this is 
the "last resort" page ;-).

S

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: The future of "frozen" types as the number of CPU cores increases

[John Nagle]

sjdevn...@yahoo.com wrote:

On Feb 18, 2:58 pm, John Nagle  wrote:

Multiple processes are not the answer.  That means loading multiple
copies of the same code into different areas of memory.  The cache
miss rate goes up accordingly.


A decent OS will use copy-on-write with forked processes, which should
carry through to the cache for the code.


   That doesn't help much if you're using the subprocess module.  The
C code of the interpreter is shared, but all the code generated from
Python is not.

   This will get even worse when Unladen Swallow starts generating vast
swaths of unshared x86 instructions.

John Nagle
--
http://mail.python.org/mailman/listinfo/python-list

Re: Is there a way to continue after an exception ?

[Ryan Kelly]

On Sun, 2010-02-21 at 13:17 +1100, Lie Ryan wrote:
> On 02/21/10 12:02, Stef Mientki wrote:
> > On 21-02-2010 01:21, Lie Ryan wrote:
> >>> On Sun, Feb 21, 2010 at 12:52 AM, Stef Mientki
>  wrote:
> >>>
>  hello,
> 
>  I would like my program to continue on the next line after an uncaught
>  exception,
>  is that possible ?
> 
>  thanks
>  Stef Mientki
> 
> 
> >> That reminds me of VB's "On Error Resume Next"
> >>
> > I think that's what I'm after ...
>
> A much better approach is to use callbacks, the callbacks determines
> whether to raise an exception or continue execution:
> 
> def handler(e):
> if datetime.datetime.now() >= datetime.datetime(2012, 12, 21):
> raise Exception('The world has ended')
> # else: ignore, it's fine
> 
> def add_ten_error_if_zero(args, handler):
> if args == 0:
> handler(args)
> return args + 10
> 
> print add_ten_error_if_zero(0, handler)
> print add_ten_error_if_zero(10, handler)
> print add_ten_error_if_zero(0, lambda e: None) # always succeeds


Or if you don't like having to explicitly manage callbacks, you can try
the "withrestart" module:

http://pypi.python.org/pypi/withrestart/

It tries to pinch some of the good ideas from Common Lisp's
error-handling system.

  from withrestart import *

  def add_ten_error_if_zero(n):
  #  This gives calling code the option to ignore
  #  the error, or raise a different one.
  with restarts(skip,raise_error):
  if n == 0:
  raise ValueError
  return n + 10

  # This will raise ValueError
  print add_ten_error_if_zero(0)

  # This will print 10
  with Handler(ValueError,"skip"):
  print add_ten_error_if_zero(0)

  # This will exit the python interpreter
  with Handler(ValueError,"raise_error",SystemExit):
  print add_ten_error_if_zero(0)



  Cheers,

  Ryan


-- 
Ryan Kelly
http://www.rfk.id.au  |  This message is digitally signed. Please visit
r...@rfk.id.au|  http://www.rfk.id.au/ramblings/gpg/ for details



signature.asc
Description: This is a digitally signed message part
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Efficient way to break up a list into two pieces

[marwie]

On 21 Feb., 02:41, Steven D'Aprano  wrote:
> What the OP is doing is quite different:
>
> (1) copy l1[:10]
> (2) assign the name l2 to it
> (3) resize l1 in place to the first 10 items.
>
> What the OP wants is:
>
> (1) assign the name l2 to l1[:10] without copying
> (2) resize l1 in place to the first 10 items without affecting l2.

Assuming you meant l1[10:] instead of l1[:10], that's what I wanted,
yes.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Not sure why this is filling my sys memory

[Vincent Davis]

Here is a sample of the output, It almost instantly uses 2GB and then starts
using VMem. This is probably the right suggestion but it's another thing to
install

> It's probably also worth being aware of guppy's heapy stuff:

http://guppy-pe.sourceforge.net/heapy_tutorialI
find it quite nice to have the following to get a quick point-in-time
estimate of my app's memory usage:

print 'Current memory usage: %iMB' % (hpy().heap().stat.size/(1024*1024))


19 files of 43 using 3352 bytes of memory
Loading into memory: 3_veg_nm_rep1.txt
3_veg_nm_rep1.txt has 228484 rows of data
3_veg_nm_rep1.txt has 0 rows of masked data
3_veg_nm_rep1.txt has 141 rows of outliers
3_veg_nm_rep1.txt has 0 modified rows of data
280bytes of memory used for 3_veg_nm_rep1.txt

20 files of 43 using 3352 bytes of memory
Loading into memory: 3_veg_nm_rep2.txt
3_veg_nm_rep2.txt has 228484 rows of data
3_veg_nm_rep2.txt has 0 rows of masked data
3_veg_nm_rep2.txt has 119 rows of outliers
3_veg_nm_rep2.txt has 0 modified rows of data
280bytes of memory used for 3_veg_nm_rep2.txt

21 files of 43 using 3352 bytes of memory
Loading into memory: 3_veg_phd_rep1.txt
3_veg_phd_rep1.txt has 228484 rows of data
3_veg_phd_rep1.txt has 0 rows of masked data
3_veg_phd_rep1.txt has 63 rows of outliers
3_veg_phd_rep1.txt has 0 modified rows of data
280bytes of memory used for 3_veg_phd_rep1.txt

22 files of 43 using 6424 bytes of memory
Loading into memory: 3g_c285-11.txt
3g_c285-11.txt has 228484 rows of data
3g_c285-11.txt has 0 rows of masked data
3g_c285-11.txt has 65 rows of outliers
3g_c285-11.txt has 0 modified rows of data
280bytes of memory used for 3g_c285-11.txt

23 files of 43 using 6424 bytes of memory
Loading into memory: 3g_c285-42.txt
3g_c285-42.txt has 228484 rows of data
3g_c285-42.txt has 0 rows of masked data
3g_c285-42.txt has 27 rows of outliers
3g_c285-42.txt has 0 modified rows of data
280bytes of memory used for 3g_c285-42.txt

24 files of 43 using 6424 bytes of memory
Loading into memory: A6AF.txt
A6AF.txt has 228484 rows of data
A6AF.txt has 0 rows of masked data
A6AF.txt has 36 rows of outliers
A6AF.txt has 0 modified rows of data
280bytes of memory used for A6AF.txt

25 files of 43 using 6424 bytes of memory
Loading into memory: Grigg_3026_rep1.txt
Grigg_3026_rep1.txt has 228484 rows of data
Grigg_3026_rep1.txt has 0 rows of masked data
Grigg_3026_rep1.txt has 949 rows of outliers
Grigg_3026_rep1.txt has 0 modified rows of data
280bytes of memory used for Grigg_3026_rep1.txt

26 files of 43 using 6424 bytes of memory
Loading into memory: Grigg_3026_rep2.txt
Grigg_3026_rep2.txt has 228484 rows of data
Grigg_3026_rep2.txt has 0 rows of masked data
Grigg_3026_rep2.txt has 361 rows of outliers
Grigg_3026_rep2.txt has 0 modified rows of data
280bytes of memory used for Grigg_3026_rep2.txt

27 files of 43 using 6424 bytes of memory
Loading into memory: Grigg_3026_rep3_both.txt
Grigg_3026_rep3_both.txt has 228484 rows of data
Grigg_3026_rep3_both.txt has 0 rows of masked data
Grigg_3026_rep3_both.txt has 41 rows of outliers
Grigg_3026_rep3_both.txt has 0 modified rows of data
280bytes of memory used for Grigg_3026_rep3_both.txt

28 files of 43 using 6424 bytes of memory
Loading into memory: Grigg_3131_rep1.txt
Grigg_3131_rep1.txt has 228484 rows of data
Grigg_3131_rep1.txt has 0 rows of masked data
Grigg_3131_rep1.txt has 537 rows of outliers
Grigg_3131_rep1.txt has 0 modified rows of data
280bytes of memory used for Grigg_3131_rep1.txt

29 files of 43 using 6424 bytes of memory
Loading into memory: Grigg_3131_rep2.txt
Grigg_3131_rep2.txt has 228484 rows of data
Grigg_3131_rep2.txt has 0 rows of masked data
Grigg_3131_rep2.txt has 238 rows of outliers
Grigg_3131_rep2.txt has 0 modified rows of data
280bytes of memory used for Grigg_3131_rep2.txt

30 files of 43 using 6424 bytes of memory

  *Vincent Davis
720-301-3003 *
vinc...@vincentdavis.net
 my blog  |
LinkedIn


On Sat, Feb 20, 2010 at 7:40 PM, sstein...@gmail.com wrote:

>
> On Feb 20, 2010, at 9:21 PM, Vincent Davis wrote:
>
> See this article for some more info about the reported sizes of things:
>> http://www.doughellmann.com/PyMOTW/sys/limits.html
>
>
> Nice article but I
> must have missed something useful to my current issue. Do I get any hints?
>
>
> Oh, sorry, there was the part about getsizeof() not including attributes
> unless the class supplies a __sizeof__ method and a comment at the bottom:
>
> It's probably also worth being aware of guppy's heapy stuff:
> http://guppy-pe.sourceforge.net/heapy_tutorial
>
> I find it quite nice to have the following to get a quick point-in-time
> estimate of my app's memory usage:
>
> print 'Current memory usage: %iMB' % (hpy().heap().stat.size/(1024*1024))
>
> Put a few of those in at various places in

Re: netcdf4-python

[Matt Newville]

On Feb 20, 7:47 pm, deadpickle  wrote:
> I'm trying to use the python module "netcdf4-python" to read a netcdf
> file. So far I'm trying to do the basics and just open the script:
>
> from netCDF4 import Dataset
> rootgrp = Dataset('20060402-201025.netcdf', 'r',
> format='NETCDF3_CLASSIC')
> print rootgrp.file_format
> rootgrp.close()
>
> when I do this I get the exit code error (when I run in Scite):
>
> >pythonw -u "netcdf_test.py"
> >Exit code: -1073741819
>
>  or Python stops responding (in windows cmd).
>
> I'm not sure what is wrong so if anyone as any ideas I would gladly
> send you the netcdf file to try. Thanks.

If the file is really of NetCDF3 format, scipy.io.netcdf should work.
Replace
  netCDF4.Dataset(filename,'r',format='NETCDF3_CLASSIC')
with
  scipy.io.netcdf.netcdf_open(filename,'r')

--Matt Newville
-- 
http://mail.python.org/mailman/listinfo/python-list

lists of variables

[Michael Pardee]

I'm relatively new to python and I was very surprised by the following behavior:

>>> a=1
>>> b=2
>>> mylist=[a,b]
>>> print mylist
[1, 2]
>>> a=3
>>> print mylist
[1, 2]

Whoah!  Are python lists only for literals?  Nope:

>>> c={}
>>> d={}
>>> mydlist=[c,d]
>>> print mydlist
[{}, {}]
>>> c['x']=1
>>> print mydlist
[{'x': 1}, {}]

So it looks like variables in a list are stored as object references.
This seems to confirm that:

mydlist[1]['y']=4
>>> print mydlist
[{}, {'y': 4}]

So I figure my initial example doesn't work because if you assign a
literal to something it is changing the object.  But modifying a list
or dict (as long as you don't re-construct it) does not change the
object.

I can think of some ways to work around this, including using single
element lists as "pointers":

>>> aa=[1]
>>> bb=[2]
>>> myplist=[aa,bb]
>>> print myplist
[[1], [2]]
>>> aa[0]=3
>>> print myplist
[[3], [2]]


But what would be "the python way" to accomplish "list of variables"
functionality?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: lists of variables

[Chris Rebert]

On Sat, Feb 20, 2010 at 7:25 PM, Michael Pardee
 wrote:
> I'm relatively new to python and I was very surprised by the following 
> behavior:
>
 a=1
 b=2
 mylist=[a,b]
 print mylist
> [1, 2]
 a=3
 print mylist
> [1, 2]
>
> Whoah!  Are python lists only for literals?  Nope:
>
 c={}
 d={}
 mydlist=[c,d]
 print mydlist
> [{}, {}]
 c['x']=1
 print mydlist
> [{'x': 1}, {}]
>
> So it looks like variables in a list are stored as object references.
> This seems to confirm that:
>
> mydlist[1]['y']=4
 print mydlist
> [{}, {'y': 4}]
>
> So I figure my initial example doesn't work because if you assign a
> literal to something it is changing the object.  But modifying a list
> or dict (as long as you don't re-construct it) does not change the
> object.

Correct. If you want more gory details, read
http://effbot.org/zone/call-by-object.htm

> I can think of some ways to work around this, including using single
> element lists as "pointers":
>
 aa=[1]
 bb=[2]
 myplist=[aa,bb]
 print myplist
> [[1], [2]]
 aa[0]=3
 print myplist
> [[3], [2]]
>
>
> But what would be "the python way" to accomplish "list of variables"
> functionality?

What do you need that functionality for exactly? It's a rather low-level notion.

Cheers,
Chris
--
http://blog.rebertia.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Efficient way to break up a list into two pieces

[Steven D'Aprano]

On Sat, 20 Feb 2010 17:55:18 -0800, marwie wrote:

> On 21 Feb., 02:30, Steven D'Aprano  cybersource.com.au> wrote:
>> Python lists are arrays of pointers to objects, so copying a slice is
>> fast: it doesn't have to copy the objects, just pointers. Deleting from
>> the end of the list is also quick, because you don't have to move
>> memory, just clear some pointers and change the length field.
>>
>> Splitting such an array without copying data is, essentially,
>> impossible. Python lists aren't linked lists.
> 
> Well, to split a C array I would simply set l2 to point to l1[10] and
> then change the length of l1 (which I store somewhere else). No copying
> of elements needed. I would have assumed that python can do something
> like this with its internal arrays of pointers, too.

Python lists aren't C arrays either.

Python lists are *objects*. Everything in Python is an object. 
Consequently, Python lists have a header, which includes the len. You 
don't store the length somewhere else, you store it in the object: this 
makes for less complexity. You can't just point l2 at an arbitrary index 
in an array and expect it to work, it needs a header.

Additionally, Python lists are over-allocated so that appends are fast. A 
list of (say) 1000 items might be over-allocated to (say) 1024 items, so 
that you can do 24 appends before the array is full and the array needs 
to be resized. This means that, on average, Python list appending is O(1) 
instead of O(N). You can't just change the length blindly, you need to 
worry about the over-allocation.

I'm sympathetic to your concern: I've often felt offended that doing 
something like this:

x = SomeReallyBigListOrString
for item in x[1:]:
process(item)

has to copy the entire list or string (less the first item). But 
honestly, I've never found a situation where it actually mattered.


> Anyway, this was more a question about coding style. I use l1.extend(l2)
> or l1 += l2 rather than l1 = l1 + l2 because it's as readable and
> possibly faster.

I really, really hate augmented assignment for everything other than ints 
and floats because you can't predict what it will do. Take this for 
example:


>>> mylist = [1, 2, 3, 4]
>>> same_again = mylist
>>> mylist.append(5)
>>> same_again
[1, 2, 3, 4, 5]

mylist and same_again are the same object, so append and extend behave 
predictably and simply. So is simple too:

>>> mylist = mylist + [-1, -2, -3]
>>> mylist
[1, 2, 3, 4, 5, -1, -2, -3]
>>> same_again
[1, 2, 3, 4, 5]

Because you have re-bound the name mylist to a new list, same_again 
doesn't get modified. Nice.

But what about mylist += something else? Is it an in-place modification, 
like extend and append, or a rebinding, like +? Who can remember? The 
answer will depend on whether mylist is a builtin list, or a subclass.

And then there's this mystery:

>>> mylist = [1, 2, 3]
>>> mytuple = (mylist, None)
>>> mytuple[0] += [4]
Traceback (most recent call last):
  File "", line 1, in 
TypeError: 'tuple' object does not support item assignment
>>> mylist
[1, 2, 3, 4]

So in fact, += is *always* a rebinding, but *sometimes* it is an in-place 
modification as well. Yuck.


-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: lists of variables

[Stephen Hansen]

On Sat, Feb 20, 2010 at 7:25 PM, Michael Pardee
wrote:

> But what would be "the python way" to accomplish "list of variables"
> functionality?
>

The problem is... Python doesn't have variables. At least not in the way
that you may be used to from other languages. Yeah, it's got data, and data
obviously has to be identifiable in some way, but they aren't put together
like "variables" are. A variable is a box with a name on it, and in that
context a 'list of variables' makes sense. You're moving around the box, or
referencing the box itself. It may happen to have something in it, but that
'something' doesn't really have an identity of its own. Its just.. the thing
that's in box-A.

In Python, everything's an object. And objects don't really have identities
of their own(okay, they do, but its not meaningful except to an identity
test), but they can be named. They object has no flabbering clue what its
named, or how many different people know its name, but hey.

a = 1

In many languages, that's making a variable "a" and assigning it the value
of 1.

In Python, that's making an int object with a value of 1, and giving it a
name in the current namespace. This -isn't- just using dumber/simpler words
to say the same thing :) All(*) python namespaces are really dictionaries
behind the scenes. a = 1 is actually doing, current_namespaces["a"] = 1.

Those may sound similar, but they're really not: most importantly, because
the "a" in another language is a -thing- unto itself which may hold an
object. But "a" in Python is just a label in some dictionary somewhere. You
can get the object a is naming, but you can't get /a/ itself. Its not a
'thing'.

There's no way to make a "list of variables", because /variables/ are the
part that's really missing. If you do:

lst = [a, b]

Python doesn't create a list and put two variables in it. It creates a list,
and while doing so, sees that we want to pass 'a' and 'b' into that list. So
it grabs the actual objects a and b are naming, and puts them in the list.

So, long story short: what's "the python way" to accomplish a "list of
variables"? Well... IMHO, it's to not do so. : )

Make a dict, pass the dict around, there's your list of variables. True, you
can't ever access them /as/ variables, and have to access them as dict
items, but... that's the Python way to do that. If order matters, do an
ordered dict.

HTH,
--S

*: Yes, I know about local namespaces being optimized in CPython to be an
array-like structure. Implementation detail.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: The future of "frozen" types as the number of CPU cores increases

[Paul Rubin]

John Nagle  writes:
>> A decent OS will use copy-on-write with forked processes, which should
>> carry through to the cache for the code.
>
>That doesn't help much if you're using the subprocess module.  The
> C code of the interpreter is shared, but all the code generated from
> Python is not.

Emacs in days of yore used a hack called "unexec" to bring its Lisp code
into the pure text segment.  I think it's not used any more because
machines are now big enough that the benefits aren't that great.
Basically in the Emacs build process, you'd start up the C portion of
Emacs, then tell it to load all its Lisp code, then call "unexec".
Unexec basically performed a core dump of the process, then ran
something that manipulated the original Emacs image and the core dump
into a new executable that had the static Lisp code and data now marked
as part of the (immutable) program area.  Unexec necessarily had
platform dependencies, but this was managed with a bunch of ifdefs in a
reasonably clean and maintainable way.  I could imagine doing something
similar with a large Python program.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: lists of variables

[Ben Finney]

Michael Pardee  writes:

> But what would be "the python way" to accomplish "list of variables"
> functionality?

You'll need to explain what “list of variables” functionality is.

If you mean “collection of name-to-value mappings”, the native mapping
type in Python is ‘dict’. If that doesn't meet your needs, you'll need
to be more specific about what behaviour you want.

-- 
 \“Most people don't realize that large pieces of coral, which |
  `\   have been painted brown and attached to the skull by common |
_o__)wood screws, can make a child look like a deer.” —Jack Handey |
Ben Finney
-- 
http://mail.python.org/mailman/listinfo/python-list