python path separator

[swapnil]

I could not find any documentation for variables os.path.sep and
os.path.altsep. Although the first is pretty straightforward can
anyone explain the purpose of the second variable? Is it even useful?
According to issue http://bugs.python.org/issue709428, os.path.altsep
was 'None' till a long time and it didn't bother anyone?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Selecting k smallest or largest elements from a large list in python; (benchmarking)

[Peter Otten]

Dmitry Chichkov wrote:

> Given: a large list (10,000,000) of floating point numbers;
> Task: fastest python code that finds k (small, e.g. 10) smallest
> items, preferably with item indexes;
> Limitations: in python, using only standard libraries (numpy & scipy
> is Ok);
> 
> I've tried several methods. With N = 10,000,000, K = 10 The fastest so
> far (without item indexes) was pure python implementation
> nsmallest_slott_bisect (using bisect/insert). And with indexes
> nargsmallest_numpy_argmin (argmin() in the numpy array k times).
> 
> Anyone up to the challenge beating my code with some clever selection
> algorithm?

If you don't care about stability, i. e. whether nlargest(2, [1, 2, 2.0]) 
returns [1, 2] or [1, 2.0], use

_heapq.nlargest() directly

$ python nsmallest_perf2.py
nsmallest  --> 0.142784833908
nsmallest_slott_bisect --> 0.19291305542
$ cat nsmallest_perf2.py
from random import randint, random
import time
from bisectimport insort
from itertools import islice
import _heapq

_funcs = []
def timeit(f):
_funcs.append(f)

def time_all(*args):
funcs = _funcs
width = max(len(f.__name__) for f in funcs)
prev = None
for f in funcs:
start = time.time()
result = f(*args)
end = time.time()
print "%-*s --> %10s" % (width, f.__name__, end - start)
if prev is None:
prev = result
else:
assert prev == result

timeit(_heapq.nsmallest)

@timeit
def nsmallest_slott_bisect(n, iterable, insort=insort):
it   = iter(iterable)
mins = sorted(islice(it, n))
for el in it:
if el <= mins[-1]: #NOTE: equal sign is to preserve duplicates
insort(mins, el)
mins.pop()
return mins

test_data = [randint(10, 50) + random() for i in range(10**6)]
K = 10

time_all(K, test_data)

Peter
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Selecting k smallest or largest elements from a large list in python; (benchmarking)

[Peter Otten]

Dmitry Chichkov wrote:

> Code:

A lot of the following doesn't run or returns incorrect results. 
To give but one example:

> def nargsmallest_numpy_argmin(iter, k):
> distances = N.asarray(iter)
> mins = []

Could you please provide an up-to-date version?

Peter

PS: for an easy way to ensure consistency see timeit/time_all in my previous 
post.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: python path separator

[Vlastimil Brom]

2010/9/2 swapnil :
> I could not find any documentation for variables os.path.sep and
> os.path.altsep. Although the first is pretty straightforward can
> anyone explain the purpose of the second variable? Is it even useful?
> According to issue http://bugs.python.org/issue709428, os.path.altsep
> was 'None' till a long time and it didn't bother anyone?
> --
> http://mail.python.org/mailman/listinfo/python-list
>
see
http://docs.python.org/library/os.html#os.sep
http://docs.python.org/library/os.html#os.altsep

On windows it returns a slash
>>> os.altsep
'/'
which is often easier to use (i.e. if I understand correctly, in most
usual cases the forward slash should probably just work on most of the
recent OSes).

   vbr
-- 
http://mail.python.org/mailman/listinfo/python-list

Is python is the safest language ...

[Anand Sadasivam]

Hi All,

Is python is the safest language is my question. I feel its good language,
but however java has good occupancy. About four years back I did few python
programs and worked with some of add-on product over ZOPE and Plone.

The suggestion to me is mostly welcome.

Regards,
-- 
Anand.S
anand.sadasi...@gmail.com
asada...@gmail.com
+91 99025 38904
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python-list Digest, Vol 84, Issue 8

[Nally Kaunda-Bukenya]

Dear Peter, that did wonders!! thanks so much for the code fix; I will check 
back with you later for the meaning of some functions you used.

Many thanks to all those good people who gave me pointers: Rami, Mathew, Bob,  

Best wishes for now:)



From: "python-list-requ...@python.org" 
To: python-list@python.org
Sent: Wed, September 1, 2010 11:52:02 AM
Subject: Python-list Digest, Vol 84, Issue 8

Note: Forwarded message is attached.

Send Python-list mailing list submissions to
    python-list@python.org

To subscribe or unsubscribe via the World Wide Web, visit
    http://mail.python.org/mailman/listinfo/python-list
or, via email, send a message with subject or body 'help' to
    python-list-requ...@python.org

You can reach the person managing the list at
    python-list-ow...@python.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Python-list digest..."
Today's Topics:

  1. DeprecationWarning (cerr)
  2. Re: Performance: sets vs dicts. (Stefan Behnel)
  3. Better multiprocessing and data persistance with C level
      serialisation (ipatrol6...@yahoo.com)
  4. Re: Dumb Stupid Question About List and String (Xavier Ho)
  5. Re: Dumb Stupid Question About List and String (Alban Nona)
  6. Re: Dumb Stupid Question About List and String (Alban Nona)
  7. Re: Newby Needs Help with Python code (Peter Otten)
-- 
http://mail.python.org/mailman/listinfo/python-list


  -- 
http://mail.python.org/mailman/listinfo/python-list

Re: Selecting k smallest or largest elements from a large list in python; (benchmarking)

[Arnaud Delobelle]

On Sep 2, 7:59 am, Peter Otten <__pete...@web.de> wrote:
> Dmitry Chichkov wrote:
> > Given: a large list (10,000,000) of floating point numbers;
> > Task: fastest python code that finds k (small, e.g. 10) smallest
> > items, preferably with item indexes;
> > Limitations: in python, using only standard libraries (numpy & scipy
> > is Ok);
>
> > I've tried several methods. With N = 10,000,000, K = 10 The fastest so
> > far (without item indexes) was pure python implementation
> > nsmallest_slott_bisect (using bisect/insert). And with indexes
> > nargsmallest_numpy_argmin (argmin() in the numpy array k times).
>
> > Anyone up to the challenge beating my code with some clever selection
> > algorithm?
>
> If you don't care about stability, i. e. whether nlargest(2, [1, 2, 2.0])
> returns [1, 2] or [1, 2.0], use
>
> _heapq.nlargest() directly
>
> $ python nsmallest_perf2.py
> nsmallest              --> 0.142784833908
> nsmallest_slott_bisect --> 0.19291305542
> $ cat nsmallest_perf2.py
> from random import randint, random
> import time
> from bisect    import insort
> from itertools import islice
> import _heapq
>
> _funcs = []
> def timeit(f):
>     _funcs.append(f)
>
> def time_all(*args):
>     funcs = _funcs
>     width = max(len(f.__name__) for f in funcs)
>     prev = None
>     for f in funcs:
>         start = time.time()
>         result = f(*args)
>         end = time.time()
>         print "%-*s --> %10s" % (width, f.__name__, end - start)
>         if prev is None:
>             prev = result
>         else:
>             assert prev == result
>
> timeit(_heapq.nsmallest)
>
> @timeit
> def nsmallest_slott_bisect(n, iterable, insort=insort):
>     it   = iter(iterable)
>     mins = sorted(islice(it, n))
>     for el in it:
>         if el <= mins[-1]: #NOTE: equal sign is to preserve duplicates
>             insort(mins, el)
>             mins.pop()
>     return mins
>
> test_data = [randint(10, 50) + random() for i in range(10**6)]
> K = 10
>
> time_all(K, test_data)
>
> Peter

I get:

1.46s for _heapq.nsmallest
0.85s for nsmallest_slott_bisect2 (version I posted)

I am a bit surprised that mine is so slow compared with yours.  I'll
do more tests later!
--
Arnaud
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Windows vs. file.read

[Lawrence D'Oliveiro]

In message , MRAB 
wrote:

> You should open the files in binary mode, not text mode, ie file(path,
> "rb"). Text mode is the default. Not a problem on *nix because the line
> ending is newline.

We used to pride ourselves on not having to worry about text versus binary 
I/O modes on *nix, but I’m beginning to think the reality is we have to 
adopt it.

To start with, it means we can automatically handle different newline 
conventions with text files originating on different systems.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Is python is the safest language ...

[Chris Rebert]

On Thu, Sep 2, 2010 at 12:29 AM, Anand Sadasivam
 wrote:
> Hi All,
>
> Is python is the safest language is my question. I feel its good language,
> but however java has good occupancy. About four years back I did few python
> programs and worked with some of add-on product over ZOPE and Plone.

The question is vague.
"Safest" in what sense? Type safety? Long-term viability? Difficulty
in the programmer shooting themself in the foot?

Cheers,
Chris
--
http://blog.rebertia.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Is there a Python equivalent to Perl's File::Listing::parse_dir

[Stefan Schwarzer]

Hi John,

On 2010-08-11 20:24, John Nagle wrote:
>Perl has a function which will take a remote directory page, in
> the form that most web sites return for a file directory, and
> parse it into a useful form:
> 
>   http://www.xav.com/perl/site/lib/File/Listing.html
> 
> This is especially useful for FTP sites.
> 
> Is there a Python equivalent of this?  I'm converting some
> old Perl code.
> 
> Even the Python FTP module doesn't have a directory parser.

I saw this posting only now. I hope it's not too late to
point you to ftputil, http://ftputil.sschwarzer.net . :-)
As the name implies, it's FTP-only for now, though.

If you have any questions regarding the library, please ask.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: fairly urgent request: paid python (or other) work required

[Daniel Fetchinson]

On 9/1/10, lkcl  wrote:
> i apologise for having to contact so many people but this is fairly
> urgent, and i'm running out of time and options.   i'm a free software
> programmer, and i need some paid work - preferably python - fairly
> urgently, so that i can pay for food and keep paying rent, and so that
> my family doesn't get deported or have to leave the country.
>
> i really would not be doing this unless it was absolutely, absolutely
> essential that i get money.
>
> so that both i and the list are not unnecessarily spammed, please
> don't reply with recommendations of "where to get jobs", unless they
> are guaranteed to result in immediate work and money.
>
> if you have need of a highly skilled and experienced python-preferring
> free-software-preferring software engineer, please simply contact me,
> and tell me what you need doing: there's no need for you to read the
> rest of this message.
>
> so that people are not offended by me asking on such a high-volume
> list for work, here are some questions and answers:
>
> Q: who are you?
> A: luke leighton.  free sofware developer, free software project
> leader, and "unusual cross-project mash-up-er" (meaning: i spot the
> value of joining one or more bits of disparate "stuff" to make
> something that's more powerful than its components).
>
> Q: where's your CV?
> A: executive version of CV is at http://lkcl.net/exec_cv.txt - please
> don't ask for a proprietary microsoft word version, as a refusal and
> referral to the "sylvester response" often offends.
>
> Q: what can you do?
> A: python programming, c programming, web development, networking,
> cryptography, reverse-engineering, IT security, etc. etc. preferably
> involving free software.
>
> Q: what do you need?
> A: money to pay rent and food.  at the ABSOLUTE MINIMUM, i need as
> little as £1500 per month to pay everything, and have been earning
> approx £800 per month for the past year.   a £5000 inheritance last
> year which i was not expecting has delayed eviction and bankruptcy for
> me and my family, and deportation for my partner and 17 month old
> daughter (marie is here in the UK on a FLR/M visa)
>
> Q: why are you asking here?
> A: because it's urgent that i get money really really soon; my family
> members are refusing to assist, and the few friends that i have do not
> have any spare money to lend.
>
> Q: why here and not "monster jobs" or "python-jobs list" or the
> various "recruitment agencies"?
> A: those are full-time employment positions, which i have been
> frequently applying for and get rejected for various reasons, and i'm
> running out of time and money.  further interviews cost money, and do
> not result in guaranteed work.  i need work - and money - _now_.
>
> Q: why here and not "peopleperhour.com"?
> A: if you've ever bid on peopleperhour.com you will know that you are
> bidding against "offshore" contrators and even being undercut by 1st
> world country bidders who, insanely, appear to be happy to do work for
> as little as £2 / hour.
>
> Q: why are you getting rejected from interviews?
> A: that's complex.  a) i simply don't interview well.  people with the
> classic symptoms of asperger's just don't. b) my daughter is 17 months
> old.  when i go away for as little as 3 days, which i've done three
> times now, she is extremely upset both when i am away and when i
> return.  i think what would happen if i was doing some sort of full-
> time job, away from home, and... i can't do it.  subconsciously that
> affects how i react when speaking to interviewers.
>
> Q: why do you not go "get a job at tesco's" or "drive a truck"?
> A: tescos and HGV driving etc. pay around £12 per hour.  £12 per hour
> after tax comes down to about £8 to £9 per hour.  £9 per hour requires
> 35 hours per week to earn as little as £1500.  35 hours per week is
> effectively full-time, and means that a) my programming and software
> engineering skills are utterly, utterly wasted b) my daughter gets
> extremely upset because i won't be at home.
>
> so you get the gist, and thank you for putting up with me needing to
> take this action.



For the sake of your family, I'd recommend taking the following
paragraphs off your cv:

"""
If you require someone whom you do NOT want to take
the initiative to assess all aspects of the required
work and beyond; if you require someone who will
"stay in the box", do NOT contact me.
"""

"""
The speed at which I acquire new knowledge and be
able to integrate and recommend new ideas tends to
make insecure people feel frightened and threatened.
"""

"""
Pet Hates:  - "peer to peer networking" [peer equals neanderthal...]
- "microsoft marketing machine" [FUD at its best...]
- "restrictive anti-competitive business practices"
- "information-restricive laws" [DMCA etc.]
- Unreliable systems that cannot be made to do the

Re: Queue cleanup

[Paul Rubin]

Dennis Lee Bieber  writes:
>> GC's for large systems ... copy the live objects to a new contiguous heap

>   That sounds suspiciously like the original Macintosh OS, with its
> "handles"... IE, double-indirection. 

Nah, a double indirection on every access would be a terrible
performance hit.  The classic approach is when you move an object to the
new heap, you leave a tagged forwarding pointer at its former location
the old heap, giving the its location in the new heap.  Then as you move
other objects, you dereference the pointers in them to see whether they
point to moved or unmoved objects, and relocate any unmoved ones.  A
more complete explanation is here:

http://mitpress.mit.edu/sicp/full-text/book/book-Z-H-33.html#%_sec_5.3.2 
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Selecting k smallest or largest elements from a large list in python; (benchmarking)

[Peter Otten]

Arnaud Delobelle wrote:

> I get:
> 
> 1.46s for _heapq.nsmallest
> 0.85s for nsmallest_slott_bisect2 (version I posted)
> 
> I am a bit surprised that mine is so slow compared with yours.  I'll
> do more tests later!

Strange. I see a significant difference only for python3 (on 64bit Linux)

$ python3 nsmallest_perf3.py
3.1.1+ (r311:74480, Nov  2 2009, 15:45:00)
[GCC 4.4.1]
pick the 10 smallest items out of 500
nsmallest   --> 0.310983181
nsmallest_slott_bisect  --> 1.02625894547
nsmallest_slott_bisect2 --> 0.612951040268

$ python nsmallest_perf3.py
2.6.4 (r264:75706, Dec  7 2009, 18:43:55)
[GCC 4.4.1]
pick the 10 smallest items out of 500
nsmallest   --> 0.743387937546
nsmallest_slott_bisect  --> 0.961116075516
nsmallest_slott_bisect2 --> 0.746085882187

Peter
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: PyPy and RPython

[sarvi]

When I think about it these restrictions below seem a very reasonable
tradeoff for performance.
And I can use this for just the modules/sections that are performance
critical.

Essentially, the PyPy interpreter can have a restricted mode that
enforces these restriction.
This will help write such RPython code that can then be compiled into
C/ASM using the PyPy Compiler which as I understand can do this today.

If Shedskin generated C++ code is faster than PyPy generate C++ code.
Isn't that just another reason why PyPy and Shedskin should be joining
forces.

Wouldn't we want PyPy generated C code to be just as fast as
Shedskin's

Afterall thats how the PyPy compiler is built, right? and we do want
that to be fast too?

Sarvi



On Sep 1, 11:39 pm, John Nagle  wrote:
> On 9/1/2010 10:49 AM, sarvi wrote:
>
>
>
>
>
>
>
> > Is there a plan to adopt PyPy and RPython under the python foundation
> > in attempt to standardize both.
>
> > I have been watching PyPy and RPython evolve over the years.
>
> > PyPy seems to have momentum and is rapidly gaining followers and
> > performance.
>
> > PyPy JIT and performance would be a good thing for the Python
> > Community
> > And it seems to be well ahead of Unladen Swallow in performance and in
> > a position to improve quite a bit.
>
> > Secondly I have always fantasized of never having to write C code yet
> > get its compiled performance.
> > With RPython(a strict subset of Python), I can actually compile it to
> > C/Machine code
>
> > These 2 seem like spectacular advantages for Python to pickup on.
> > And all this by just showing the PyPy and the Python foundation's
> > support and direction to adopt them.
>
> > Yet I see this forum relatively quiet on PyPy or Rpython ?  Any
> > reasons???
>
> > Sarvi
>
>      The winner on performance, by a huge margin, is Shed Skin,
> the optimizing type-inferring compiler for a restricted subset
> of Python.  PyPy and Unladen Swallow have run into the problem
> that if you want to keep some of the less useful dynamic semantics
> of Python, the heavy-duty optimizations become extremely difficult.
>
>      However, if we defined a High Performance Python language, with
> some restrictions, the problem becomes much easier.  The necessary
> restrictions are roughly this:
>
> -- Functions, once defined, cannot be redefined.
>     (Inlining and redefinition do not play well
>     together.)
>
> -- Variables are implicitly typed for the base types:
>     integer, float, bool, and everything else.  The
>     compiler figures this out automatically.
>     (Shed Skin does this now.)
>
> -- Unless a class uses a "setattr" function or has
>     a __setattr__ method, its entire list of attributes is
>     known at compile time.
>     (In other words, you can't patch in new attributes
>     from outside the class unless the class indicates
>     it supports that.  You can subclass, of course.)
>
> -- Mutable objects (other than some form of synchronized
>     object) cannot be shared between threads.  This is the
>     key step in getting rid of the Global Interpreter Lock.
>
> -- "eval" must be restricted to the form that has a list of
>     the variables it can access.
>
> -- Import after startup probably won't work.
>
> Those are the essential restrictions.  With those, Python
> could go 20x to 60x faster than CPython.  The failures
> of PyPy and Unladen Swallow to get any significant
> performance gains over CPython demonstrate the futility
> of trying to make the current language go fast.
>
> Reference counts aren't a huge issue.  With some static
> analysis, most reference count updates can be optimized out.
> (As for how this is done, the key issue is to determine whether
> each function "keeps" a reference to each parameter.  For
> any function which does not, that parameter doesn't have
> to have reference count updates within the function.
> Most math library functions have this property.
> You do have to analyze the entire program globally, though.)
>
>                                 John Nagle

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Selecting k smallest or largest elements from a large list in python; (benchmarking)

[Dmitry Chichkov]

Uh. I'm sorry about the confusion. Last three items are just O(N)
baselines. Python min(), Numpy argmin(), Numpy asarray().
I'll update the code. Thanks!

> A lot of the following doesn't run or returns incorrect results.
> To give but one example:
>
> > def nargsmallest_numpy_argmin(iter, k):
> >     distances = N.asarray(iter)
> >     mins = []
>
> Could you please provide an up-to-date version?
>
> Peter
>
> PS: for an easy way to ensure consistency see timeit/time_all in my previous
> post.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Selecting k smallest or largest elements from a large list in python; (benchmarking)

[Dmitry Chichkov]

By the way, improving n-ARG-smallest (that returns indexes as well as
values) is actually more desirable than just regular n-smallest:

== Result ==
1.38639092445 nargsmallest
3.1569879055 nargsmallest_numpy_argsort
1.29344892502 nargsmallest_numpy_argmin

Note that numpy array constructor eats around 0.789.


== Code ==
from operator import itemgetter
from random import randint, random
from itertools import islice
from time import time

def nargsmallest(iterable, n):
it   = enumerate(iterable)
mins = sorted(islice(it, n), key = itemgetter(1))
loser = mins[-1][1] # largest of smallest
for el in it:
if el[1] <= loser:  # NOTE: preserve dups
mins.append(el)
mins.sort(key = itemgetter(1))
mins.pop()
loser = mins[-1][1]
return mins

def nargsmallest_numpy_argsort(iter, k):
distances = N.asarray(iter)
return [(i, distances[i]) for i in distances.argsort()[0:k]]

def nargsmallest_numpy_argmin(iter, k):
mins = []
distances = N.asarray(iter)
for i in xrange(k):
j = distances.argmin()
mins.append((j, distances[j]))
distances[j] = float('inf')
return mins

test_data = [randint(10, 50) + random() for i in range(1000)]
K = 10

init = time()
mins = nargsmallest(test_data, K)
print time() - init, 'nargsmallest:', mins[:2]

import numpy as N
init = time()
mins = nargsmallest_numpy_argsort(test_data, K)
print time() - init, 'nargsmallest_numpy_argsort:', mins[:2]

init = time()
mins = nargsmallest_numpy_argmin(test_data, K)
print time() - init, 'nargsmallest_numpy_argmin:', mins[:2]
-- 
http://mail.python.org/mailman/listinfo/python-list

comp.lang.python

[roshini begum]

www.127760.blogspot.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Speed-up for loops

[Michael Kreim]

Hi,

I was comparing the speed of a simple loop program between Matlab and 
Python.


My Codes:
$ cat addition.py
imax = 10
a = 0
for i in xrange(imax):
a = a + 10
print a

$ cat addition.m
imax = 1e9;
a = 0;
for i=0:imax-1
a = a + 10;
end
disp(a);
exit;

The results look like this:
$ /usr/bin/time --verbose python addition.py
100
Command being timed: "python addition.py"
User time (seconds): 107.30
System time (seconds): 0.08
Percent of CPU this job got: 97%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:50.09
[...]

$ /usr/bin/time --verbose matlab -nodesktop -nosplash -r "addition"
[...]
1.e+10
Command being timed: "matlab -nodesktop -nosplash -r addition"
User time (seconds): 7.65
System time (seconds): 0.18
Percent of CPU this job got: 94%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:08.25
[...]

Unfortunately my Python Code was much slower and I do not understand why.

Are there any ways to speed up the for/xrange loop?
Or do I have to live with the fact that Matlab beats Python in this example?

Thanks a lot for your answers.

With best regards,

Michael
--
http://mail.python.org/mailman/listinfo/python-list

Re: python path separator

[swapnil]

On Sep 2, 12:25 pm, Vlastimil Brom  wrote:
> 2010/9/2 swapnil :> I could not find any documentation 
> for variables os.path.sep and
> > os.path.altsep. Although the first is pretty straightforward can
> > anyone explain the purpose of the second variable? Is it even useful?
> > According to issuehttp://bugs.python.org/issue709428, os.path.altsep
> > was 'None' till a long time and it didn't bother anyone?
> > --
> >http://mail.python.org/mailman/listinfo/python-list
>
> seehttp://docs.python.org/library/os.html#os.sephttp://docs.python.org/library/os.html#os.altsep
>
> On windows it returns a slash>>> os.altsep
>
> '/'
> which is often easier to use (i.e. if I understand correctly, in most
> usual cases the forward slash should probably just work on most of the
> recent OSes).
>
>    vbr

Thanks for help
-- 
http://mail.python.org/mailman/listinfo/python-list

I GOT $5000 FROM PAYPAL BY SIMPLE HACK.

[paypal cash]

I GOT $5000 FROM PAYPAL BY SIMPLE HACK At  http://happyandeasy.co.cc

Due to high security risks, i have hidden the PayPal Form link in an
image.  in that website On RIGHT SIDE Below search box , click on
image and enter your PAYPAL id And Your name.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Speed-up for loops

[Peter Otten]

Michael Kreim wrote:

> I was comparing the speed of a simple loop program between Matlab and
> Python.
> 
> My Codes:
> $ cat addition.py
> imax = 10
> a = 0
> for i in xrange(imax):
>  a = a + 10
> print a

> Are there any ways to speed up the for/xrange loop?

Move it into a function; this turns a and i into local variables.

def f():
imax = 10
a = 0
for i in xrange(imax):
a = a + 10
print a
f()

> Or do I have to live with the fact that Matlab beats Python in this
> example?

I think so.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: dirty problem 3 lines

[bussiere bussiere]

It seems to work perfectly
thanks a lot
Bussiere
Google Fan boy



On Thu, Sep 2, 2010 at 7:32 AM, alex23  wrote:
> bussiere bussiere  wrote:
>> it's just as it seems :
>> i want to know how does ti works to get back an object from a string in 
>> python :
>> pickle.loads("""b'\x80\x03]q\x00(K\x00K\x01e.'""") #doesn't work
>
> Repeating the question without providing any further information
> doesn't really help.
>
> This is a byte string: b'\x80\x03]q\x00(K\x00K\x01e.'
> As MRAB points out, you can unpickle a byte string directly.
>
> This is a doc string: """note the triplet of double quotes"""
> What you have is a doc string that appears to contain a byte string:
> """b'\x80\x03]q\x00(K\x00K\x01e.'"""
>
> So the question for you is: what is putting the byte string inside of
> a doc string? If you can stop that from happening, then you'll have a
> byte string you can directly unpickle.
>
> Now, if you _don't_ have control over whatever is handing you the dump
> string, then you can just use string manipulation to reproduce the
> byte string:
>
 dump = """b'\x80\x03]q\x00(K\x00K\x01e.'"""
 badump = dump[2:-1].encode()[1:]
 pickle.loads(badump)
> [0, 1]
>
> So:
>  - dump[2:-1] strips off string representation of the byte string
> (b'...')
>  - .encode() turns it into an actual byte string
>  - [1:] strips a unicode blank from the start of the byte string (not
> entirely sure how that gets there...)
>
> After that it should be fine to unpickle.
>
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Speed-up for loops

[Michael Kreim]

Peter Otten wrote:

Move it into a function; this turns a and i into local variables.

def f():
imax = 10
a = 0
for i in xrange(imax):
a = a + 10
print a
f()


Wow. It is still slower than Matlab, but your suggestion speeds up the 
code by ca 50%.
But I do not understand why the change of a global to a local variable 
gives such a big difference.



$ cat addition.py
imax = 10
a = 0
for i in xrange(imax):
a = a + 10
print a

$ cat additionOtten.py
def f():
imax = 10
a = 0
for i in xrange(imax):
a = a + 10
print a
f()

$ /usr/bin/time --verbose python addition.py
100
Command being timed: "python addition.py"
User time (seconds): 110.52
System time (seconds): 0.01
Percent of CPU this job got: 98%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:52.76
[...]

$ /usr/bin/time --verbose python additionOtten.py
100
Command being timed: "python additionOtten.py"
User time (seconds): 56.38
System time (seconds): 0.00
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:56.64
[...]
--
http://mail.python.org/mailman/listinfo/python-list

Re: Speed-up for loops

[Peter Otten]

Michael Kreim wrote:

> Peter Otten wrote:
>> Move it into a function; this turns a and i into local variables.
>> 
>> def f():
>> imax = 10
>> a = 0
>> for i in xrange(imax):
>> a = a + 10
>> print a
>> f()
> 
> Wow. It is still slower than Matlab, but your suggestion speeds up the
> code by ca 50%.
> But I do not understand why the change of a global to a local variable
> gives such a big difference.

Basically the local namespace is a C array where accessing an item is just 
pointer arithmetic while the global namespace is a Python dictionary. 

There may be optimisations for the latter. If you can read C have a look at 
the LOAD/STORE_FAST and LOAD/STORE_GLOBAL implementations for the gory 
details:

http://svn.python.org/view/python/trunk/Python/ceval.c?view=markup

Peter

-- 
http://mail.python.org/mailman/listinfo/python-list

argparse list

[Neal Becker]

I'm interested in using argparse to parse a string formatted as:

my_prog --option1=1,10,37

That is, a list of comma delimited values.  I guess nargs almost does it, 
but expects options to be space-delimited.

What would be the easiest approach?

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: argparse list

[Peter Otten]

Neal Becker wrote:

> I'm interested in using argparse to parse a string formatted as:
> 
> my_prog --option1=1,10,37
> 
> That is, a list of comma delimited values.  I guess nargs almost does it,
> but expects options to be space-delimited.
> 
> What would be the easiest approach?

>>> import argparse
>>> def csv(value):
... return map(int, value.split(","))
...
>>> p = argparse.ArgumentParser()
>>> p.add_argument("--option1", type=csv) and None
>>> p.parse_args(["--option1=1,10,37"])
Namespace(option1=[1, 10, 37])
>>> _.option1[0]
1

Peter
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: argparse list

[Neal Becker]

Peter Otten wrote:

 import argparse
 def csv(value):
> ... return map(int, value.split(","))
> ...
 p = argparse.ArgumentParser()
 p.add_argument("--option1", type=csv) and None
 p.parse_args(["--option1=1,10,37"])

Thanks!  But, why the 'and None'?

-- 
http://mail.python.org/mailman/listinfo/python-list

Safely decoding user input

[Tom Miller]

Hello everyone,

Before I pose my question, I should mention that I'm still pretty unfamiliar
with proper terminology for string encoding, so I might get some of it
wrong. Please bear with me.

I'm writing a program that accepts arguments from the command line. Some of
my users are using Windows with a non-unicode locale setting and characters
outside of the ascii set. So something like

$ program --option 

ultimately results in "UnicodeDecodeError: 'utf8' codec can't decode bytes
in position 0-3: invalid data"

My questions:
1) Is it safe to immediately decode all strings in sys.argv[] with something
like sys.argv = [string.decode(sys.stdin.encoding) for string in sys.argv]?
2) Can something similar be done to anything returned by raw_input()?

Thanks,
Tom
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: argparse list

[Peter Otten]

Neal Becker wrote:

> Peter Otten wrote:
> 
> import argparse
> def csv(value):
>> ... return map(int, value.split(","))
>> ...
> p = argparse.ArgumentParser()
> p.add_argument("--option1", type=csv) and None
> p.parse_args(["--option1=1,10,37"])
> 
> Thanks!  But, why the 'and None'?

To hide the result of p.add_argument().

>>> "irrelevant junk"
'irrelevant junk'
>>> "irrelevant junk" and None
>>>

I'm not going to do it again ;)

Peter
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: parsing string into dict

[Aleksey]

On Sep 2, 12:46 am, Tim Arnold  wrote:
> Hi,
> I have a set of strings that are *basically* comma separated, but with
> the exception that if a comma occur insides curly braces it is not a
> delimiter.  Here's an example:
>
> [code=one, caption={My Analysis for \textbf{t}, Version 1}, continued]
>
> I'd like to parse that into a dictionary (note that 'continued' gets
> the value 'true'):
> {'code':'one', 'caption':'{My Analysis for \textbf{t}, Version
> 1}','continued':'true'}
>
> I know and love pyparsing, but for this particular code I need to rely
> only on the standard library (I'm running 2.7). Here's what I've got,
> and it works. I wonder if there's a simpler way?
> thanks,
> --Tim Arnold
>
> The 'line' is like my example above but it comes in without the ending
> bracket, so I append one on the 6th line.
>


You can use regular expression (also you not need adding ending
bracket):

import re
patt = re.compile(ur'\[code=(?P\w+),\scaption=(?P\{.+\})
(?P,\scontinued)?\]?')
def parse_options(s):
g=patt.match(s).groupdict()
return {'caption' : g['CAPTION'], 'code' : g['CODE'], 'continued' :
g['CONTINUED'] and True or False}


Test is next:


>>> s=u'[code=one, caption={My Analysis for \textbf{t}, Version 1}, continued]'
>>> s1=u'[code=one, caption={My Analysis for \textbf{t}, Version 1}]'
>>> s2=u'[code=one, caption={My Analysis for \textbf{t}, Version 1}, continued'
>>> s3=u'[code=one, caption={My Analysis for \textbf{t}, Version 1}'

>>> parse_options(s)
{'caption': u'{My Analysis for \textbf{t}, Version 1}', 'code':
u'one', 'continued': True}
>>> parse_options(s1)
{'caption': u'{My Analysis for \textbf{t}, Version 1}', 'code':
u'one', 'continued': False}
>>> parse_options(s2)
{'caption': u'{My Analysis for \textbf{t}, Version 1}', 'code':
u'one', 'continued': True}
>>> parse_options(s3)
{'caption': u'{My Analysis for \textbf{t}, Version 1}', 'code':
u'one', 'continued': False}
>>>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: parsing string into dict

[Aleksey]

On Sep 2, 12:46 am, Tim Arnold  wrote:
> Hi,
> I have a set of strings that are *basically* comma separated, but with
> the exception that if a comma occur insides curly braces it is not a
> delimiter.  Here's an example:
>
> [code=one, caption={My Analysis for \textbf{t}, Version 1}, continued]
>
> I'd like to parse that into a dictionary (note that 'continued' gets
> the value 'true'):
> {'code':'one', 'caption':'{My Analysis for \textbf{t}, Version
> 1}','continued':'true'}
>
> I know and love pyparsing, but for this particular code I need to rely
> only on the standard library (I'm running 2.7). Here's what I've got,
> and it works. I wonder if there's a simpler way?
> thanks,
> --Tim Arnold
>
> The 'line' is like my example above but it comes in without the ending
> bracket, so I append one on the 6th line.
>


You can use regular expression (also you not need adding ending
bracket):

import re
patt = re.compile(ur'\[code=(?P\w+),\scaption=(?P\{.+\})
(?P,\scontinued)?\]?')
def parse_options(s):
g=patt.match(s).groupdict()
return {'caption' : g['CAPTION'], 'code' : g['CODE'], 'continued' :
g['CONTINUED'] and True or False}


Test is next:


>>> s=u'[code=one, caption={My Analysis for \textbf{t}, Version 1}, continued]'
>>> s1=u'[code=one, caption={My Analysis for \textbf{t}, Version 1}]'
>>> s2=u'[code=one, caption={My Analysis for \textbf{t}, Version 1}, continued'
>>> s3=u'[code=one, caption={My Analysis for \textbf{t}, Version 1}'

>>> parse_options(s)
{'caption': u'{My Analysis for \textbf{t}, Version 1}', 'code':
u'one', 'continued': True}
>>> parse_options(s1)
{'caption': u'{My Analysis for \textbf{t}, Version 1}', 'code':
u'one', 'continued': False}
>>> parse_options(s2)
{'caption': u'{My Analysis for \textbf{t}, Version 1}', 'code':
u'one', 'continued': True}
>>> parse_options(s3)
{'caption': u'{My Analysis for \textbf{t}, Version 1}', 'code':
u'one', 'continued': False}
>>>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Speed-up for loops

[Tim Wintle]

On Thu, 2010-09-02 at 12:02 +0200, Michael Kreim wrote:
> Hi,
> 
> I was comparing the speed of a simple loop program between Matlab and 
> Python.

> Unfortunately my Python Code was much slower and I do not understand why.

The main reason is that, under the hood, cpython does something like
this (in psudo-code)

itterator = xrange(imax)
while 1:
  next_attribute = itterator.next
  try:
i = next_attribute()
  except:
break
  a = a + 10

where C (and I'm assuming matlab) does this:

while 1:
  i = i + 1
  if (i > imax):
break
  a = a + 10

And the function call in the python is fairly expensive on it's own.
Plus it has to do all the standard interpreter stuff for memory
management and deciding when to give up the GIL etc.

> Are there any ways to speed up the for/xrange loop?

leaving it in python, no. (well, "range" is faster in 2.x, but once you
get some cache misses due to increased memory usage it's much slower)

avoiding iteration by using list comprehensions can help a lot though as
it avoids most of the function calls.

If you really need to optimise it then you can convert that module to
cython by adding a cdef, and then compile it:

cdef int i
for i in xrange(imax):
 a = a + 10
print a

or you can write it in C it'll run a lot faster.


-- 
http://mail.python.org/mailman/listinfo/python-list

profiling qt programs

[Duim]

I'm trying to get a qt program a little faster by looking at the most
expensive functions.

To find out which functions are most important I wanted to profile the
application using cProfile module.

Unfortunately this runs through the complete code in 1 go without
waiting until all threads (or in whatever way QT is running), are
finished.

Is there a way to globally profile all running python functions?

I saw in this mail http://old.nabble.com/cProfile-and-threads-td29458757.html
mention of a profile.enable()/disable() function. Or is this "wishful
thinking/mailing"?

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Speed-up for loops

[Stefan Behnel]

Tim Wintle, 02.09.2010 14:55:

If you really need to optimise it then you can convert that module to
cython by adding a cdef, and then compile it:

cdef int i
for i in xrange(imax):
  a = a + 10
print a

or you can write it in C it'll run a lot faster.


Just to get the context right here: a C implementation won't run even a tad 
faster than the obvious Cython version, but both will run "a lot faster" 
than the Python version.


Plus, if Cython knows that the imax value is small enough, it'll infer 
"int" for the "i" variable automatically, so you won't need the "cdef" 
annotation. It won't automatically do that for "a", though, as that might 
break Python's unlimited integer semantics if "imax" and/or "a" are large 
enough.


Stefan

--
http://mail.python.org/mailman/listinfo/python-list

first non-null element in a list, otherwise None

[wheres pythonmonks]

This should be trivial:


I am looking to extract the first non-None element in a list, and
"None" otherwise.  Here's one implementation:

>>> x = reduce(lambda x,y: x or y, [None,None,1,None,2,None], None)
>>> print x
1

I thought maybe a generator expression would be better, to prevent
iterating over the whole list:

>>> x = ( x for x in [None,1,2] if x is not None ).next()
>>> print x
1

However, the generator expression throws if the list is entirely None.

With list comprehensions, a solution is:

>>> x = ([ x for x in [None,1,2] if x is not None ] + [ None ] )[0]

But this can be expensive memory wise.  Is there a way to concatenate
generator expressions?

More importantly,

Is there a better way?  (In one line?)

Thanks,

W
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: first non-null element in a list, otherwise None

[Peter Otten]

wheres pythonmonks wrote:

> I am looking to extract the first non-None element in a list, and
> "None" otherwise.  Here's one implementation:
> 
 x = reduce(lambda x,y: x or y, [None,None,1,None,2,None], None)
 print x
> 1
> 
> I thought maybe a generator expression would be better, to prevent
> iterating over the whole list:
> 
 x = ( x for x in [None,1,2] if x is not None ).next()
 print x
> 1
> 
> However, the generator expression throws if the list is entirely None.

The next() builtin (new in Python 2.6) allows you to provide a default:

>>> next((item for item in [None, None, "found"] if item is not None), "no 
match")
'found'
>>> next((item for item in [None, None, None] if item is not None), "no 
match")
'no match'


> With list comprehensions, a solution is:
> 
 x = ([ x for x in [None,1,2] if x is not None ] + [ None ] )[0]
> 
> But this can be expensive memory wise.  Is there a way to concatenate
> generator expressions?

itertools.chain()

Peter
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Speed-up for loops

[Hrvoje Niksic]

Michael Kreim  writes:

> Are there any ways to speed up the for/xrange loop?
> Or do I have to live with the fact that Matlab beats Python in this
> example?

To a point, yes.  However, there are things you can do to make your
Python code go faster.  One has been pointed out by Peter.

Another is that Python treats numbers as regular heap objects, so
creating a bunch of unneeded integers by xrange slows things down
(despite allocation of integer objects being heavily optimized).  For
this reason, you may want to change xrange(10) to
itertools.repeat(None, 10).

$ python -m timeit -s 'from itertools import repeat' 'for _ in repeat(None, 
10): pass'
1000 loops, best of 3: 1.71 msec per loop
$ python -m timeit -s 'from itertools import repeat' 'for _ in xrange(10): 
pass'
100 loops, best of 3: 2.43 msec per loop
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: first non-null element in a list, otherwise None

[wheres pythonmonks]

Peter wrote:

>> But this can be expensive memory wise.  Is there a way to concatenate
>> generator expressions?
>
> itertools.chain()
>

Aha!

import itertools
>>> x = itertools.chain( (x for x in [None,None] if x is not None), [ None ] 
>>> ).next()
>>> print x
None
>>> x = itertools.chain( (x for x in [None,7] if x is not None), [ None ] 
>>> ).next()
>>> print x
7
>>>


W
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Speed-up for loops

[Roland Koebler]

Hi,

> Are there any ways to speed up the for/xrange loop?
You can use psyco.

The following example should be about 4-times as fast as your example:

import psyco
psyco.full()
def f():
imax = 10
a = 0
for i in xrange(imax):
a += 10 
print a
f()


regards,
Roland

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: argparse list

[Michele Simionato]

On Sep 2, 1:45 pm, Neal Becker  wrote:
> I'm interested in using argparse to parse a string formatted as:
>
> my_prog --option1=1,10,37
>
> That is, a list of comma delimited values.  I guess nargs almost does it,
> but expects options to be space-delimited.
>
> What would be the easiest approach?

In plac (http://pypi.python.org/pypi/plac) you would write the
following:

import plac

@plac.annotations(
option1=('option with comma separated values',
 'option',
 'o',
 lambda x: x.split(',')))
def main(option1):
print option1

if __name__ == '__main__':
plac.call(main)
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: first non-null element in a list, otherwise None

[Arnaud Delobelle]

On Sep 2, 2:48 pm, wheres pythonmonks 
wrote:
> This should be trivial:
>
> I am looking to extract the first non-None element in a list, and
> "None" otherwise.  Here's one implementation:
>
> >>> x = reduce(lambda x,y: x or y, [None,None,1,None,2,None], None)
> >>> print x
>
> 1
>
> I thought maybe a generator expression would be better, to prevent
> iterating over the whole list:
>
> >>> x = ( x for x in [None,1,2] if x is not None ).next()
> >>> print x
>
> 1
>
> However, the generator expression throws if the list is entirely None.
>
> With list comprehensions, a solution is:
>
> >>> x = ([ x for x in [None,1,2] if x is not None ] + [ None ] )[0]
>
> But this can be expensive memory wise.  Is there a way to concatenate
> generator expressions?
>
> More importantly,
>
> Is there a better way?  (In one line?)
>
> Thanks,
>
> W

Just for fun:

>>> print min([None, 2, None, None, 1], key=lambda x: x is None)
2
>>> print min([None, None, None], key=lambda x: x is None)
None

Looks clever but:
min([], key=lambda x: x is None) throws an exception.

--
Arnaud
-- 
http://mail.python.org/mailman/listinfo/python-list

Help needed with Windows Service in Python

[Ian Hobson]

Hi All,

I am attempting to create a Windows Service in Python.

I have the framework (from Mark Hammond and Andy Robinason's book) 
running - see below. It starts fine - but it will not stop. :(


net stop "Python Service"

and using the services GUI both leave the services showing it as "stopping"

I guess this means SvcStop is called but it is not enough to get it out 
of the machine.


Does anyone know why not?

Python 2.7 with win32 extensions, sunning on Windows 7.

Many thanks

Ian

the (complete) source code is
#!/usr/bin/env python
# coding=utf8
#   service.py  = testing services and Named pipes
#
import win32serviceutil
import win32service
import win32event
class PythonService(win32serviceutil.ServiceFramework):
  _svc_name_ = "Python Service"
  _svc_display_name_ = "Test Service in Python"
  def __init__(self, args):
win32serviceutil.ServiceFramework.__init__(self,args)
self.hWaitStop = win32event.CreateEvent(None,0,0,None)
  def SvcStop(self):
self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
wind32event.SetEvent(self.hWaitStop)
  def SvcDoRun(self):
win32event.WaitForSingleObject(self.hWaitStop,win32event.INFINITE)
if __name__ == '__main__':
  win32serviceutil.HandleCommandLine(PythonService)
--
http://mail.python.org/mailman/listinfo/python-list

Re: first non-null element in a list, otherwise None

[Gerard Flanagan]

wheres pythonmonks wrote:

This should be trivial:


I am looking to extract the first non-None element in a list, and
"None" otherwise.  Here's one implementation:


x = reduce(lambda x,y: x or y, [None,None,1,None,2,None], None)
print x

1

I thought maybe a generator expression would be better, to prevent
iterating over the whole list:


x = ( x for x in [None,1,2] if x is not None ).next()
print x

1

However, the generator expression throws if the list is entirely None.

With list comprehensions, a solution is:


x = ([ x for x in [None,1,2] if x is not None ] + [ None ] )[0]


But this can be expensive memory wise.  Is there a way to concatenate
generator expressions?

More importantly,

Is there a better way?  (In one line?)

Thanks,

W


itertools.dropwhile(lambda x: x is None, inlist + [0]).next() or None


--
http://mail.python.org/mailman/listinfo/python-list

Re: killing all subprocess childrens

[Nobody]

On Thu, 02 Sep 2010 13:12:07 +1000, Astan Chee wrote:

> I have a piece of code that looks like this:
> 
> import subprocess
> retcode = subprocess.call(["java","test","string"])
> print "Exited with retcode " + str(retcode)
> 
> What I'm trying to do (and wondering if its possible) is to make sure 
> that any children (and any descendants) of this process is killed when 
> the main java process is killed (or dies).
> How do I do this in windows, linux and OSX?

I don't think that you can do it reliably on any of those platforms.

Consider: A spawns B, B spawns C, C spawns D, B and C terminate. There is
no information available to tie D to A. A knows that it spawned B, and D's
PPID is C, but C no longer exists so you can't tell that B spawned C.

Process groups won't help, as subprocess.Popen() doesn't put the child
into a new process group, so all of its descendents will share the PGID of
the Python process and any children spawned from it. Even it did use a new
process group, the descendents might have different process groups (quite
likely if the initial child is a shell script, as the shell executes each
"command" in a new process group).

If you avoid the subprocess module and use os.fork(), the child can
create a new session by fork()ing again, and having the grandchild call
os.setsid(). All descendents will share this session unless they
explicitly create a new session (and you probably shouldn't be killing any
descendents which explicitly create a new session). 

Another option is if to create the child with std{in,out,err} set to a
descriptor created for that specific child. Descendents inherit their
descriptors unless explicitly changed. The same applies to the CWD, and
(by convention) the environment. On Linux, you can discover a process'
descriptors, cwd and environment via the /proc filesystem.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: python path separator

[Nobody]

On Wed, 01 Sep 2010 23:57:21 -0700, swapnil wrote:

> I could not find any documentation for variables os.path.sep and
> os.path.altsep. Although the first is pretty straightforward can
> anyone explain the purpose of the second variable? Is it even useful?

The purpose is so that you can do e.g.:

if c == os.path.sep or c == os.path.altsep:
...

On Windows, both '\' and '/' are directory separators; you shouldn't rely
upon a particular one being used, but should check for either.

-- 
http://mail.python.org/mailman/listinfo/python-list

Customizing cgitb

[D'Arcy J.M. Cain]

I have a small problem with the cgitb module.  I know that I can
basically write my own version but it seems kind of silly to rewrite
something that does 99% what I want.  Here is an excerpt from the
output of an exception.

 1520 (current_job['job_id'], job['_SELECT_']))
 1521 
 1522 elif job['_ACTION_'].startswith('queue_'):
 1523 action = job['_ACTION_'][6:]
 1524 if action == 'mod':
job = {'_PARENTDIR_': '', '__BAD_DATA__': [], '_REMOTE...orage('npq7',
'5')]), '_REMOTE_HOST_': 'unknown'} ].startswith undefined

My problem is that the "job =" line is abridging the value.  I need to
see all of the variables in the dictionary.  Is there any way I can
wrap cgitb and get the behaviour I want or do I have to write my own
method for sys.excepthook?

-- 
D'Arcy J.M. Cain  |  Democracy is three wolves
http://www.druid.net/darcy/|  and a sheep voting on
+1 416 425 1212 (DoD#0082)(eNTP)   |  what's for dinner.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Speed-up for loops

[Nobody]

On Thu, 02 Sep 2010 12:02:40 +0200, Michael Kreim wrote:

> I was comparing the speed of a simple loop program between Matlab and 
> Python.

> imax = 10
> a = 0
> for i in xrange(imax):
>  a = a + 10
> print a

> Are there any ways to speed up the for/xrange loop?

Sure; the above can be reduced to just:

print imax * 10
;)

More seriously, if you're comparing against Matlab, you should look at
NumPy. If there's a reasonably direct approach using NumPy, it will be
much quicker than a Python "for" loop (in a sense, NumPy is a library of
useful "for" loops implemented in C).

Even a fairly indirect NumPy approach is often quicker than pure Python.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Windows vs. file.read

[MRAB]

On 02/09/2010 08:49, Lawrence D'Oliveiro wrote:

In message, MRAB
wrote:


You should open the files in binary mode, not text mode, ie file(path,
"rb"). Text mode is the default. Not a problem on *nix because the line
ending is newline.


We used to pride ourselves on not having to worry about text versus binary
I/O modes on *nix, but I’m beginning to think the reality is we have to
adopt it.

To start with, it means we can automatically handle different newline
conventions with text files originating on different systems.


In Python 3 the difference is important because binary mode works with
bytes and text mode works with strings, plus the file encoding and line
endings.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Help needed with Windows Service in Python

[David]

Il Thu, 02 Sep 2010 16:22:04 +0100, Ian Hobson ha scritto:

> self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)

You may try to give a WaitHint parameter to ReportServiceStatus call,
otherwise the Service Manager will expect the service is stopped istantly.

self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING, 1000)


David
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Customizing cgitb

[Peter Otten]

D'Arcy J.M. Cain wrote:

> I have a small problem with the cgitb module.  I know that I can
> basically write my own version but it seems kind of silly to rewrite
> something that does 99% what I want.  Here is an excerpt from the
> output of an exception.
> 
>  1520 (current_job['job_id'], job['_SELECT_']))
>  1521
>  1522 elif job['_ACTION_'].startswith('queue_'):
>  1523 action = job['_ACTION_'][6:]
>  1524 if action == 'mod':
> job = {'_PARENTDIR_': '', '__BAD_DATA__': [], '_REMOTE...orage('npq7',
> '5')]), '_REMOTE_HOST_': 'unknown'} ].startswith undefined
> 
> My problem is that the "job =" line is abridging the value.  I need to
> see all of the variables in the dictionary.  Is there any way I can
> wrap cgitb and get the behaviour I want or do I have to write my own
> method for sys.excepthook?

You could try to monkey-patch pydoc:

import pydoc
def cram(text, maxlen):
return text 
pydoc.cram = cram

Peter
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Speed-up for loops

[Tim Wintle]

On Thu, 2010-09-02 at 16:13 +0200, Roland Koebler wrote:
> Hi,
> 
> > Are there any ways to speed up the for/xrange loop?
> You can use psyco.

Assuming you've got a 32-bit machine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Customizing cgitb

[D'Arcy J.M. Cain]

On Thu, 02 Sep 2010 19:02:35 +0200
Peter Otten <__pete...@web.de> wrote:
> You could try to monkey-patch pydoc:

I suppose so.  Not a comfortable solution of course.  It's bad enough
when you get too familiar with the internals of a module but even worse
when you need to get familiar with the internals of modules imported by
a module.  I'll give it a shot though.

> import pydoc
> def cram(text, maxlen):
> return text 
> pydoc.cram = cram

Or...

import pydoc
pydoc.cram = lambda text,maxlen: text

Thanks.

-- 
D'Arcy J.M. Cain  |  Democracy is three wolves
http://www.druid.net/darcy/|  and a sheep voting on
+1 416 425 1212 (DoD#0082)(eNTP)   |  what's for dinner.
-- 
http://mail.python.org/mailman/listinfo/python-list

Financial time series data

[Virgil Stokes]

 Has anyone written code or worked with Python software for downloading 
financial time series data (e.g. from Yahoo financial)? If yes,  would you 
please contact me.


--Thanks,
V. Stokes
--
http://mail.python.org/mailman/listinfo/python-list

RE: Speed-up for loops

[Philip Bloom]

Uh.  

Try:
Imax=10
a=0
i=0
While(imailto:python-list-bounces+pbloom=crystald@python.org] On Behalf Of
Nobody
Sent: Thursday, September 02, 2010 9:05 AM
To: python-list@python.org
Subject: Re: Speed-up for loops

On Thu, 02 Sep 2010 12:02:40 +0200, Michael Kreim wrote:

> I was comparing the speed of a simple loop program between Matlab and 
> Python.

> imax = 10
> a = 0
> for i in xrange(imax):
>  a = a + 10
> print a

> Are there any ways to speed up the for/xrange loop?

Sure; the above can be reduced to just:

print imax * 10
;)

More seriously, if you're comparing against Matlab, you should look at
NumPy. If there's a reasonably direct approach using NumPy, it will be
much quicker than a Python "for" loop (in a sense, NumPy is a library of
useful "for" loops implemented in C).

Even a fairly indirect NumPy approach is often quicker than pure Python.

-- 
http://mail.python.org/mailman/listinfo/python-list

__
This email has been scanned by the MessageLabs
__

__
This email has been scanned by the MessageLabs
__
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Help needed with Windows Service in Python

[ipatrol6...@yahoo.com]

Well for one, if you're writing with pywin32, you certainly don't need the 
shbang line. #! /usr/bin/env is purely a POSIX thing.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Financial time series data

[Hidura]

But what kind of data you want to download?, because the financial
time it's basicly html code and you can work very well with a parser

2010/9/2, Virgil Stokes :
>   Has anyone written code or worked with Python software for downloading
> financial time series data (e.g. from Yahoo financial)? If yes,  would you
> please contact me.
>
> --Thanks,
> V. Stokes
> --
> http://mail.python.org/mailman/listinfo/python-list
>

-- 
Enviado desde mi dispositivo móvil

Diego I. Hidalgo D.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Windows vs. file.read

[ipatrol6...@yahoo.com]

Correct in that regard. In Python 3.x, strings are by default considered UTF-8. 
Wheras ASCII isn't a problem because it's fixed-width, UTF-8 will give you a 
different character depending on the last byte value. Therefore handling any 
kind of data that is not UTF-8 will need you to open it with 'b' and uses bytes 
instead of str (literals with b'some value').

Just an FYI.
-- 
http://mail.python.org/mailman/listinfo/python-list

MySQL Problem

[Victor Subervi]

Hi;
I have this code:

print 'select * from spreadsheets s join products p on p.Item=s.Item
join productsCategories pc on p.ID=pc.ProductsID join categories c on
pc.CategoryID=c.ID where s.Client="%s" order by c.Category, c.Parent' %
(client,)
cursor.execute('select * from spreadsheets s join products p on
p.Item=s.Item join productsCategories pc on p.ID=pc.ProductsID join
categories c on pc.CategoryID=c.ID where s.Client=%s order by c.Category,
c.Parent', (client,))
tmp = cursor.fetchall()
print tmp

Now, when I run the print of the mysql command in the mysql interpreter, it
gives me the following data:

mysql> select * from spreadsheets s join products p on p.Item=s.Item join
productsCategories pc on p.ID=pc.ProductsID join categories c on
pc.CategoryID=c.ID where s.Client='Lincoln_Properties' order by c.Category,
c.Parent;
+-++---+++--+-++--+-+---+-++++---+--+
| ID  | Client | Multi | Item   | Markup | Temp | ID  |
Item   | Description  | UOM
| Cost  | ID  | ProductsID | CategoryID | ID | Category  |
Parent   |
+-++---+++--+-++--+-+---+-++++---+--+
| 163 | Lincoln_Properties | 0 | 5349513|  53.22 | NULL |  39 |
5349513| 24833RNH 24" x 33 " 8 Mic Natural Can Liners | 1000/CS
| 25.63 |  39 | 39 | 23 | 23 | Can Liners|
Bags |
| 156 | Lincoln_Properties | 0 | 5349520|  30.00 | NULL |  31 |
5349520| Can Liners Coreless 45 gal Clear | 250/CS
| 28.69 |  31 | 31 | 23 | 23 | Can Liners|
Bags |
| 161 | Lincoln_Properties | 0 | 40x48Green |  99.32 | NULL |  37 |
40x48Green | Green Can Liners | 1000/cs
| 17.56 |  37 | 37 | 23 | 23 | Can Liners|
Bags |
| 160 | Lincoln_Properties | 0 | 24x33Green |  60.04 | NULL |  36 |
24x33Green | Green Can Liners | 1000/CS
| 20.27 |  36 | 36 | 23 | 23 | Can Liners|
Bags |
| 162 | Lincoln_Properties | 0 | 5349518|  26.02 | NULL |  38 |
5349518| 28048VNR01 40" x 48" HI-D| 250/CS
| 25.63 |  38 | 38 |  1 |  1 | Facility Supplies |
BASE |
| 152 | Lincoln_Properties | 0 | 4440120|  72.44 | NULL |  35 |
4440120| 91315 Pearlescent White Lotion Gallon Soap   | 4/CS
| 17.85 |  35 | 35 | 67 | 67 | Liquid Hand   |
Soap |
| 609 | Lincoln_Properties | 0 | 2030572|   0.00 |1 | 343 |
2030572| Americo 20" Beige Floor Pad  | 5/cs
| 15.88 | 335 |343 | 49 | 49 | Mats  |
Restaurant Paper/Pla |
| 159 | Lincoln_Properties | 0 | 2028071|  20.00 | NULL |  34 |
2028071| 25025088 77 Sanisac Liner| 500/CS
| 14.88 |  34 | 34 | 34 | 34 | Tampons   | Bathroom
Paper   |
| 158 | Lincoln_Properties | 0 | 2062866|  33.31 | NULL |  33 |
2062866| 7410 1/2 Fold Seat Cover | 3000/CS
| 35.81 |  33 | 33 | 35 | 35 | Toilet Seat Cover | Bathroom
Paper   |
| 155 | Lincoln_Properties | 0 | 5380447|  30.00 | NULL |  30 |
5380447| Scottfold Foldedd Towel 1-Ply White  | 4375/CS
| 35.00 |  30 | 30 | 33 | 33 | Toilet Tissue | Bathroom
Paper   |
| 154 | Lincoln_Properties | 0 | 5207270|   7.01 | NULL |  29 |
5207270| Standard Roll Bath Tissue 2-Ply White| 80/CS
| 43.50 |  29 | 29 | 33 | 33 | Toilet Tissue | Bathroom
Paper   |
| 164 | Lincoln_Properties | 0 | 5207269|  20.00 | NULL |  28 |
5207269| 17713-00 Kleenex Cottonelle 2-Ply Toilet Tissue  | 60/CS
| 35.50 |  28 | 28 | 33 | 33 | Toilet Tissue | Bathroom
Paper   |
| 157 | Lincoln_Properties | 0 | 5217344|  17.49 | NULL |  32 |
5217344| 1900-60 Kleenex Scottfold White Towel| 2400/CS
| 26.81 |  32 | 32 | 32 | 32 | Towels| Bathroom
Paper   |
+-++---+++--+-++--+-+---+-++++---+--+
13 rows in set (0.00 sec)

You will notice there is this data therein:

| 609 | Lincoln_Properties | 0 | 2030572|   0.00 |1 

Re: Help needed with Windows Service in Python

[Ian]

 On 02/09/2010 18:03, David wrote:

Il Thu, 02 Sep 2010 16:22:04 +0100, Ian Hobson ha scritto:


self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)

You may try to give a WaitHint parameter to ReportServiceStatus call,
otherwise the Service Manager will expect the service is stopped istantly.

self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING, 1000)


David

Thanks David.

Sadly that makes no difference.

--
http://mail.python.org/mailman/listinfo/python-list

Re: Reversing a List

[Victor Subervi]

On Wed, Sep 1, 2010 at 9:26 AM, Shashwat Anand wrote:

>
>
> On Wed, Sep 1, 2010 at 6:45 PM, Matt Saxton  wrote:
>
>> On Wed, 1 Sep 2010 09:00:03 -0400
>> Victor Subervi  wrote:
>>
>> > Hi;
>> > I have this code:
>> >
>> >   cursor.execute('describe products;')
>> >   cols = [item[0] for item in cursor]
>> >   cols = cols.reverse()
>> >   cols.append('Delete')
>> >   cols = cols.reverse()
>> >
>> > Unfortunately, the list doesn't reverse. If I print cols after the first
>> > reverse(), it prints None. Please advise.
>>
>> The reverse() method modifies the list in place, but returns None, so just
>> use
>> >>> cols.reverse()
>>
>> rather than
>> >>> cols = cols.reverse()
>>
>
> Alternatively you can do \,
>
> >>>cols = reversed(cols)
>

Thanks again, all
beno
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: MySQL Problem

[Ian]

 On 02/09/2010 19:34, Victor Subervi wrote:
for some reason running the command through python *omits* this one 
data!! The only difference is that a flag in spreadsheets (Temp) is 
set to 1. Why on earth doesn't it work in python??

Some ideas to follow up.  (These are only guesses).

1) One of the enum type fields contains an invalid value (perhaps a 
value removed from the column definition).


2) The second id field (products.id?) appears to be very large. I wonder 
what would happen if it was larger than the auto-increment value?


3) A field in the one of the rows in the missing data contains bytes 
that are invalid in the character encoding you are using in python.


4) The python field type used for some column in the missing row, 
contains a value that cannot be held in the python variable assigned.


Regards

Ian



--
http://mail.python.org/mailman/listinfo/python-list

Re: Help needed with Windows Service in Python

[Edward Kozlowski]

On Sep 2, 10:22 am, Ian Hobson  wrote:
> Hi All,
>
> I am attempting to create a Windows Service in Python.
>
> I have the framework (from Mark Hammond and Andy Robinason's book)
> running - see below. It starts fine - but it will not stop. :(
>
> net stop "Python Service"
>
> and using the services GUI both leave the services showing it as "stopping"
>
> I guess this means SvcStop is called but it is not enough to get it out
> of the machine.
>
> Does anyone know why not?
>
> Python 2.7 with win32 extensions, sunning on Windows 7.
>
> Many thanks
>
> Ian
>
> the (complete) source code is
> #!/usr/bin/env python
> # coding=utf8
> #   service.py  = testing services and Named pipes
> #
> import win32serviceutil
> import win32service
> import win32event
> class PythonService(win32serviceutil.ServiceFramework):
>    _svc_name_ = "Python Service"
>    _svc_display_name_ = "Test Service in Python"
>    def __init__(self, args):
>      win32serviceutil.ServiceFramework.__init__(self,args)
>      self.hWaitStop = win32event.CreateEvent(None,0,0,None)
>    def SvcStop(self):
>      self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
>      wind32event.SetEvent(self.hWaitStop)
>    def SvcDoRun(self):
>      win32event.WaitForSingleObject(self.hWaitStop,win32event.INFINITE)
> if __name__ == '__main__':
>    win32serviceutil.HandleCommandLine(PythonService)

Looks to me like there may be a typo in your code.

You probably meant win32event.SetEvent(self.hWaitStop), not
wind32event.

Regards,
-Edward Kozlowski
-- 
http://mail.python.org/mailman/listinfo/python-list

RE: Speed-up for loops

[Peter Otten]

Philip Bloom wrote:

> Uh.
 
> Try:
> Imax=10
> a=0
> i=0
> While(i a= a+10
>   i=i+1
> print a
 
> I suspect you will find it is way faster than using range or xrange for
> large numbers and map far more closely in the final result to what you
> are doing on matlab's side.  At least last I checked, xrange and range
> both involve iterating through an array, which is much slower in all
> cases than just doing an int vs int compare (which is what your matlab
> is doing).

How did you check? 

$ python -m timeit "for i in xrange(100): pass"
10 loops, best of 3: 47.5 msec per loop
$ python -m timeit "i = 0" "while i < 100: i += 1"
10 loops, best of 3: 152 msec per loop
$

So an empty while loop takes about three times as long as the equivalent for 
loop. Also:

"""
class xrange(object)
 |  xrange([start,] stop[, step]) -> xrange object
 |
 |  Like range(), but instead of returning a list, returns an object that
 |  generates the numbers in the range on demand.  For looping, this is
 |  slightly faster than range() and more memory efficient.
"""

Peter
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Windows vs. file.read

[Thomas Jollans]

On Thursday 02 September 2010, it occurred to ipatrol6...@yahoo.com to 
exclaim:
> Correct in that regard. In Python 3.x, strings are by default considered
> UTF-8. Wheras ASCII isn't a problem because it's fixed-width, UTF-8 will
> give you a different character depending on the last byte value. Therefore
> handling any kind of data that is not UTF-8 will need you to open it with
> 'b' and uses bytes instead of str (literals with b'some value').

Except if it's text of a known encoding, in which case you just open it with 
that encoding.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Help needed with Windows Service in Python

[Ian]

 On 02/09/2010 20:06, Edward Kozlowski wrote:

On Sep 2, 10:22 am, Ian Hobson  wrote:

Hi All,

I am attempting to create a Windows Service in Python.

I have the framework (from Mark Hammond and Andy Robinason's book)
running - see below. It starts fine - but it will not stop. :(

net stop "Python Service"

and using the services GUI both leave the services showing it as "stopping"

I guess this means SvcStop is called but it is not enough to get it out
of the machine.

Does anyone know why not?

Python 2.7 with win32 extensions, sunning on Windows 7.

Many thanks

Ian

the (complete) source code is
#!/usr/bin/env python
# coding=utf8
#   service.py  = testing services and Named pipes
#
import win32serviceutil
import win32service
import win32event
class PythonService(win32serviceutil.ServiceFramework):
_svc_name_ = "Python Service"
_svc_display_name_ = "Test Service in Python"
def __init__(self, args):
  win32serviceutil.ServiceFramework.__init__(self,args)
  self.hWaitStop = win32event.CreateEvent(None,0,0,None)
def SvcStop(self):
  self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
  wind32event.SetEvent(self.hWaitStop)
def SvcDoRun(self):
  win32event.WaitForSingleObject(self.hWaitStop,win32event.INFINITE)
if __name__ == '__main__':
win32serviceutil.HandleCommandLine(PythonService)

Looks to me like there may be a typo in your code.

You probably meant win32event.SetEvent(self.hWaitStop), not
wind32event.

Regards,
-Edward Kozlowski

A huge big thank you Edward.  That was the problem.

Regards

Ian
--
http://mail.python.org/mailman/listinfo/python-list

Re: Help needed with Windows Service in Python

[Edward Kozlowski]

On Sep 2, 2:38 pm, Ian  wrote:
>   On 02/09/2010 20:06, Edward Kozlowski wrote:
>
>
>
> > On Sep 2, 10:22 am, Ian Hobson  wrote:
> >> Hi All,
>
> >> I am attempting to create a Windows Service in Python.
>
> >> I have the framework (from Mark Hammond and Andy Robinason's book)
> >> running - see below. It starts fine - but it will not stop. :(
>
> >> net stop "Python Service"
>
> >> and using the services GUI both leave the services showing it as "stopping"
>
> >> I guess this means SvcStop is called but it is not enough to get it out
> >> of the machine.
>
> >> Does anyone know why not?
>
> >> Python 2.7 with win32 extensions, sunning on Windows 7.
>
> >> Many thanks
>
> >> Ian
>
> >> the (complete) source code is
> >> #!/usr/bin/env python
> >> # coding=utf8
> >> #   service.py  = testing services and Named pipes
> >> #
> >> import win32serviceutil
> >> import win32service
> >> import win32event
> >> class PythonService(win32serviceutil.ServiceFramework):
> >>     _svc_name_ = "Python Service"
> >>     _svc_display_name_ = "Test Service in Python"
> >>     def __init__(self, args):
> >>       win32serviceutil.ServiceFramework.__init__(self,args)
> >>       self.hWaitStop = win32event.CreateEvent(None,0,0,None)
> >>     def SvcStop(self):
> >>       self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
> >>       wind32event.SetEvent(self.hWaitStop)
> >>     def SvcDoRun(self):
> >>       win32event.WaitForSingleObject(self.hWaitStop,win32event.INFINITE)
> >> if __name__ == '__main__':
> >>     win32serviceutil.HandleCommandLine(PythonService)
> > Looks to me like there may be a typo in your code.
>
> > You probably meant win32event.SetEvent(self.hWaitStop), not
> > wind32event.
>
> > Regards,
> > -Edward Kozlowski
>
> A huge big thank you Edward.  That was the problem.
>
> Regards
>
> Ian

You're most welcome.

If you're looking at running services in Windows using Python, one
other hangup I ran into was that my services would freeze for no
reason.  At Pycon '09, I learned that there were buffers for stdout
and stderr that were filling.  I wish I could remember who gave the
talk that included the jewel of knowledge, because I'd love to give
credit where it's due...

After I redirected stdout and stderr to files, my problems with the
services freezing went away.

Regards,
-Edward Kozlowski
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: killing all subprocess childrens

[Aahz]

In article ,
Astan Chee   wrote:
>Chris Rebert wrote:
>>
>> import os
>> import psutil # http://code.google.com/p/psutil/
>>
>> # your piece of code goes here
>>
>> myself = os.getpid()
>> for proc in psutil.process_iter():
>   
>Is there a way to do this without psutil or installing any external 
>modules or doing it from python2.5?

Lightly edited code I wrote yesterday:

cmd = ['ps', '-eo', 'pid,ppid']
output = Popen(cmd, stdout=PIPE).communicate()[0]
output = output.split('\n')[1:]  # skip the header
for row in output:
if not row:
continue
child_pid, parent_pid = row.split()
if parent_pid == str(pid):
child_pid = int(child_pid)
os.kill(child_pid, signal.SIGUSR1)
-- 
Aahz (a...@pythoncraft.com)   <*> http://www.pythoncraft.com/

"...if I were on life-support, I'd rather have it run by a Gameboy than a
Windows box."  --Cliff Wells
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Speed-up for loops

[Carl Banks]

On Sep 2, 5:55 am, Tim Wintle  wrote:
> On Thu, 2010-09-02 at 12:02 +0200, Michael Kreim wrote:
> > Hi,
>
> > I was comparing the speed of a simple loop program between Matlab and
> > Python.
> > Unfortunately my Python Code was much slower and I do not understand why.
>
> The main reason is that, under the hood, cpython does something like
> this (in psudo-code)
>
> itterator = xrange(imax)
> while 1:
>   next_attribute = itterator.next
>   try:
>     i = next_attribute()
>   except:
>     break
>   a = a + 10
>
> where C (and I'm assuming matlab) does this:
>
> while 1:
>   i = i + 1
>   if (i > imax):
>     break
>   a = a + 10

Not really.  Someone already posted timings of the while-loop version
in Python and it's much slower than the for loop.  The iterator stuff
is a minor overhead.

The real reason is simple and boring: many languages optimize loops
like this, Python doesn't.

Matlab has a hundred paid engineers who's job is to optimize it, and
its focus is mathematics, so of course they're going to pull out every
stop to get simple loops like the above as fast as possible.


> And the function call in the python is fairly expensive on it's own.
> Plus it has to do all the standard interpreter stuff for memory
> management and deciding when to give up the GIL etc.

Matlab has all that stuff too (it's memory management is much, much
worse than Python's, in fact, but memory management usually doesn't
play into tight loop timings).


> > Are there any ways to speed up the for/xrange loop?
>
> leaving it in python, no. (well, "range" is faster in 2.x, but once you
> get some cache misses due to increased memory usage it's much slower)
>
> avoiding iteration by using list comprehensions can help a lot though as
> it avoids most of the function calls.

List comprehensions use iteration and don't avoid function calls
relative to equivalent for-loops.  I think the main reason they're a
little faster is they can use tighter bytecode.

> If you really need to optimise it then you can convert that module to
> cython by adding a cdef, and then compile it:
>
> cdef int i
> for i in xrange(imax):
>      a = a + 10
> print a
>
> or you can write it in C it'll run a lot faster.

numpy is terrific when you can use it, and I've found that it can do a
lot more than most people expect.  The hard part is figuring out how.

In particular, numpy will trounce Matlab's performance for large
amounts of data, because of the aforementioned memory management
problem.


Carl Banks
-- 
http://mail.python.org/mailman/listinfo/python-list

bisection method: Simulating a retirement fund

[Baba]

level: beginner

exercise source:
http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-00-introduction-to-computer-science-and-programming-fall-2008/assignments/pset4.pdf

Problem 4

Can my code be optimised?
I think my approach is correct but i am hesitant about the initial max
value.

def nestEgVariable(salary, save, preRetireGrowthRates):
SavingsRecord = []
fund = 0
depositPerYear = salary * save * 0.01
for i in preRetireGrowthRates:
fund = fund * (1 + 0.01 * i) + depositPerYear
SavingsRecord += [fund,]
startingPot = SavingsRecord[-1]
return startingPot


def
expenseCalculator(postRetireGrowthRates,fundsize,epsilon,yearsOfretirement ):
low = 0
high = fundsize
guess = (low + high) /2
while abs(guess * yearsOfretirement - fundsize) > epsilon:
if guess * yearsOfretirement > fundsize :
high = guess
else: low = guess
guess = (low + high) /2
return guess


def findMaxExpenses(salary,save,preRetireGrowthRates,
postRetireGrowthRates,epsilon):
yearsOfretirement = len(postRetireGrowthRates)
fundsize = nestEgVariable(salary, save, preRetireGrowthRates)
for growthRate in postRetireGrowthRates:
expenses = expenseCalculator(postRetireGrowthRates,
 
fundsize ,epsilon,yearsOfretirement )
fundsize  = (fundsize  * (1 + 0.01 * growthRate)) - expenses
print 'fundsize', fundsize
print 'expenses', expenses
yearsOfretirement -=1
return fundsize



findMaxExpenses(1,10,[3, 4, 5, 0, 3],[10, 5, 0, 5, 1],0.01)

thanks
Baba
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Performance: sets vs dicts.

[Terry Reedy]

On 9/1/2010 10:57 PM, ru...@yahoo.com wrote:


So while you may "think" most people rarely read
the docs for basic language features and objects
(I presume you don't mean to restrict your statement
to only sets), I and most people I know *do* read
them.  And when read them I expect them, as any good
reference documentation does, to completely and
accurately describe the behavior of the item I am
reading about.  If big-O performance is deemed an
intrinsic behavior of an (operation of) an object,
it should be described in the documentation for
that object.


However, big-O performance is intentionally NOT so deemed. And I have 
and would continue to argue that it should not be, for multiple reasons.


Performance is a feature of implementations, and I think they should be 
documented.



This is not to say that a performance HOWTO or FAQ
in addition to the reference manual would not be good.


I have writing a draft of such for CPython on my personal todo list.

--
Terry Jan Reedy

--
http://mail.python.org/mailman/listinfo/python-list

Obscure MySQLdb question - duplicating a database handle

[John Nagle]

  I have a system which does error logging to its database:

db = MySQLdb.connect(...) # get database connection
...
errorlog(db, "Message")

The problem is that I want errorlog to commit its message to
the table used for error logging, but don't want to commit
whatever the caller was doing - that may well revert.

MySQL doesn't support nested transactions, so that won't help.
At the many points errorlog is called, only the database
handle is available, not the params used for "connect".

Is there something like UNIX "dup" for database connections
in MySQLdb, or can I get the connection parameters (username,
password, database, etc.) from the "db" object?

John Nagle
--
http://mail.python.org/mailman/listinfo/python-list

Re: PyPy and RPython

[John Nagle]

On 9/2/2010 1:29 AM, sarvi wrote:

When I think about it these restrictions below seem a very reasonable
tradeoff for performance.


   Yes.


And I can use this for just the modules/sections that are performance
critical.


   Not quite.  Neither Shed Skin nor RPython let you call from
restricted code to unrestricted code.  That tends to happen
implicitly as objects are passed around.  It's the global
analysis that makes this work; when you call something, you
need to know more about it than how to call it.

John Nagle
--
http://mail.python.org/mailman/listinfo/python-list

Re: Performance: sets vs dicts.

[Terry Reedy]

On 9/1/2010 8:11 PM, John Bokma wrote:

Terry Reedy  writes:


On 9/1/2010 5:40 PM, John Bokma wrote:


[..]


Yes, I switched, because 'constant time' is a comprehensible claim
that can be refuted and because that is how some will interpret O(1)
(see below for proof;-).


You make it now sound as if this interpretation is not correct or out of
place.


Saying that an interpretation is comprehensible and common is hardly a 
putdown ;-). It simply is not unique. I also said that the other 
interpretation is not coherent for size-limited problems. However, if 
you mean 'constant', whatever you mean by that, why not say so?


Here is the technical definition given in
https://secure.wikimedia.org/wikipedia/en/wiki/Big_O_notation
I assure you that it is the standard one invented by a mathematiciam 
over a century ago and used by mathematicians ever since. It was 
popularized in computer science by Donald Knuth in 1976 and used in The 
Art of Computer Programming. It is related to the definition of limits.

'''
Formal definition

Let f(x) and g(x) be two functions defined on some subset of the real 
numbers. One writes


f(x)=O(g(x))\mbox{ as }x\to\infty\,

if and only if, for sufficiently large values of x, f(x) is at most a 
constant multiplied by g(x) in absolute value. That is, f(x) = O(g(x)) 
if and only if there exists a positive real number M and a real number 
x0 such that


|f(x)| \le \; M |g(x)|\mbox{ for all }x>x_0.
'''
For g(x) == 1, this reduces to |f(x)| < M for some M (and large x), 
which is to say, f(x) is bounded (at least for large x).


Hmmm. By that definition, 1/x is O(1). Let x0 be anything but 0 and M = 
1/x0. Some might find that disturbing. But it is the nature of the 
notation, as with limit notation, that is says *nothing* about the 
behavior of f for small x.


> People who have bothered to read ItA will use O(1) and constant

time interchangeably while talking of the order of growth of the running
time algorithms


People who have bothered to read the Bidle will use terms the way they 
are used in the Bible. So what? In other words, I do not regard ItA as 
scripture in the way you seem to. (I presume you mean the Cormen, etal 
book. I partially read a library copy over 20 years ago but never 
bothered to buy a copy. I see it is now into its third edition.) If they 
disagree with Knuth and most others (which I would not claim without 
seeing their actual text) then so much the worse for them.


> and most of those are aware that 'big oh' hides a

constant, and that in the real world a O(log n) algorithm can outperform
an O(1) algorithm for small values of n.


Right. And if 'small values of n' include all possible values, then 
rejecting a particular O(log n) algorithm as 'unacceptable' relative to 
all O(1) algorithms is pretty absurd.



Uhm, O(1) /is/ constant time, see page 45 of Introduction to Algorithms,
2nd edition.


Given that the Wikipedia article references that section also, I wonder 
if it really disagrees with the definition above.


--
Terry Jan Reedy

--
http://mail.python.org/mailman/listinfo/python-list

Sum of product-please help

[Nally Kaunda-Bukenya]

Dear all, kindly help me with this code;

This script  is supposed to calculate Rvi for each row by first summing the 
product of #fields (Ai*Rv) and dividing by another field (Tot) such that  
Rvi=sum(Ai*Rv)/Tot. First it's acting like I need another parenthesis and it 
doesn't seem to work at all. i even imported the math module, but not sure if a 
need it. Please advice, your help is highly appreciated.  Please see the code 
below:

import arcpy, math
arcpy.Workspace = "C:\\data\\basins.mdb"
fc = "wshed"
sum = 0
    
# Create the update cursor and advance the cursor to the first row
cur = arcpy.UpdateCursor(fc)
row = cur.Next()
# Perform the update and move to the next row as long as there are
#  rows left
for row:
    row.GetValue(Rvi) = sum(row.Ai*row.Rv)/row.ATot
    cur.UpdateRow(row)
    row = cur.Next()
# Delete the cursors to remove any data locks
del row, cur  


  -- 
http://mail.python.org/mailman/listinfo/python-list

Re: bisection method: Simulating a retirement fund

[MRAB]

On 02/09/2010 21:37, Baba wrote:

level: beginner

exercise source:
http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-00-introduction-to-computer-science-and-programming-fall-2008/assignments/pset4.pdf

Problem 4

Can my code be optimised?
I think my approach is correct but i am hesitant about the initial max
value.

def nestEgVariable(salary, save, preRetireGrowthRates):
 SavingsRecord = []
 fund = 0
 depositPerYear = salary * save * 0.01
 for i in preRetireGrowthRates:
 fund = fund * (1 + 0.01 * i) + depositPerYear
 SavingsRecord += [fund,]
 startingPot = SavingsRecord[-1]
 return startingPot


Why are you saving 'fund' in SavingsRecord if you're returning just the
last and discarding others? Basically you're returning the final value
of fund.


def
expenseCalculator(postRetireGrowthRates,fundsize,epsilon,yearsOfretirement ):
 low = 0
 high = fundsize
 guess = (low + high) /2
 while abs(guess * yearsOfretirement - fundsize)>  epsilon:
 if guess * yearsOfretirement>  fundsize :
 high = guess
 else: low = guess
 guess = (low + high) /2
 return guess


When performing this type of 'search' make sure that the interval (high
- low) reduces at every step. If, for example:

low + 1 == high

then:

guess == (low + (low + 1)) / 2 == (low * 2 + 1) / 2 == low

(integer division) and if the 'if' condition happens to be false then
the value of 'low' won't change for the next iteration, leading to an
infinite loop.


def findMaxExpenses(salary,save,preRetireGrowthRates,
postRetireGrowthRates,epsilon):
 yearsOfretirement = len(postRetireGrowthRates)
 fundsize = nestEgVariable(salary, save, preRetireGrowthRates)
 for growthRate in postRetireGrowthRates:
 expenses = expenseCalculator(postRetireGrowthRates,

fundsize ,epsilon,yearsOfretirement )
 fundsize  = (fundsize  * (1 + 0.01 * growthRate)) - expenses
 print 'fundsize', fundsize
 print 'expenses', expenses
 yearsOfretirement -=1
 return fundsize



findMaxExpenses(1,10,[3, 4, 5, 0, 3],[10, 5, 0, 5, 1],0.01)


--
http://mail.python.org/mailman/listinfo/python-list

Re: Selecting k smallest or largest elements from a large list in python; (benchmarking)

[Terry Reedy]

On 9/1/2010 9:08 PM, Dmitry Chichkov wrote:

Given: a large list (10,000,000) of floating point numbers;
Task: fastest python code that finds k (small, e.g. 10) smallest
items, preferably with item indexes;
Limitations: in python, using only standard libraries (numpy&  scipy
is Ok);

I've tried several methods. With N = 10,000,000, K = 10 The fastest so
far (without item indexes) was pure python implementation
nsmallest_slott_bisect (using bisect/insert). And with indexes
nargsmallest_numpy_argmin (argmin() in the numpy array k times).

Anyone up to the challenge beating my code with some clever selection
algorithm?

Current Table:
1.66864395142 mins_heapq(items, n):
0.946580886841 nsmallest_slott_bisect(items, n):
1.38014793396 nargsmallest(items, n):
10.0732769966 sorted(items)[:n]:
3.17916202545 nargsmallest_numpy_argsort(items, n):
1.31794500351 nargsmallest_numpy_argmin(items, n):
2.37499308586 nargsmallest_numpy_array_argsort(items, n):
0.524670124054 nargsmallest_numpy_array_argmin(items, n):

0.0525538921356 numpy argmin(items): 1892997
0.364673852921 min(items): 10.026786


Your problem is underspecified;-).
Detailed timing comparisons are only valid for a particular Python 
version running under a particular OS on particular hardware. So, to 
actually run a contest, you would have to specify a version and OS and 
offer to run entries on your machine, with as much else as possible 
turned off, or else enlist a neutral judge to do so. And the timing 
method should also be specified.


--
Terry Jan Reedy

--
http://mail.python.org/mailman/listinfo/python-list

Re: Sum of product-please help

[MRAB]

On 02/09/2010 23:01, Nally Kaunda-Bukenya wrote:

Dear all, kindly help me with this code;
This script is supposed to calculate Rvi for each row by first summing
the product of #fields (Ai*Rv) and dividing by another field (Tot) such
that Rvi=sum(Ai*Rv)/Tot. First it's acting like I need another
parenthesis and it doesn't seem to work at all. i even imported the math
module, but not sure if a need it. Please advice, your help is highly
appreciated. Please see the code below:
import arcpy, math
arcpy.Workspace = "C:\\data\\basins.mdb"
fc = "wshed"
sum = 0


It looks like you're using 'sum' as a variable. Bad idea. That hides
the 'sum' function which you need later.


# Create the update cursor and advance the cursor to the first row
cur = arcpy.UpdateCursor(fc)
row = cur.Next()
# Perform the update and move to the next row as long as there are
# rows left
for row:


Syntax error. It's:

for variable in iterator:
...


row.GetValue(Rvi) = sum(row.Ai*row.Rv)/row.ATot


Assign to the result of row.GetValue(Rvi)? Is that right? I don't know
arcpy, but somehow that doesn't look right to me.


cur.UpdateRow(row)
row = cur.Next()
# Delete the cursors to remove any data locks
del row, cur


--
http://mail.python.org/mailman/listinfo/python-list

Re: Speed-up for loops

[Terry Reedy]

On 9/2/2010 8:55 AM, Tim Wintle wrote:

On Thu, 2010-09-02 at 12:02 +0200, Michael Kreim wrote:

Hi,

I was comparing the speed of a simple loop program between Matlab and
Python.



Unfortunately my Python Code was much slower and I do not understand why.


The main reason is that, under the hood, cpython does something like
this (in psudo-code)

itterator = xrange(imax)
while 1:
   next_attribute = itterator.next
   try:
 i = next_attribute()
   except:
 break
   a = a + 10

where C (and I'm assuming matlab) does this:

while 1:
   i = i + 1
   if (i>  imax):
 break
   a = a + 10


Which is to say, 'for' in python is not the same as 'for' in C/matlab 
and the latter is what Michael should use in a fair comparison. 
Otherwise, apples and oranges.



--
Terry Jan Reedy

--
http://mail.python.org/mailman/listinfo/python-list

Re: killing all subprocess childrens

[Chris Rebert]

On Thu, Sep 2, 2010 at 12:58 PM, Aahz  wrote:
> In article ,
> Astan Chee   wrote:
>>Chris Rebert wrote:
>>>
>>> import os
>>> import psutil # http://code.google.com/p/psutil/
>>>
>>> # your piece of code goes here
>>>
>>> myself = os.getpid()
>>> for proc in psutil.process_iter():
>>
>>Is there a way to do this without psutil or installing any external
>>modules or doing it from python2.5?
>
> Lightly edited code I wrote yesterday:
>
>        cmd = ['ps', '-eo', 'pid,ppid']
>        output = Popen(cmd, stdout=PIPE).communicate()[0]
>        output = output.split('\n')[1:]  # skip the header
>        for row in output:
>            if not row:
>                continue
>            child_pid, parent_pid = row.split()
>            if parent_pid == str(pid):
>                child_pid = int(child_pid)
>                os.kill(child_pid, signal.SIGUSR1)

Although this doesn't meet the OP's Windows-compatibility requirement.

Cheers,
Chris
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Fibonacci: returning a selection of the series

[Baba]

On Aug 29, 7:18 pm, Alain Ketterlin 
wrote:

> In general, if you have a program that produces something just to
> remove/ignore it five lines later, you have a problem. In your case:
>
> 1) are you sure you need to append to list(*) at every iteration? When
> do you *really* need to? And...
>
> 2) your loop runs up to n (the index of the fib number), but you want to
> stop on some fib number value (not index).
>
> So, why not pass start and end to i_fib, and use them where appropriate?
>

Hi Alain

Thank you for your (pragmatic) suggestions! Based on your input i was
able to considerably optimise my approach:


def fib_range(start, end):
fib_1 = 0
fib_2 = 1
range = []
while fib_2 < end:
fib_1, fib_2 = fib_2, fib_1 + fib_2
if fib_2 >= start and fib_2 <= end:
range.append(fib_2)
return range

print fib_range(4,76)

Thanks
Baba
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Obscure MySQLdb question - duplicating a database handle

[John Bokma]

John Nagle  writes:

>   I have a system which does error logging to its database:
>
>   db = MySQLdb.connect(...) # get database connection
>   ...
>   errorlog(db, "Message")
>
> The problem is that I want errorlog to commit its message to
> the table used for error logging, but don't want to commit
> whatever the caller was doing - that may well revert.
>
> MySQL doesn't support nested transactions, so that won't help.
> At the many points errorlog is called, only the database
> handle is available, not the params used for "connect".
>
> Is there something like UNIX "dup" for database connections
> in MySQLdb, or can I get the connection parameters (username,
> password, database, etc.) from the "db" object?

Maybe I am not clear anymore (long day), but why not make 2 db
connections? It might even be better since you can give each one its own
user and hence, privileges.

-- 
John Bokma   j3b

Blog: http://johnbokma.com/Facebook: http://www.facebook.com/j.j.j.bokma
Freelance Perl & Python Development: http://castleamber.com/
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: DeprecationWarning

[cerr]

On Sep 1, 5:04 pm, Chris Rebert  wrote:
> On Wed, Sep 1, 2010 at 8:58 AM, cerr  wrote:
> > Hi There,
>
> > I would like to create an scp handle and download a file from a
> > client. I have following code:
> 
> > but what i'm getting is this and no file is downloaded...:
> > /opt/lampp/cgi-bin/attachment.py:243: DeprecationWarning:
> > BaseException.message has been deprecated as of Python 2.6
> >  chan.send('\x01'+e.message)
> > 09/01/2010 08:53:56 : Downloading P-file failed.
>
> > What does that mean and how do i resolve this?
>
> http://stackoverflow.com/questions/1272138/baseexception-message-depr...
> As the warning message says, line 243 of
> /opt/lampp/cgi-bin/attachment.py is the cause of the warning.
>
> However, that's only a warning (albeit probably about a small part of
> some error-raising code), not an error itself, so it's not the cause
> of the download failure.
> Printing out the IOError encountered would be the first step in
> debugging the download failure.
>
> Cheers,
> Chris
> --http://blog.rebertia.com

Hi Chris,

Thanks for getting back!
I get an "I/O error(2): No such file or directory" error even tho the
file i'm trying to copy at /usr/share/NovaxTSP/P0086_2003.xml is
present, an "ls /usr/share/NovaxTSP/P0086_2003.xml" on the target is
succesful... :( What am i doing wrong?
Thanks,
Ron
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Selecting k smallest or largest elements from a large list in python; (benchmarking)

[Dmitry Chichkov]

Yes, you are right of course. But it is not really a contest. And if
you could improve algorithm or implementation on "your Python version
running under your OS on your hardware" it may as well improve
performance for other people under other OS's.


On Sep 2, 3:14 pm, Terry Reedy  wrote:
> On 9/1/2010 9:08 PM, Dmitry Chichkov wrote:
>
>
> Your problem is underspecified;-).
> Detailed timing comparisons are only valid for a particular Python
> version running under a particular OS on particular hardware. So, to
> actually run a contest, you would have to specify a version and OS and
> offer to run entries on your machine, with as much else as possible
> turned off, or else enlist a neutral judge to do so. And the timing
> method should also be specified.
>
> --
> Terry Jan Reedy

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: DeprecationWarning

[Chris Rebert]

On Thu, Sep 2, 2010 at 4:19 PM, cerr  wrote:
> On Sep 1, 5:04 pm, Chris Rebert  wrote:
>> On Wed, Sep 1, 2010 at 8:58 AM, cerr  wrote:
>> > Hi There,
>>
>> > I would like to create an scp handle and download a file from a
>> > client. I have following code:
>> 
>> > but what i'm getting is this and no file is downloaded...:
>> > /opt/lampp/cgi-bin/attachment.py:243: DeprecationWarning:
>> > BaseException.message has been deprecated as of Python 2.6
>> >  chan.send('\x01'+e.message)
>> > 09/01/2010 08:53:56 : Downloading P-file failed.
>>
>> > What does that mean and how do i resolve this?
>>
>> http://stackoverflow.com/questions/1272138/baseexception-message-depr...
>> As the warning message says, line 243 of
>> /opt/lampp/cgi-bin/attachment.py is the cause of the warning.
>>
>> However, that's only a warning (albeit probably about a small part of
>> some error-raising code), not an error itself, so it's not the cause
>> of the download failure.
>> Printing out the IOError encountered would be the first step in
>> debugging the download failure.
>>
>> Cheers,
>> Chris
>
> Hi Chris,
>
> Thanks for getting back!
> I get an "I/O error(2): No such file or directory" error even tho the
> file i'm trying to copy at /usr/share/NovaxTSP/P0086_2003.xml is
> present, an "ls /usr/share/NovaxTSP/P0086_2003.xml" on the target is
> succesful... :( What am i doing wrong?

Asking on the Paramiko mailinglist might be more fruitful:
http://www.lag.net/mailman/listinfo/paramiko

Cheers,
Chris
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Financial time series data

[Virgil Stokes]

On 09/02/2010 08:15 PM, Hidura wrote:

But what kind of data you want to download?, because the financial
time it's basicly html code and you can work very well with a parser

2010/9/2, Virgil Stokes:
   

   Has anyone written code or worked with Python software for downloading
financial time series data (e.g. from Yahoo financial)? If yes,  would you
please contact me.

--Thanks,
V. Stokes
--
http://mail.python.org/mailman/listinfo/python-list

 
   
Here is a snippet of python code that I am trying to use for downloading 
financial data; but, I do not understand why it returns information from 
the second HTML page.


  import urllib2
  '''
   I am trying to read each row of the table at:
http://finance.yahoo.com/q/cp?s=^GSPC
  '''
  ticker = []
  url = 
urllib2.urlopen("http://download.finance.yahoo.com/d/quotes.csv...@%5egspc&f=sl1d1t1c1ohgv&e=.csv&h=PAGE".replace('PAGE', 
str(0)))

  data = url.read()

Note, it does get all 50 rows of the first page; but, why does it also 
get the first row of the "next" HTML page?


--V



--
http://mail.python.org/mailman/listinfo/python-list

Re: Help needed with Windows Service in Python

[Mark Hammond]

On 3/09/2010 1:22 AM, Ian Hobson wrote:

Hi All,

I am attempting to create a Windows Service in Python.

I have the framework (from Mark Hammond and Andy Robinason's book)
running - see below. It starts fine - but it will not stop. :(

net stop "Python Service"

and using the services GUI both leave the services showing it as "stopping"

I guess this means SvcStop is called but it is not enough to get it out
of the machine.

Does anyone know why not?


I expect that the Windows Event Log might have some clues, as would 
attempting to use it in "debug" mode.



def SvcStop(self):
self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
wind32event.SetEvent(self.hWaitStop)


Note the typo in the line above...

Mark
--
http://mail.python.org/mailman/listinfo/python-list

Re: Financial time series data

[MRAB]

On 03/09/2010 00:56, Virgil Stokes wrote:

On 09/02/2010 08:15 PM, Hidura wrote:

But what kind of data you want to download?, because the financial
time it's basicly html code and you can work very well with a parser

2010/9/2, Virgil Stokes:

Has anyone written code or worked with Python software for downloading
financial time series data (e.g. from Yahoo financial)? If yes, would
you
please contact me.

--Thanks,
V. Stokes
--
http://mail.python.org/mailman/listinfo/python-list


Here is a snippet of python code that I am trying to use for downloading
financial data; but, I do not understand why it returns information from
the second HTML page.

import urllib2
'''
I am trying to read each row of the table at:
http://finance.yahoo.com/q/cp?s=^GSPC
'''
ticker = []
url =
urllib2.urlopen("http://download.finance.yahoo.com/d/quotes.csv...@%5egspc&f=sl1d1t1c1ohgv&e=.csv&h=PAGE".replace('PAGE',
str(0)))
data = url.read()

Note, it does get all 50 rows of the first page; but, why does it also
get the first row of the "next" HTML page?


Did you try downloading from a browser? That also returns an extra row.

Looks like an idiosyncrasy of the site.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Re: Financial time series data

[hidura]

I've tried to see the page and the code GSPC it's wrong i has used ^DJI,  
and when you download the page code use a xml parser localize the table  
element and read it. I can't access from the browser to the next page it  
doesn't appear as a link.

El , Virgil Stokes  escribió:

On 09/02/2010 08:15 PM, Hidura wrote:




But what kind of data you want to download?, because the financial



time it's basicly html code and you can work very well with a parser





2010/9/2, Virgil stoke...@it.uu.se>:






Has anyone written code or worked with Python software for downloading



financial time series data (eg from Yahoo financial)? If yes, would you



please contact me.





--Thanks,



V. Stokes



--



http://mail.python.org/mailman/listinfo/python-list










Here is a snippet of python code that I am trying to use for downloading  
financial data; but, I do not understand why it returns information from  
the second HTML page.





import urllib2



'''



I am trying to read each row of the table at:



http://finance.yahoo.com/q/cp?s=^GSPC



'''



ticker = []


url =  
urllib2.urlopen("http://download.finance.yahoo.com/d/quotes.csv...@%5egspc&f=sl1d1t1c1ohgv&e=.csv&h=PAGE".replace('PAGE',  
str(0)))



data = url.read()




Note, it does get all 50 rows of the first page; but, why does it also  
get the first row of the "next" HTML page?





--V








-- 
http://mail.python.org/mailman/listinfo/python-list

Re: DeprecationWarning

[cerr]

On Sep 2, 4:25 pm, Chris Rebert  wrote:
> On Thu, Sep 2, 2010 at 4:19 PM, cerr  wrote:
> > On Sep 1, 5:04 pm, Chris Rebert  wrote:
> >> On Wed, Sep 1, 2010 at 8:58 AM, cerr  wrote:
> >> > Hi There,
>
> >> > I would like to create an scp handle and download a file from a
> >> > client. I have following code:
> >> 
> >> > but what i'm getting is this and no file is downloaded...:
> >> > /opt/lampp/cgi-bin/attachment.py:243: DeprecationWarning:
> >> > BaseException.message has been deprecated as of Python 2.6
> >> >  chan.send('\x01'+e.message)
> >> > 09/01/2010 08:53:56 : Downloading P-file failed.
>
> >> > What does that mean and how do i resolve this?
>
> >>http://stackoverflow.com/questions/1272138/baseexception-message-depr...
> >> As the warning message says, line 243 of
> >> /opt/lampp/cgi-bin/attachment.py is the cause of the warning.
>
> >> However, that's only a warning (albeit probably about a small part of
> >> some error-raising code), not an error itself, so it's not the cause
> >> of the download failure.
> >> Printing out the IOError encountered would be the first step in
> >> debugging the download failure.
>
> >> Cheers,
> >> Chris
>
> > Hi Chris,
>
> > Thanks for getting back!
> > I get an "I/O error(2): No such file or directory" error even tho the
> > file i'm trying to copy at /usr/share/NovaxTSP/P0086_2003.xml is
> > present, an "ls /usr/share/NovaxTSP/P0086_2003.xml" on the target is
> > succesful... :( What am i doing wrong?
>
> Asking on the Paramiko mailinglist might be more 
> fruitful:http://www.lag.net/mailman/listinfo/paramiko

Yep, thanks. I guess I try my luck there then. Thanks!
Ron
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Windows vs. file.read

[Lawrence D'Oliveiro]

In message , MRAB 
wrote:

> On 02/09/2010 08:49, Lawrence D'Oliveiro wrote:
>> In message, MRAB
>> wrote:
>>
>>> You should open the files in binary mode, not text mode, ie file(path,
>>> "rb"). Text mode is the default. Not a problem on *nix because the line
>>> ending is newline.
>>
>> We used to pride ourselves on not having to worry about text versus
>> binary I/O modes on *nix, but I’m beginning to think the reality is we
>> have to adopt it.
>>
>> To start with, it means we can automatically handle different newline
>> conventions with text files originating on different systems.
> 
> In Python 3 the difference is important because binary mode works with
> bytes and text mode works with strings, plus the file encoding and line
> endings.

Yeah, that seems like a reasonable approach—make you decide up-front what 
exactly is the sort of file content you’re dealing with.
-- 
http://mail.python.org/mailman/listinfo/python-list

Does MySQLdb rollback on control-C? Maybe not.

[John Nagle]

   I would expect MySQLdb to rollback on a control-C, but it doesn't
seem to have done so.  I have a program which does a thousand or
so INSERT operations in one transaction on an InnoDB table.

   I kill it with a control-C on Windows, and it aborts.  But
it looks like some of the INSERT operations took place.  Here's
the abort, from the middle of an INSERT operation.

Filing index item already in database under another date: 
edgar/data/1455650/950123-10-062814.txt

Traceback (most recent call last):
   
  File "C:\projects\sitetruth\edgar\edgarfilingindex.py", line 93, in 
dbinsert

cursor.execute(sql, values) # do the insert
  File "C:\python26\lib\site-packages\MySQLdb\cursors.py", line 166, in 
execute

self.errorhandler(self, exc, value)
  File "C:\python26\lib\site-packages\MySQLdb\connections.py", line 35, 
in defaulterrorhandler

raise errorclass, errorvalue
KeyboardInterrupt
Terminate batch job (Y/N)? y

   Checking with the manual MySQL client, looking at the table
before and after the run, the table did change when the program ran.
Not good.

   The INSERT looks like this:

INSERT INTO edgarfilingindex
(cik, company_name, form_type,
date_filed, file_name, date_indexed, confidence)
VALUES (%s,%s,%s,%s,%s,%s,%s)

   SHOW CREATE TABLE returns this:

CREATE TABLE `edgarfilingindex` (
  ...
) ENGINE=InnoDB DEFAULT CHARSET=utf8

so it's definitely an InnoDB table.

I have print statements at all commit calls, and there don't seem to
be any unwanted commits happening.  I'm not doing anything that
forces a commit, like RENAME or ALTER or CREATE; it's just
simple INSERT operations.  I'm not doing an explicit START
TRANSACTION, but I shouldn't have to.

I put a SHOW VARIABLES LIKE "autocommit" in the program, and
it comes back with ("autocommit", "OFF").  So that's not the problem.

I even did an explicit START TRANSACTION at the beginning of the
update, and it doesn't help.

Something is broken.

Python 2.6, MySQLdb 1.2.2-py2.6, MySQL 5.1.47-com, Windows 7.

John Nagle
--
http://mail.python.org/mailman/listinfo/python-list

Re: PyPy and RPython

[sarvi]

On Sep 2, 2:19 pm, John Nagle  wrote:
> On 9/2/2010 1:29 AM, sarvi wrote:
>
> > When I think about it these restrictions below seem a very reasonable
> > tradeoff for performance.
>
>     Yes.
>
> > And I can use this for just the modules/sections that are performance
> > critical.
>
>     Not quite.  Neither Shed Skin nor RPython let you call from
> restricted code to unrestricted code.  That tends to happen
> implicitly as objects are passed around.  It's the global
> analysis that makes this work; when you call something, you
> need to know more about it than how to call it.

It should technically be possible to allow Python to call a module
written in RPython?
It should also compile RPython to a python module.so right?

Sarvi
>
>                                 John Nagle

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Performance: sets vs dicts.

[Arnaud Delobelle]

Terry Reedy  writes:

> On 9/1/2010 8:11 PM, John Bokma wrote:
[...]
 Uhm, O(1) /is/ constant time, see page 45 of Introduction to Algorithms,
 2nd edition.
>
> Given that the Wikipedia article references that section also, I
> wonder if it really disagrees with the definition above.

In simple terms, O(1) is *bounded* time.

This is not the same as constant, even though it is true that the
misleading expression "constant time" is widely used.  I emphasized the
difference as you seemed to say that describing list element access as
O(1) was misleading because for example accessing contiguous elements
would be significantly faster that accessing distant ones.  This may or
may not be the case, but it remains true that this time is bounded and
this is what O(1) really means.

-- 
Arnaud
-- 
http://mail.python.org/mailman/listinfo/python-list

python database

[shai garcia]

can you pls help me to make a database program in python? It is a Quiz system 
which is database driven. The program has a choices which add question, edit, 
delete, list, take a quiz, quiz results, and exit. in take a quiz 
choice,questions should be randomly displayed and the result and name of the 
examinee should be saved as well as the date and time taken. pls 
help..tnx.Godbless

Submitted via EggHeadCafe - Software Developer Portal of Choice 
Book Review: Excel 2010 - The Missing Manual [OReilly]
http://www.eggheadcafe.com/tutorials/aspnet/7d211741-221d-46c7-b9c3-d34bf84568be/book-review-excel-2010--the-missing-manual-oreilly.aspx
-- 
http://mail.python.org/mailman/listinfo/python-list

python database

[shai garcia]

can you pls help me to make a database program in python? It is a Quiz system 
which is database driven. The program has a choices which add question, edit, 
delete, list, take a quiz, quiz results, and exit. in take a quiz 
choice,questions should be randomly displayed and the result and name of the 
examinee should be saved as well as the date and time taken. pls 
help..tnx.Godbless

Submitted via EggHeadCafe - Software Developer Portal of Choice 
Kentico CMS for ASP.NET Sites
http://www.eggheadcafe.com/tutorials/aspnet/ee551a85-2206-446e-bc7d-c978f60ec671/kentico-cms-for-aspnet-sites.aspx
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Obscure MySQLdb question - duplicating a database handle

[Lawrence D'Oliveiro]

In message <4c801218$0$1622$742ec...@news.sonic.net>, John Nagle wrote:

> The problem is that I want errorlog to commit its message to
> the table used for error logging, but don't want to commit
> whatever the caller was doing - that may well revert.

Any particular reason you’re using a database table for logging? It’s 
usually easier to write to text files for this purpose.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: killing all subprocess childrens

[Lawrence D'Oliveiro]

In message , Astan Chee 
wrote:

> What I'm trying to do (and wondering if its possible) is to make sure
> that any children (and any descendants) of this process is killed when
> the main java process is killed (or dies).
> How do I do this in windows, linux and OSX?

A linux-specific solution could be implemented in terms of cgroups 
. I can’t see any 
way to automatically ensure all descendants are killed, but the parent can 
at least scan the cgroup when the immediate child exits, and kill every 
leftover process it finds.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Speed-up for loops

[Ulrich Eckhardt]

Tim Wintle wrote:
> [..] under the hood, cpython does something like this (in psudo-code)
> 
> itterator = xrange(imax)
> while 1:
>   next_attribute = itterator.next
>   try:
> i = next_attribute()
>   except:
> break
>   a = a + 10

There is one thing that strikes me here: The code claims that each iteration
there is a lookup of the 'next' field in the iterator. I would expect that
this is looked up once before the loop only.

Can you confirm that or am I misinterpreting your intention here?

Uli

-- 
Sator Laser GmbH
Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932

-- 
http://mail.python.org/mailman/listinfo/python-list