Need feedback on subprocess-using function
I'm relatively new to Python, and I'm trying to get the hang of
using Python's subprocess module. As an exercise, I wrote the Tac
class below, which can prints output to a file "in reverse order",
by piping it through the Unix tac utility. (The idea is to delegate
the problem of managing the memory for an arbitrarily large task
to tac.)
class Tac(object):
def __init__(self, path):
out = open(path, 'w')
self.pipe = subprocess.Popen(['tac'], stdout=out,
stdin=subprocess.PIPE,
stderr=subprocess.PIPE)
def prn(self, string):
try:
self.pipe.stdin.write('%s\n' % string)
except:
self.close()
raise
def close(self):
p = self.pipe
p.stdin.close()
err = p.stderr.read()
if err:
raise OSError(err)
This works OK, as far as I can tell, but I'm not sure that I've
dotted all the i's and crossed all the t's... E.g., I had to add
the line "p.stdin.close()" to the close method when I when I ran
into sporadic deadlock at the p.stderr.read() statement. Are there
other similar problems lurking in this code? Also, there's no
robust mechanism for invoking this close method in case of an
exception (unless the exception happens during the execution of
prn).
Any comments and suggestions would be greatly appreciated.
G
--
http://mail.python.org/mailman/listinfo/python-list
Re: Need feedback on subprocess-using function
In Nobody writes: >You could always lift the code from Popen._communicate(), which uses >threads for Windows and select() for POSIX. Thanks. A lot of useful advice in your replies. G. -- http://mail.python.org/mailman/listinfo/python-list
Re: rfc: a self-editing script
In <0059c2b1$0$26930$c3e8...@news.astraweb.com> Steven D'Aprano
writes:
>But if you absolutely have to write to the program file...
No, don't have to, beyond the urge to satisfy a very idiosyncratic
aesthetic imperative...
>then append your
>data to the end of the file (as a comment) and later read that, rather
>than modifying the actual code in place. That is, you fetch the
>LAST_VERSION by reading the last non-empty line in the file, something
>like this:
># Untested
>def get_last_version(filename):
>"""Retrieves the last version number from the given filename,
>taken from the last non-empty line."""
>candidate = ''
>for line in open(filename, 'r'):
>line = line.strip()
>if line and line.startswith('#'):
>candidate = line.lstrip('# \t')
># error checking goes here
>return candidate
>LAST_VERSION = get_last_version(sys.argv[0])
>...
>more code goes here
>...
># ==
># === Version number history goes here. ===
># === DO NOT insert any code after this point!!! ===
># ==
># 1.0.1
># 1.0.2a
># 1.0.2
># 1.0.5
>This has the added advantage that you can track the updates made to the
>version number.
Thanks, these are great ideas. Just the feedback I was looking for.
G.
--
http://mail.python.org/mailman/listinfo/python-list
rfc: a self-editing script
The following fragment is from a tiny maintenance script that,
among other things, edits itself, by rewriting the line that ends
with '### REPLACE'.
##
import re
import fileinput
LAST_VERSION = 'VERSION 155' ### REPLACE
service = Service(url='http://url.to.service')
if service.version_string == LAST_VERSION:
sys.exit(0)
for line in fileinput.input(sys.argv[0], inplace=True):
if re.search(r"### REPLACE$", line):
print ("LAST_VERSION = '%s' ### REPLACE" %
service.version_string)
else:
print line,
# ...and goes on to do more stuff...
##
This script is meant to run periodically (via cron), and "do more
stuff" whenever the fetched value of service.version_string differs
from what it was at the time of the script's prior invocation.
(The interval of time between such changes of value varies from
one change to the next, but it is always of the order of several
weeks.)
Hence this script needs to preserve state between invocations.
The rationale for the acrobatics with fileinput above is to make
this script completely self-contained, by circumventing the need
some external means (e.g. a second file, or a DB) of preserving
state between invocations.
Is there a better way to circumvent the requirement for an external
repository of state information?
G
--
http://mail.python.org/mailman/listinfo/python-list
Re: Most efficient way to "pre-grow" a list?
In Luis Alberto Zarrabeitia
Gomez writes:
>¿Have you ever tried to read list/matrix that you know it is not sparse, but
>you
jdon't know the size, and it may not be in order? A "grow-able" array would just
>be the right thing to use - currently I have to settle with either hacking
>together my own grow-able array, or...
>...Not hard, not worthy of a
>PEP, but certainly not so easy to dismiss.
I'm pretty new to Python, and I thought I'd give this a try:
class array(list):
"""A list that grows as needed upon assignment.
Any required padding is done with the value of the fill option.
Attempts to retrieve an index greater than the maximum assigned
index still produces an IndexError.
"""
def __init__(self, seq='', fill=None):
super(array, self).__init__(seq)
self.fill = fill
@property
def max(self):
return len(self) - 1
def __setitem__(self, index, item):
m = self.max
while m < index:
self.append(self.fill)
m += 1
super(array, self).__setitem__(index, item)
if __name__ == '__main__':
x = array(('zero',))
x[3] = 'it works!'
print x
---
['zero', None, None, 'it works!']
Looks like it works, but it seems almost too easy. Did I miss
anything? Comments welcome. (E.g. I'm not sure that '' is the
best choice of default value for the seq parameter to __init__.)
Thanks in advance,
Gabe
--
http://mail.python.org/mailman/listinfo/python-list
Need help w 'Broken pipe' handling
Hi! I'm having a hard time figuring out how to handle a Unix SIGPIPE exception cleanly. The following short script illustrates the problem: -- #!/usr/bin/python # sigpipebug.py import sys import random from signal import signal, SIGPIPE, SIGINT def handle_sigpipe(signo, frame): print >> sys.stderr, "caught SIGPIPE (%d)" % signo sys.exit(0) def handle_sigint(signo, frame): print >> sys.stderr, "caught SIGINT (%d)" % signo sys.exit(1) def my_excepthook(exc_type, exc_obj, exc_tb): print >> sys.stderr, "excepthook called" sys.exit(0) signal(SIGPIPE, handle_sigpipe) signal(SIGINT, handle_sigint) # sys.excepthook = my_excepthook while True: try: print random.random() except IOError as (_, error): if error == 'Broken pipe': print >> sys.stderr, 'EXCEPTION: %s' % error sys.exit(0) raise -- The problem shows up only occasionally; the following bash one-liner helps to see the problem (terminate with Ctrl-C): while [ 1 ]; do (./sigpipebug.py||break) | head -1; done (The pipe through head should cause the script to receive a PIPE signal.) The typical output from the above one-liner will start out as something like this: 0.666233280308 caught SIGPIPE (13) 0.554289690682 caught SIGPIPE (13) 0.438033929588 caught SIGPIPE (13) 0.969307976257 caught SIGPIPE (13) close failed in file object destructor: Error in sys.excepthook: Original exception was: 0.916260186232 caught SIGPIPE (13) 0.798590903019 caught SIGPIPE (13) 0.737496617527 ...though the length of the output preceding "close failed..." is highly variable. My problem is that I can't figure out how to get rid of this unwanted extra output: close failed in file object destructor: Error in sys.excepthook: Original exception was: I figure that the way the script is handling the SIGPIPE is somehow incorrect, but I don't see what is it that I'm doing wrong (BTW, I'm a Python noob). Is there some cleanup method I should invoke in the sigpipe handler before executing sys.exit(0)? (BTW, I'm using sys.exit in this script instead of plain exit to avoid a strange error of the form "NameError: global name 'exit' is not defined".) The same thing happens when I redefine sys.excepthook (by uncommenting the commented-out line). Also, is there a way to define the SIGPIPE handler to obviate the need to trap the IOError exception? (Without this try block, I get a traceback when the script receives the PIPE signal. This traceback is also unwanted output.) Any suggestions or comments would be appreciated! Gabe -- http://mail.python.org/mailman/listinfo/python-list
Need advice on starting a Python group
I'm hoping to get advice from anyone with prior experience setting up a Python group. A friend of mine and I have been trying to start a scientific-programming-oriented Python group in our school (of medecine and bio research), with not much success. The main problem is attendance. Even though a *ton* of people have told us that it's a great idea, that they're *very* interested, and have asked to be added to our mailing list, the attendance to our first few meeting has never been more than 5, including my friend and I. Last time just he and I showed up. The second problem is getting content. The format we'd envisioned for this group was centered around code review (though not limited to it). The idea was that at every meeting a different member would show some code. This could be for any of a number of reasons, such as, for example, 1) illustrate a cool module or technique; 2) present a scientific research problem and how they used Python to solve it, or get help solving it; 3) get general feedback (e.g. on code clarity, software usability, module architecture, etc.). But in principle just about anything is OK: e.g. a talk on favorite Python resources, or a comparison of Python with some other language, or an overview of Python gotchas would all be fair game. Also, we stressed that the talks were not expected to be polished: no need for PowerPoint slides, etc. Just project any old code onto the screen, and talk about it, or scribble stuff on the chalkboard. Still, we have a hard time finding volunteers. And even when we've had volunteers, hardly anyone shows up! Any suggestions would be appreciated. GB P.S. There's a Python Meetup we could go to, but it does not fit the bill for us: it doesn't meet often enough, it's sort of out of the way, and has practically no one doing scientific programming. -- http://mail.python.org/mailman/listinfo/python-list
UnicodeEncodeError during repr()
I'm getting a UnicodeEncodeError during a call to repr: Traceback (most recent call last): File "bug.py", line 142, in element = parser.parse(INPUT) File "bug.py", line 136, in parse ps = Parser.Parse(open(filename,'r').read(), 1) File "bug.py", line 97, in end_item r = repr(CURRENT_ENTRY) UnicodeEncodeError: 'ascii' codec can't encode character u'\u3003' in position 0: o\ rdinal not in range(128) This is what CURRENT_ENTRY.__repr__ looks like: def __repr__(self): k = SEP.join(self.k) r = SEP.join(self.r) s = SEP.join(self.s) ret = u'\t'.join((k, r, s)) print type(ret) # prints "", as expected return ret If I "inline" this CURRENT_ENTRY.__repr__ code so that the call to repr(CURRENT_ENTRY) can be bypassed altogether, then the error disappears. Therefore, it is clear from the above that the problem, whatever it is, occurs during the execution of the repr() built-in *after* it gets the value returned by CURRENT_ENTRY.__repr__. It is also clearly that repr is trying to encode something using the ascii codec, but I don't understand why it needs to encode anything. Do I need to do something especial to get repr to work strictly with unicode? Or should __repr__ *always* return bytes rather than unicode? What about __str__ ? If both of these are supposed to return bytes, then what method should I use to define the unicode representation for instances of a class? Thanks! Gabe -- http://mail.python.org/mailman/listinfo/python-list
Re: UnicodeEncodeError during repr()
In "Martin v. Loewis" writes: >> Do I need to do something especial to get repr to work strictly >> with unicode? >Yes, you need to switch to Python 3 :-) >> Or should __repr__ *always* return bytes rather than unicode? >In Python 2.x: yes. >> What about __str__ ? >Likewise. >> If both of these are supposed to return bytes, >> then what method should I use to define the unicode representation >> for instances of a class? >__unicode__. Thanks! -- http://mail.python.org/mailman/listinfo/python-list
Re: Looking for registration package
In a...@pythoncraft.com (Aahz) writes: >In article , kj wrote: >> >>I'm looking for a Python-based, small, self-contained package to >>hand out API keys, in the same spirit as Google API keys. >> >>The basic specs are simple: 1) enforce the "one key per customer" rule; >>2) be robot-proof; 3) be reasonably difficult to circumvent even for >>humans. >Define "customer". You probably cannot do better than defining it as an >e-mail address, which makes requirements 2) and 3) pretty much impossible >unless you add invite codes or something. Sorry to ask, but are you being cute here? As I wrote, what I'm looking for is something in the spirit of Google API keys. Therefore, if you understand Google's "one key per customer" rule, then you understand what I want. I realize that with enough determination, any scheme for limiting keys to one per "customer" can be circumvented, but as long as the "enough determination" threshold is "high enough" the requirement is met for practical purposes. (This paragraph applies to **any** security measure, of course.) -- http://mail.python.org/mailman/listinfo/python-list
Need feedback on ORF-extracting code
Hi everyone. I'm relatively new to Python, and could use your
feedback on the code below.
First some nomenclature. A "nucleotide" is one of A, C, G, T, or
U. (In practice, a sequence of such nucleotides never contains
both T and U, but this fact is not important in what follows.) A
"codon" is a sequence of 3 of these. A "stop codon" is any one of
UAA, UAG, UGA, TAA, TAG, or TGA. Two codons are said to be "in
frame" in a containing sequence of nucleotides if their positions
differ by a multiple of 3. An "open reading frame," or ORF, is
defined as a maximal subsequence of nucleotides whose length is a
multiple of 3, begins with either AUG or ATG, terminates right
before a stop codon in the original sequence, and contains no stop
codons that are in frame with the initial codon (AUG or ATG).
The fact that ORFs have lengths that are multiples of 3 means that
there are three possible "registers" for ORFs (depending the modulo
3 of their starting positions), and that ORFs in different registers
can overlap. I'll refer to these registers as 0, 1, and 2, because
they contain ORFs that are in frame with the codons at positions
0, 1, and 2 of the original sequence, respectively.
In the code below, routine extract_orfs takes as input a string,
assumed to be a sequence of nucleotides, and returns a tuple of
tuples, describing ORFs. These ORFs can overlap.
The helper routine _xo takes as input a string (a sequence of
nucleotides), and an offset, and returns a tuple of tuples, again
representing ORFs, but this time all in the same register, since
they are all in frame with the position passed as the second argument
to the function.
I would appreciate your comments on this code. I feel that it is
not as clear as it could be, but I don't know how to make it any
clearer.
(NOTE: I realize that, in all likelihood, publicly available Python
code already exists for this. At the moment I'm more interested
in improving my Python skills.)
Many thanks in advance!
Gabe
# BEGINNING OF CODE
import sys
import re
_start = r'A[TU]G'
_stop = r'(?:[TU]A[AG]|[TU]GA)'
_nonstop = r'(?:[CAG][TUCAG]{2}|[TU](?:[TUC][TUCAG]|[AG][TUC])|[TU]GG)'
_codon = r'(?:[TUCAG]{3})'
_orf_re = re.compile('(' + _codon + r'*?)(A[TU]G' +
_nonstop + '*)(' + _stop + ')', flags=re.I)
_lead_re = re.compile(r'[TUCAG]*?A[TU]G', flags=re.I)
def _xo(seq, pos):
"""
Helper routine that finds all the non-overlapping in-frame orfs
starting from a specific position in the input sequence.
input: a string of letters in the set 'tucagTUCAG', and a starting
position;
output: a tuple of tuples; each internal tuple consists of a
starting position, an orf, and the stop codon that
terminates it.
"""
ret = []
while True:
m = _orf_re.match(seq, pos)
if not m:
break
orf = m.group(2)
stop = m.group(3)
assert len(orf) % 3 == 0
ret.append((m.start() + len(m.group(1)), orf, stop))
pos = m.end()
return ret
def extract_orfs(seq):
"""
Extracts all (possibly overlapping) maximal open reading frames,
defined as sequences beginning with AUG (or ATG), ending with an
in-frame stop codon, and containing no in-frame stop codons
in-between.
input: a string of letters in the set 'tucagTUCAG';
output: a tuple of tuples; each internal tuple consists of a
starting position, an orf, and the stop codon that
terminates it.
"""
m = _lead_re.match(seq)
if not m:
return ()
pos = m.start()
ret = []
for i in range(min(3, len(seq))):
ret.extend(_xo(seq, pos + i))
ret.sort()
return tuple(ret)
# END OF CODE
--
http://mail.python.org/mailman/listinfo/python-list
Modules/packages by GvR?
Are there any Python-only modules or packages in the latest releases of Python 2.x or Python 3.x that were largely written by Guido van Rossum? What's the best way to find this out? I know that some modules mention the author(s) in the source code, but this does not seem to be true most of the time, as far as I can tell. I'm interested in reading this code as prime examplars of "Pythonicity". (I'm sure that many other programmers could serve as models of the Pythonic ideal, but I doubt that there would be a *less debatable* choice in this category than GvR.) Many thanks in advance, Gabe -- http://mail.python.org/mailman/listinfo/python-list
Module for Fisher's exact test?
Before I roll my own, is there a good Python module for computing the Fisher's exact test stastics on 2 x 2 contingency tables? Many thanks in advance, Gabe -- http://mail.python.org/mailman/listinfo/python-list
Unix-head needs to Windows-ize his Python script
I have a handy Python script, which takes a few command-line arguments, and accepts a few options. I developed it on Unix, with very much of a Unix-mindset. Some Windows-using colleagues have asked me to make the script "easy to use under Windows 7". I.e.: no command-line. Therefore, I want to adapt my script, with the minimum amount of work, so that it can have a double-clickable icon that brings up a small GUI to accept command-line options (including a couple of file-selectors for input and output files). I am Windows-illiterate, so I really would like to keep this as barebones as possible. Where should I look to learn more about how to do this? Thx! --G (P.S. in case it matters, it's OK to assume that Python will be installed on the Windows system; IOW, the script need not bring with it a Python interpreter and libraries.) -- http://mail.python.org/mailman/listinfo/python-list
Unix-head needs to Windows-ize his Python script (II)
In gb345 writes: >I have a handy Python script, which takes a few command-line >arguments, and accepts a few options. I developed it on Unix, with >very much of a Unix-mindset. Some Windows-using colleagues have >asked me to make the script "easy to use under Windows 7". I.e.: >no command-line. >Therefore, I want to adapt my script, with the minimum amount of >work, so that it can have a double-clickable icon that brings up >a small GUI to accept command-line options (including a couple of >file-selectors for input and output files). >I am Windows-illiterate, so I really would like to keep this as >barebones as possible. Where should I look to learn more about >how to do this? >(P.S. in case it matters, it's OK to assume that Python will be >installed on the Windows system; IOW, the script need not bring >with it a Python interpreter and libraries.) Thank you all for your comments and suggestions. Here's where I am now. 1. I first implemented a GUI using wx, but later (after reading on of your comments)... 2. ...I re-implemented the GUI using Tkinter; (FWIW: the wx version was infinitely easier to write, and looks much nicer too; I hope that wx will be added to the standard library at some point); 3. Both versions of the app work fine on Windows 7, as long as I do the following: a. run CMD b. cd to where the GUI script and my original script live c. execute either C:\Python27\python myapp_tk.py or C:\Python27\python myapp_wx.py So far so good. Still missing from my goal is a clickable app that obviates the need to do the cumbersome (for the Windows-head) steps listed under #3. One possibility would be to use a "batch file" script to do the stuff in #3, but the problem is that this script would have to know the location of the GUI script and the original script. I think it would be too much to ask the user of this little app to edit the batch script to make sure these paths are correct. As far as I can tell, the only alternative left is to plunge into py2exe. I've looked into it, and I agree that it's not for the fainthearted. I'd appreciate some clues for my setup.py. My original script (I'll call it myscript.py) imports the following standard modules: sys, os, platform, csv, re, logging, collections, optparse My tk-based GUI script (I'll call it myapp_tk.py) imports the following standard modules: sys, re, Tkinter, tkFileDialog ...plus, crucially, it imports the original script, myscript.py. For completeness I'll also mention that my wx-based GUI script imports os, re, wx, and myscript Question: how do I call the setup function (within setup.py) so that the the GUI script can find myscript.py when the app's icon is double-clicked? As I said, I much prefer the wx version of the GUI over the tk version of my app. If I'm going to the trouble of making an exe version of the app, it would make sense to do it for the wx version rather than the tk version. How would I need to modify the call to setup() so that the required wx libraries are included in the resulting executable? TIA! ~G -- http://mail.python.org/mailman/listinfo/python-list
Re: Unix-head needs to Windows-ize his Python script (II)
In Tim Golden writes: >On 22/10/2010 15:25, gb345 wrote: >> 3. Both versions of the app work fine on Windows 7, as long as >> I do the following: >>a. run CMD >>b. cd to where the GUI script and my original script live >>c. execute either >> >> C:\Python27\python myapp_tk.py >> >> or >> >> C:\Python27\python myapp_wx.py >The standard python.org associates .py & .pyw files with >the installed interpreter. If you double-click on either >of those files it should just run. If you were to rename >the .py to a .pyw it would run without a console window >showing up. Thanks for the tip! That would be great if I could get it to work. When I click on the (renamed) myapp_tk.pyw or myapp_wx.pyw a console appears and immediately disappears (it just flashes, really), too quickly for me to see what it says. (I imagine it's some error message.) The renamed files still work fine if I run them as shown above, though. I see how clicking directly on these files would obviate the need to specify the path of the interpreter, but it's still not clear to me how the interpreter would know where to look for the myscript.py module that both the GUI scripts require. ~G -- http://mail.python.org/mailman/listinfo/python-list
How to get dynamically-created fxn's source?
For a project I'm working on I need a way to retrieve the source code of dynamically generated Python functions. (These functions are implemented dynamically in order to simulate "partial application" in Python.[1]) The ultimate goal is to preserve a textual record of transformations performed on data, along with all the data (both pre- and post- transformation) itself. These transformation functions could be dynamically generated as closures, but I suspect that this would make it more difficult to extract source code that could serve as a reasonably self-contained description of the transformation (because this source code would refer to variables defined outside of the function). An alternative would be to generate the *source code* for the functions dynamically, from a template having slots (within the function's definition) that gets filled in with actual parameter values, and pass this source code to exec. In any case, the problem remains of how to extract the dynamically-generated function's source code. One possibility would be to define a Transformation wrapper class whose __init__ takes the dynamically-generated source code (a string) as argument, keeps it around as an instance attribute for future reference, and exec's it to define its __call__ method. Is this overkill/reinventing the wheel? IOW, does Python already have a built-in way to achieve this same effect? (Also, the wrapper scheme above is somewhat loose: it's relatively easy for the source code instance attribute (as described) to fall out of sync with the function that actually executes when __call__ runs. Maybe a tighter connection between the obtained source code and the code that actually executes when __call__ runs is possible.) I'm aware of the inspect module, but from reading its source code I gather that it is designed for inspecting source code that is explicitly written in files, and would not be too good at inspecting functions that are generated dynamically (i.e. not from source code explicitly given in a source file--I hope that made sense). Your comments and suggestions would be much appreciated. Many thanks in advance! G [1] For example, given a base function spam that has the signature (typeT, typeT, typeT, int, int, int) and three specific integer values X, Y, and Z, dynamically generate a new function spamXYZ with signature (typeT, typeT, typeT) such that spamXYZ(A, B, C) is identical to spam(A, B, C, X, Y, Z), for all possible values of A, B, C. -- http://mail.python.org/mailman/listinfo/python-list
Re: How to get dynamically-created fxn's source?
In Peter Otten <__pete...@web.de> writes:
>gb345 wrote:
>> For a project I'm working on I need a way to retrieve the source
>> code of dynamically generated Python functions. (These functions
>> are implemented dynamically in order to simulate "partial application"
>> in Python.[1])
>Are you aware of functools.partial?
>>>> from functools import partial
>>>> def f(a, b, c, x, y, z):
>... return a*x + b*y*y + c*z*z*z
>...
>>>> fstar = partial(f, x=1, y=2, z=3)
>>>> fstar(1, 0, 0)
>1
>>>> fstar(0, 1, 0)
>4
>>>> fstar(0, 0, 1)
>27
>>>> fstar.args, fstar.keywords
>((), {'y': 2, 'x': 1, 'z': 3})
>>>> fstar.func
>
I was not aware of functools.partial. This makes the problem a
lot easier. Thanks!
G
--
http://mail.python.org/mailman/listinfo/python-list
