I've one big (6.9 Gb) .gz file with text inside it.
zcat bigfile.gz > /dev/null does the job in 4 minutes 50 seconds
python code have been doing the same job for 25 minutes and still
doesn't finish =( the code is simpliest I could ever imagine:
def main():
fh = gzip.open(sys.argv[1])
all(fh)
I've sinlge 8-way node dedicated for executing long running tasks. To
be able to execute multiple tasks on this node it shoud spawn each
task in another process. At the same time it should accept network
connection with new tasks without blocking of client and put it on job
queue.
What is "task"
On 11 фев, 20:26, "bruce" wrote:
> hi...
>
> not sure exactly what you're looking for, but "condor" has a robust job
> scheduling architecture for dealing with grid/distributed setups over
> multiple systems..
>
> give us more information, and there might be other suggestions!
Condor, Globus or a
>
> I think parallel python will take of that for you
> (http://www.parallelpython.com/)
I've found that RPyC (http://rpyc.wikidot.com/) is quite usefull for
my task. It allows me to build RPC service which accepts ordinary
python function from client and return result in synchronous or
asynchron
> ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/2.6/
> lib/python2.6/site-packages/cx_Oracle.so, 2): Symbol not found:
> ___divdi3
You didn't link cx_Oracle.so all libs which it use. run "ldd -r
cx_Oracle.so" and you'll have an idea about all missing symbols. The
names of misse
Don't really sure, but try to define your class as new-style one.
Like
class GeoMap(object):
...
--
http://mail.python.org/mailman/listinfo/python-list
> New to python I have a large file that I need to break up into
> multiple smaller files. I need to break the large file into sections
> where there are 65535 lines and then write those sections to seperate
> files.
If your lines are variable-length, then look at itertools recipes.
from iter
I try to write setup.py which compiles C Extenstion (A). The problem
is the fact, that my C Extension depends on another C lib (B). I've
digged it setuptools sources a bit and found "libraries" option for
setuptools.setup.
Now it compile B library at build_clib stage and A Extenstion at
build_ext
I run into problem with queue from multiprocessing. Even if I
queue.qsize() != 0 queue.get() still blocks and queue.get_nowait()
raises Emtpy error.
I'm unable to cut my big part to small test case, because smaller test
case similair to my real app by design is works. In what conditions is
it poss
I stuck in new multiprocessing module (ex. processing). I dont'
understand why queue.get_nowait() never returns data, but always
raises Empty, even if it is guaranteed that queue is not empty.
I've created small test case, here it is: http://pastebin.ca/1227666
Hope someone could explain why I'm
my fault. changing "continue" to "break" solves the problem
--
http://mail.python.org/mailman/listinfo/python-list
On 26 сент, 04:20, Istvan Albert <[EMAIL PROTECTED]> wrote:
> On Sep 25, 8:40 am, "Max Ivanov" <[EMAIL PROTECTED]> wrote:
>
> > At any time in main process there are shouldn't be no more than two copies
> > of data
> > (one original data and one result).
>
> From the looks of it you are storing a
On 26 сент, 17:03, MRAB <[EMAIL PROTECTED]> wrote:
> On Sep 26, 9:52 am, redbaron <[EMAIL PROTECTED]> wrote:
>
> > On 26 ÓÅÎÔ, 04:20, Istvan Albert <[EMAIL PROTECTED]> wrote:
>
> > > On Sep 25, 8:40šam, "Max Ivanov" <[EMAIL PROTECTED]&
> When processing data in parallel you will use up as muchmemoryas
> many datasets you are processing at any given time.
Worker processes eats 2-4 times more than I pass to them.
>If you need to
> reducememoryuse then you need to start fewer processes and use some
> mechanism to distribute the wo
Hi,
I am beginner to python and i am writing a program that does a lot of
things. One of the requirements is that the program shud generate a
log file. I came across python loggging module and found it very
useful. But I have a few problems
Suppose by giving option '-v' along with the program the u
On Dec 7, 7:33 pm, Jean-Michel Pichavant
wrote:
> RedBaron wrote:
> > Hi,
> > I am beginner to python and i am writing a program that does a lot of
> > things. One of the requirements is that the program shud generate a
> > log file. I came across python logggi
Is there any way by which configParser's get() function can be made
case insensitive?
--
http://mail.python.org/mailman/listinfo/python-list
17 matches
Mail list logo
