Re: Is Forth for real?

2006-02-20 Thread fox
rickman wrote:
> The original post seems to be missing, but my answer to the title
> question is, No, Forth is not real.

Not for real, for Integer.

-- 
http://mail.python.org/mailman/listinfo/python-list


Re: argparse epilog call function?

2017-06-27 Thread Fox


what  " -h " are you even talkin bout ?




def Examples():
text = """Lots of examples"""
print(text.format())



epilog='where the heck to put a -h ?? '

epilog=Examples()


--
https://mail.python.org/mailman/listinfo/python-list


Re: Pythonic cross-platform GUI desingers ?? la Interface Builder (Re: what gui designer is everyone using)

2012-06-17 Thread Chris Fox
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 17/06/2012 03:42, Chris Angelico wrote:
> On Fri, Jun 15, 2012 at 7:47 AM, Dietmar Schwertberger 
>  wrote:
>> The point is, that if you want to promote Python as replacement 
>> for e.g. VB, Labview etc., then an easy-to-use GUI builder is
>> required. The typical GUI programs will just have an input mask,
>> a button and one or two output fields.
> 
> I want to promote Linux as a replacement for Windows. But I do not
> see that Linux needs to be able to run Internet Explorer in order
> to do that. Maybe when people move to a replacement, they need to
> learn a slightly different way of doing things; and in this case, I
> would strongly recommend the "build your UI in code" method.
> 
> ChrisA

So you use wget on linux and read the html code in a terminal? That
would seem to be a reasonable analogy.

(a different) Chris


-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk/dnQUACgkQ/UgLKfoJxI6FzgCg0zRwrwQJwwCMatEEMhYyMvPN
eGsAn111DGx2lgo7Y6vMJr0EcD+zGDHD
=GjZF
-END PGP SIGNATURE-
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Learning Python

2005-10-10 Thread Jeff Fox
Lots of links to all levels of tutorials and documentation here:
http://www.awaretek.com/plf.html

Python Podcast too!


-- 
http://mail.python.org/mailman/listinfo/python-list


Concurrent Python

2005-02-11 Thread Dominic Fox
I've created a few classes to support some concurrent programming
concepts in Python:

AsyncResult represents the state of a process currently running in a
separate thread.
MultiEvent allows listeners to wait for any one of a list of events to
be signalled.
MultiQueue allows listeners to wait for an item to be added to any one
of a list of queues.
DataflowObject can be used to block while waiting for a
single-assignment variable to be bound.

The code can be found here:

http://www.codepoetics.com/code/concurrent.py

Comments and constructive criticism welcome.

regards,
Dominic
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Concurrent Python

2005-02-14 Thread Dominic Fox
> For an alternative approach (based on using generators forming a dataflow
> component system) you might find our project interesting - the core
> concurrency stuff is packaged up separately with API docs (and trivial
> example) here: http://kamaelia.sourceforge.net/Docs/Axon.html

Would it be correct to characterise this approach as co-operative
multithreading?

I have a feeling that in Python, at least, where threads and locks are
not so cheap, this approach will scale rather better - at the cost of
making some of the thread management explicit (I used to write WIMP
programs on the Acorn Archimedes that had the same basic structure:
message loop picks up messages, does some processing then explicitly
relinquishes control so that other applications can take their turn).
The syntax is also agreeably Pythonic.

I haven't investigated Stackless in much detail, but I believe it
makes a lot of these trade-offs moot - no need to simulate coroutines
when you have the real thing...

Anyway, thanks for this - I have linked it from the CTM in other
languages page of the CTM Wiki:

http://codepoetics.com/wiki/index.php?title=Topics:CTM_in_other_languages#Concurrency_in_Python

Dominic
-- 
http://mail.python.org/mailman/listinfo/python-list


Monadic Parser Combinators in Python

2005-03-18 Thread Dominic Fox
Python: Now More Like Haskell Than Ever Before!

I've implemented a small monadic parser combinator library in Python
(based on Haskell code in a paper by Graham Hutton and Eric Meijer).

http://codepoetics.com/poetix/index.php?p=94

It enables you to write things like this:

> token = isalpha |seq| many(isdigit)
> tokens = token |sepBy| whitespace
> runParser("p123 q456 hello world", tokens)
['p123', 'q456']

It is probably entirely unsuited for any serious purpose, but might
qualify as an interesting hack...

Dominic
-- 
http://mail.python.org/mailman/listinfo/python-list


Tripoli: a Python-based triplespace implementation

2005-04-30 Thread Dominic Fox
I have been working on a Python implementation of a modified Tuple
Space (cf Linda, JavaSpaces) that contains only 3-tuples (triples),
and that has operators for copying and moving graphs of triples as
well as sets matching a given pattern. It's called Tripoli, and the
code for it can be found here:

http://www.codepoetics.com/code/tripoli

More explanation of what it is and what it's meant to do can be found
in these three blog posts:

http://codepoetics.com/poetix/index.php?p=110
http://codepoetics.com/poetix/index.php?p=111
http://codepoetics.com/poetix/index.php?p=113 (this post includes some
sample code, that may clarify intended usage somewhat)

At present, a simple XML-RPC server is used to expose a "manifold" - a
collection of triple spaces - to multiple clients. I will be
supplementing this with something beefier, and more REST-ful, in due
course - probably based on Twisted.

This is code in the very earliest stages of testing and development -
I would very much welcome comments and suggestions from any intrepid
persons who fancy having a play around with it.

Dominic
--
http://mail.python.org/mailman/listinfo/python-list


xml.sax.writer

2006-08-30 Thread Dominic Fox
Hi,

xml.sax.writer doesn't appear in the global module documentation, and
googling for it doesn't tell you all that much either. Is it actually
in the standard libraries, or is it an interloper from something else
(the libxml bindings, say) that just happens to have taken up
residence in the xml.sax namespace?

Dominic

-- 
Shall we be pure or impure? Today
we shall be very pure. It must always
be possible to contain
impurities in a pure way.
--Tarmo Uustalu and Varmo Vene
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Changing numbers into characters using dictionaries

2006-01-26 Thread Colin Fox
On Thu, 26 Jan 2006 20:35:26 +, Danny wrote:

> Hello again,
> 
> I am now trying to make something to change some "encrypted" text into 
> some plain text, here is the code I have so far:
> 
> text = '@[EMAIL PROTECTED]@[EMAIL PROTECTED]' // some text
> num = '213654' // Number
> s1 = '700'
> s2 = '770'
> s4 = '707' // it adds these later on.
> t = text.split('@') // splits the digits/blocks apart from each other
> a = {s2+num[3]:"l", s1+num[0]:"a", s4+num[5]:"w"}

Well, your num[3] is going to return '6', not '4', so your key lookup is
going to fail right there. Same with num[5], which is 4, not 5.

-- 
Colin Fox
President
CF Consulting Inc.

-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Tripoli: a Python-based triplespace implementation

2005-05-02 Thread Dominic Fox
karouri wrote:

> Interesting, but I wonder if you are aware of pylinda
> (http://www-users.cs.york.ac.uk/~aw/pylinda/) and if so, what's the
> difference?

Tripoli trades off some of the generality of PyLinda in order to
support some performance tweaks and some additional graph-related
operations.

Tripoli restricts the tuple space model implemented by PyLinda to
3-tuples, and extends it by adding copy_graph and copy_collect_graph
operations to copy and move graphs of triples from one triple space to
another.

PyLinda allows you to pattern match on types, which is really neat but
makes index-based tuple lookups impractical. Tripoli matches literals
or wildcards only, but it indexes every triple on all three components
so in theory it should be quicker at finding triples, matching blocked
requests to incoming triples and so on. Performance is fairly good
even with large-ish (1+). triple sets.

Some other respects in which I think Tripoli is different, but aren't
familiar enough with PyLinda to say for certain:

Tripoli uses callbacks to notify waiting requestors that a matching
triple has arrived. Requestors can block waiting for a callback, or
continue about their business and handle the callback only when it
occurs. Communication between the Tripoli client and server is
asynchronous: the client sends a triple request, and at some point in
the future the server may contact the client to send a response back.
The REST-ful version will also support client polling of a temporary
resource representing a pending request.

Tripoli will eventually support RDF/XML as a format for importing and
exporting triples.

Dominic
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Tripoli: a Python-based triplespace implementation

2005-05-02 Thread Dominic Fox
> Is there some interoperability requirement with non-Python apps?  If not,
> why not just use pickle or marshal?

I would imagine that much of the existing, current and useful
triple-based data out there is serialized (or serializable) in
RDF/XML. I wouldn't choose it as a serialization format arbitrarily...

Dominic

-- 
// Alas, this comparison function can't be total:
// bottom is beyond comparison. - Oleg Kiselyov
-- 
http://mail.python.org/mailman/listinfo/python-list


newbie: self.member syntax seems /really/ annoying

2007-09-12 Thread Charles Fox
I've just started playing around with Python, as a possible
replacement for a mix of C++, Matlab and Lisp.  The language looks
lovely and clean with one huge exception:  I do a lot of numerical
modeling, so I deal with objects (like neurons) described
mathematically in papers, by equations like
a_dot = -k(a-u)
In other languages, this translates nicely into code, but as far as I
can tell, Python needs the ugly:
self.a_dot = -self.k(self.a-self.u)
For large equations this is going to make my code seriously unreadable
to the point of needing to switch back to Matlab -- and it seems to go
against everything else about python's simplicity and elegance.  Am I
missing something?  Is there something like a 'with' command that lets
me set the scope, like

with self:
  .a_dot = -.k(.a-.u)

It's premature to make language suggestions as I am new to the
language, but I would have though that making a 'with self' explicit
in all methods would have been neat, so I could just write
  .a_dot = -.k(.a-.u)
which would still avoid confusion with local function variables, since
'.a' is different from 'a'.

Please help if I am missing something -- this looks like a great
language but I am going to mad trying to read numerical code full of
'self.'s breaking up the equations.

-- 
http://mail.python.org/mailman/listinfo/python-list


Re: newbie: self.member syntax seems /really/ annoying

2007-09-12 Thread Charles Fox
Thanks guys -- yeah these two stategies (short s.varname; and explicit
rescoping, a=self.a etc) are more or less what I was using.  That's
still kind of annoying though.

The s.varname approach still makes numerical code much harder to read.

I had a nasty bug with the boilerplate approach when forgetting to
reassign some of the variables back to members (self.a=a).  And that's
a lot of boilerplate you need -- I thought the python way was to
minimize redundant code?  (Ditching header files and curley brackets
was a main reason for me coming here).

I see the argument for making self explicit -- what would be wrong
with just .a instead of self.a though?  That's still explicit but much
easier to read.  (I think I've seen that somewhere else, is it C#?)

-- 
http://mail.python.org/mailman/listinfo/python-list


Re: newbie: self.member syntax seems /really/ annoying

2007-09-13 Thread Charles Fox
hmm, I guess this is the difference between numerical programming and
the rest -- sure, if I was writing a database server or something it
would be great to have thisObject.veryLongName to know what everything
is -- however when you are implementing a model from a published
paper, the variables tend to be single greek or roman letter names,
possibly with subscripts and superscripts, and it helps if the name
you see on the screen is the same as the name on the paper, as is the
case in matlab.  You want the equation on screen to look as similar to
the one on the paper as possible, especially if its going to be read
by another programmer who is familiar with the paper.

Maybe for now I will just fix up my emacs to display the world 'self'
in 10% gray...   :-)


-- 
http://mail.python.org/mailman/listinfo/python-list


Hide comments in emacs python mode?

2007-10-30 Thread Charles Fox
Hi guys,

I'm playing with Python in emacs, with python mode.

I'd like to be able to press a key to toggle the code comments on and
off -- to switch between beautiful clean Python code, and the full
text that tells me what's going in in English.

Is this currently possible?  I know there is a hide/show mode in
emacs, it would need to be set to hide (1) whole lines that start with
#, (2) parts of lines after the '#' for comments after code.  Has
anyone already written this?

-- 
http://mail.python.org/mailman/listinfo/python-list


array.shape() gives TypeError: 'tuple' object is not callable

2007-12-10 Thread Charles Fox
Hi gys -- I am looking at Numpy but getting this error when I try to
get array sizes.  I'm using Ubuntu Edgy with standard repositories and
scipy.  Any ideas?  Am I doing something wrong or is it my install of
scipy?

$ python
Python 2.4.4c1 (#2, Oct 11 2006, 21:51:02)
[GCC 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from numpy import *
>>> a=array([[1,2],[3,4]])
>>> a
array([[1, 2],
   [3, 4]])
>>> a.shape()
Traceback (most recent call last):
  File "", line 1, in ?
TypeError: 'tuple' object is not callable
>>>

thanks
charles



-- 
http://mail.python.org/mailman/listinfo/python-list


socket programming (client-server) error

2008-10-22 Thread ryan fox
i have implemented a small client server model to do file transfer
over a LAN network.

It work with some machines on the network and on others it doesnt.
when i run the server.py file in some machine then it pops up a
windows security alert.

The message is as follows:

 Do you want to keep blocking this program?
there are three options below it. 1. Keep Blocking 2. Unblock 3. Ask
Me later.

I selected the option  unblock.Even then the client and server are not
able to communicate.
I get a error saying that:-

socket.error: (10060, 'Operation timed out')

I guess its a firewall problem... How do i go abt it?
any help?
--
http://mail.python.org/mailman/listinfo/python-list


libc Sleep api performs a busy waiting

2010-03-08 Thread Joe Fox
Hi ,

 I am having a problem while using sleep function from libc , the
thread in which i am calling it is getting struck and not allowing
other threads to execute.  Here is a simple code that i am trying to
exeute

import threading
import time
import dl

def dummy1():
a=dl.open('/lib/libc.so.6')
print "in thread 1 Start"
a.call('sleep',2)
#time.sleep(2)
print "in thread 1 End"

def dummy2():
print "in thread 2 Start"
time.sleep(1)
print "in thread 2 End"
newThread1=threading.Thread(None,dummy1)
newThread2=threading.Thread(None,dummy2)
newThread1.start()
newThread2.start()

print "in main"

The out put of this program is  (In this case thread 1 even though i
am calling a sleep function its not allowing other threads to execute,
other threads execute only after the completion of first thread)

in thread 1 Start
in thread 1 End
in thread 2 Start
in main
in thread 2 End
where as if i use time.sleep instead of a.call(sleep) the out put is
(which i guess is right behaviour, because it start the threads and
suspends them because the have sleep , and continue executing the main
thread)
in thread 1 Start
in thread 2 Start
in main
in thread 2 End
in thread 1 End


Can any of you point me how to over come this.?
Thanks
Mahesh
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: libc Sleep api performs a busy waiting

2010-03-08 Thread Joe Fox
Hi,

 actually i have simplified my scenario a lot here ,

In my actual case , i have to call a C-api which blocks on  c select , in a
separate thread.

my thread is getting struck in that api , and thus blocking all the other
threads.
Can you point to something which will help me call this blocking C-api call
without my thread getting struck.


Thanks
Mahesh

On Mon, Mar 8, 2010 at 6:03 PM, Steve Holden  wrote:

> Mahesh wrote:
> > Hi,
> >
> >  I am having a problem while using sleep function from libc , the
> > thread in which i am calling it is getting struck and not allowing
> > other threads to execute.  Here is a simple code that i am trying to
> > exeute
> >
> > import threading
> > import time
> > import dl
> >
> >
> > def dummy1():
> > a=dl.open('/lib/libc.so.6')
> > print "in thread 1 Start"
> > a.call('sleep',2)
> > #time.sleep(2)
> > print "in thread 1 End"
> >
> > def dummy2():
> > print "in thread 2 Start"
> > time.sleep(1)
> > print "in thread 2 End"
> > newThread1=threading.Thread(None,dummy1)
> > newThread2=threading.Thread(None,dummy2)
> > newThread1.start()
> > newThread2.start()
> >
> > print "in main"
> >
> >
> >
> > The out put of this program is  (In this case thread 1 even though i
> > am calling a sleep function its not allowing other threads to execute,
> > other threads execute only after the completion of first thread)
> >
> > in thread 1 Start
> > in thread 1 End
> > in thread 2 Start
> > in main
> > in thread 2 End
> >
> >
> > where as if i use time.sleep instead of a.call(sleep) the out put is
> > (which i guess is right behaviour, because it start the threads and
> > suspends them because the have sleep , and continue executing the main
> > thread)
> > in thread 1 Start
> > in thread 2 Start
> > in main
> > in thread 2 End
> > in thread 1 End
> >
> >
> Why not just use the time module's sleep function? Unlike the libc code
> it releases Python's GIL (global interpreter lock), thus allowing other
> threads to run.
>
> regards
>  Steve
> --
> Steve Holden   +1 571 484 6266   +1 800 494 3119
> PyCon is coming! Atlanta, Feb 2010  http://us.pycon.org/
> Holden Web LLC http://www.holdenweb.com/
> UPCOMING EVENTS:http://holdenweb.eventbrite.com/
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Unable to access module attribute with underscores in class method, Python 3

2016-01-08 Thread Joseph Fox-Rabinovitz
> On Thu, Jan 7, 2016 at 11:14 AM, Joseph Fox-Rabinovitz 
>  wrote:
>
> Hi,
>
> I have a module attribute whose name starts with a pair of underscores. I am 
> apparently unable to access it directly in a class method (within the same 
> module, but that is not relevant as far as I can tell). The following bit of 
> code illustrates the situation:
>
> __a = 3
> class B:
> def __init__(self):
> global __a
> self.a = __a
> b = B()
>
> This results in a NameError because of name-mangling, despite the global 
> declaration:
>
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "", line 4, in __init__
> NameError: name '_B__a' is not defined
>
> Not using global does not make a difference. I posted a similar question on 
> Stack Overflow, where the only reasonable answer given was to wrap __a in a 
> container whose name is not mangled. For example, doing `self.a = 
> globals()['__a']` or manually creating a dictionary with a non-mangled name 
> and accessing that.
>
> I feel that there should be some way of accessing __a within the class 
> directly in Python 3. Clearly my expectation that global would fix the issue 
> is incorrect. I would appreciate either a solution or an explanation of what 
> is going on that would convince me that accessing a module attribute in such 
> a way should be forbidden.
>
> -Joseph Fox-Rabinovitz
>
> P.S. For reference, the Stack Overflow question is here: 
> http://stackoverflow.com/questions/34621484/how-to-access-private-variable-of-python-module-from-class

One more detail that makes me think that name mangling may be getting
greedy to the point of bugginess:

__a = 3
class B:
def __init__(self):
m = sys.modules[__name__]
self.a = m.__a
b = B()

Raises the same exception as all the other way I tried to access __a:
'module' object has no attribute '_B__a'!

   -Joseph
-- 
https://mail.python.org/mailman/listinfo/python-list


Unable to access module attribute with underscores in class method, Python 3

2016-01-08 Thread Joseph Fox-Rabinovitz
Hi,

I have a module attribute whose name starts with a pair of underscores. I
am apparently unable to access it directly in a class method (within the
same module, but that is not relevant as far as I can tell). The following
bit of code illustrates the situation:

__a = 3
class B:
def __init__(self):
global __a
self.a = __a
b = B()

This results in a NameError because of name-mangling, despite the global
declaration:

Traceback (most recent call last):
  File "", line 1, in 
  File "", line 4, in __init__
NameError: name '_B__a' is not defined

Not using global does not make a difference. I posted a similar question on
Stack Overflow, where the only reasonable answer given was to wrap __a in a
container whose name is not mangled. For example, doing `self.a =
globals()['__a']` or manually creating a dictionary with a non-mangled name
and accessing that.

I feel that there should be some way of accessing __a within the class
directly in Python 3. Clearly my expectation that global would fix the
issue is incorrect. I would appreciate either a solution or an explanation
of what is going on that would convince me that accessing a module
attribute in such a way should be forbidden.

-Joseph Fox-Rabinovitz

P.S. For reference, the Stack Overflow question is here:
http://stackoverflow.com/questions/34621484/how-to-access-private-variable-of-python-module-from-class
-- 
https://mail.python.org/mailman/listinfo/python-list


PDB how to define a global inspection function?

2011-02-08 Thread Charles Fox (Sheffield)
Hi guys, I'm new to this group and have a question about debugging.
I'm stepping through my code (using emacs pdbtrack and python-mode.el)
and would like to isnpect objects as I go.  So I've defined a little
object print function,

def p(obj):
print obj
print obj.__class__
d=dir(obj)
for a in d:
print a, "=", getattr(obj, a)


however it only works if the function is imported by whatever module I
am currently debugging.  I'd like it to be available globally so I can
stop and inspect anything as I step through various modules (including
external libraries).  Is there a way to put it in the global scope for
pdb to use?   Also is there a way to automatically import it whenever
pdb starts up (like a matlab startup file)? (I'm not using ipython
as it's not happy with pdbtrack in emacs, so am launching from emacs M-
x pdb command).

thanks,
Charles Fox
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: PDB how to define a global inspection function?

2011-02-10 Thread Charles Fox (Sheffield)



On Feb 8, 11:37 am, Peter Otten <__pete...@web.de> wrote:
> CharlesFox(Sheffield) wrote:
> > Hi guys, I'm new to this group and have a question about debugging.
> > I'm stepping through my code (using emacs pdbtrack and python-mode.el)
> > and would like to isnpect objects as I go.  So I've defined a little
> > object print function,
>
> > def p(obj):
> >     print obj
> >     print obj.__class__
> >     d=dir(obj)
> >     for a in d:
> >         print a, "=", getattr(obj, a)
>
> > however it only works if the function is imported by whatever module I
> > am currently debugging.  I'd like it to be available globally so I can
> > stop and inspect anything as I step through various modules (including
> > external libraries).  Is there a way to put it in the global scope for
> > pdb to use?   Also is there a way to automatically import it whenever
> > pdb starts up (like a matlab startup file)?     (I'm not using ipython
> > as it's not happy with pdbtrack in emacs, so am launching from emacs M-
> > x pdb command).
>
> For debugging purposes it's OK to put your function into the __builtin__
> namespace:
>
>
>
> >>> def p(): print 42
> ...
> >>> import __builtin__
> >>> __builtin__.p = p
>
> Try it out:
>
> >>> with open("tmp.py", "w") as f:
>
> ...     f.write("p()\n")
> ...>>> import tmp
>
> 42


Thanks very much for your help, Peter, that's exactly what I was
after :-)
Charles
-- 
http://mail.python.org/mailman/listinfo/python-list


Shared memory python between two separate shell-launched processes

2011-02-10 Thread Charles Fox (Sheffield)
Hi guys,
I'm working on debugging a large python simulation which begins by
preloading a huge cache of data.  I want to step through code on many
runs to do the debugging.   Problem is that it takes 20 seconds to
load the cache at each launch.  (Cache is a dict in a 200Mb cPickle
binary file).

So speed up the compile-test cycle I'm thinking about running a
completely separate process (not a fork, but a processed launched form
a different terminal) that can load the cache once then dunk it in an
area of shareed memory.Each time I debug the main program, it can
start up quickly and read from the shared memory instead of loading
the cache itself.

But when I look at posix_ipc and POSH it looks like you have to fork
the second process from the first one, rather than access the shared
memory though a key ID as in standard C unix shared memory.  Am I
missing something?   Are there any other ways to do this?

thanks,
Charles
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Shared memory python between two separate shell-launched processes

2011-02-10 Thread Charles Fox (Sheffield)
On Feb 10, 3:43 pm, Jean-Paul Calderone 
wrote:
> On Feb 10, 9:30 am, "Charles Fox (Sheffield)" 
> wrote:
>
> > Hi guys,
> > I'm working on debugging a large python simulation which begins by
> > preloading a huge cache of data.  I want to step through code on many
> > runs to do the debugging.   Problem is that it takes 20 seconds to
> > load the cache at each launch.  (Cache is a dict in a 200Mb cPickle
> > binary file).
>
> > So speed up the compile-test cycle I'm thinking about running a
> > completely separate process (not a fork, but a processed launched form
> > a different terminal)
>
> Why _not_ fork?  Load up your data, then go into a loop forking and
> loading/
> running the rest of your code in the child.  This should be really
> easy to
> implement compared to doing something with shared memory, and solves
> the
> problem you're trying to solve of long startup time just as well.  It
> also
> protects you from possible bugs where the data gets corrupted by the
> code
> that operates on it, since there's only one copy shared amongst all
> your
> tests.  Is there some other benefit that the shared memory approach
> gives
> you?
>
> Of course, adding unit tests that exercise your code on a smaller data
> set
> might also be a way to speed up development.
>
> Jean-Paul



Thanks Jean-Paul, I'll have a think about this.  I'm not sure if it
will get me exactly what I want though, as I would need to keep
unloading my development module and reloading it, all within the
forked process, and I don't see how my debugger (and emacs pdb
tracking) will keep up with that to let me step though the code.
(this debugging is more about integration issues than single
functions, I have a bunch of unit tests for the little bits but
something is unhappy when I put them all together...)

(I also had a reply by email, suggesting I use /dev/shm to store the
data instead of the hard disc; this speeds things up a little but not
much as the data still has to be transferred in bulk into my
process.   Unless I'm missing something and my process can just access
the data in that shm without having to load its own copy?)
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Shared memory python between two separate shell-launched processes

2011-02-11 Thread Charles Fox (Sheffield)
On Feb 10, 6:22 pm, Jean-Paul Calderone 
wrote:
> On Feb 10, 12:21 pm, "Charles Fox (Sheffield)" 
> wrote:
>
>
>
> > On Feb 10, 3:43 pm, Jean-Paul Calderone 
> > wrote:
>
> > > On Feb 10, 9:30 am, "Charles Fox (Sheffield)" 
> > > wrote:
>
> > > > Hi guys,
> > > > I'm working on debugging a large python simulation which begins by
> > > > preloading a huge cache of data.  I want to step through code on many
> > > > runs to do the debugging.   Problem is that it takes 20 seconds to
> > > > load the cache at each launch.  (Cache is a dict in a 200Mb cPickle
> > > > binary file).
>
> > > > So speed up the compile-test cycle I'm thinking about running a
> > > > completely separate process (not a fork, but a processed launched form
> > > > a different terminal)
>
> > > Why _not_ fork?  Load up your data, then go into a loop forking and
> > > loading/
> > > running the rest of your code in the child.  This should be really
> > > easy to
> > > implement compared to doing something with shared memory, and solves
> > > the
> > > problem you're trying to solve of long startup time just as well.  It
> > > also
> > > protects you from possible bugs where the data gets corrupted by the
> > > code
> > > that operates on it, since there's only one copy shared amongst all
> > > your
> > > tests.  Is there some other benefit that the shared memory approach
> > > gives
> > > you?
>
> > > Of course, adding unit tests that exercise your code on a smaller data
> > > set
> > > might also be a way to speed up development.
>
> > > Jean-Paul
>
> > Thanks Jean-Paul, I'll have a think about this.  I'm not sure if it
> > will get me exactly what I want though, as I would need to keep
> > unloading my development module and reloading it, all within the
> > forked process, and I don't see how my debugger (and emacs pdb
> > tracking) will keep up with that to let me step though the code.
>
> Not really.  Don't load your code at all in the parent.  Then there's
> nothing to unload in each child process, just some code to load for
> the very first time ever (as far as that process is concerned).
>
> Jean-Paul


Jean, sorry I'm still not sure what you mean, could you give a couple
of lines of pseudocode to illustrate it?   And explain how my emacs
pdbtrack would still be able to pick it up?
thanks,
charles

-- 
http://mail.python.org/mailman/listinfo/python-list