Re: repeating regular expressions in one string
Shane wrote:
> Hi folks,
>
> I'm new to regular expressions (and a novice at Python) but it seems to be
> the tool I need for a particular problem. I have a bunch of strings that
> looks like this:
>
> 'blahblah_sf1234-sf1238_blahblah'
>
> and I would like to use the re module to parse all the 'sf' parts of the
> string. Each 'sf' needs to be its own string when I am through. How do I
> compile a regular expression that looks for more than one instance? Currently
> my expression looks like this:
>
> myString = re.compile('sf[0-9][0-9][0-9][0-9]')
>
>
Well, since all your strings come in the same format you might try
something like
myString = re.compile(r'\w+_(sf\d\d\d\d)-(sf\d\d\d\d)_\w+')
then when you do your matching:
extracted = myString.match(originalStrnig)
your two extracted strings would be accessible via:
extracted.group(1)
extracted.group(2)
> This works great for finding the first instance of 'sf'. I hope that was
> clear :)
>
> Thanks,
>
> Shane
>
--
http://mail.python.org/mailman/listinfo/python-list
Re: about try and exception
I would think that when the exception occurs the interpreter exits the block of code it is currently in and enters the exception block. Thus the line n = 1/2 would never get executed. -Carl Ben Bush wrote: > I wrote the following code to test the use of "try...exception", > and I want n to be printed out. However, the following code's output is: > Traceback (most recent call last): > File > "C:\Python23\lib\site-packages\Pythonwin\pywin\framework\scriptutils.py", > line 310, in RunScript > exec codeObject in __main__.__dict__ > File "C:\py\use\tryExVa.py", line 7, in ? > print n > NameError: name 'n' is not defined > > the code is here: > > try: > m=2/0 > n=1/2 > except ZeroDivisionError: > pass > print "Yes!!! This line will always print" > print n -- http://mail.python.org/mailman/listinfo/python-list
Re: about try and exception
Shi Mu wrote: > On 11/17/05, Carl J. Van Arsdall <[EMAIL PROTECTED]> wrote: > >> I would think that when the exception occurs the interpreter exits the >> block of code it is currently in and enters the exception block. >> >> Thus the line n = 1/2 would never get executed. >> >> >> -Carl >> >> Ben Bush wrote: >> >>> I wrote the following code to test the use of "try...exception", >>> and I want n to be printed out. However, the following code's output is: >>> Traceback (most recent call last): >>> File >>> "C:\Python23\lib\site-packages\Pythonwin\pywin\framework\scriptutils.py", >>> line 310, in RunScript >>> exec codeObject in __main__.__dict__ >>> File "C:\py\use\tryExVa.py", line 7, in ? >>> print n >>> NameError: name 'n' is not defined >>> >>> the code is here: >>> >>> try: >>> m=2/0 >>> n=1/2 >>> except ZeroDivisionError: >>> pass >>> print "Yes!!! This line will always print" >>> print n >>> > It is true. If I want to run the statement after the division error > such as n=1/2, what should I do? > move the n =1/2 after the division error. Basically what will happen in an exception block is once an exception is thrown the interpreter will leave the block and handle the exception. Statements following the statement that caused the exception will then be ignored. So if you want n=1/2 to be executed regardless of what happens during the line m = 2/0 move it outside the block Ex: try: m=2/0 except ZeroDivisionError: pass n=1/2 print "Yes!!! This line will always print" print n or likewise you could experiment with: try: n = 1/2 m = 2/0 except: pass print 'Yay' print n Here the line "n = 1/2" will get executed BEFORE the exception is thrown. Hope that helps a bit. -carl -- -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Mutability of function arguments?
> > And, indeed, would that approach work? Would declaring: > > class FooWrapper : > __init__(fooToLoad) : > self.foo = fooToLoad > > mean that I could now declare a FooWrapper holding a foo, pass the > FooWrapper to a function, and have the function conclude with the foo > within the FooWrapper now modified? > Yes, passing FooWrapper will pass by reference to a python function thereby allowing you to edit the fooToLoad HTH, carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Threading in python
Hi everyone, I'm trying to use the threading module with python 2.2 and I have some questions regarding python's threading. 1. Who schedules which threads run and when? Is this something left up to the operating system or does python provide a mechanism for this? 2. I've read that python threads don't like to allow other threads to run except in certain situations, these would be situations where there is sleep or I/O happening, is this true? If so, what are the cases in which a python thread would not give up the processor? 3. Is there a way to which thread is running? I mean something a bit more robust than a print statement inside the thread, I want to be able to see when a context switch occurs, is this possible? The reason for this is I will have threads deadlocked waiting for I/O and I am interested to see how often context switching occurs. Thanks in advance, -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Threading in python
> These answers assume you're using CPython; Jython and IronPython have > different answers. > > This confuses me. What is CPython versus Jython and IronPython? I'm using compiled source from python.org, would this be CPython? >> 1. Who schedules which threads run and when? Is this something left up >> to the operating system or does python provide a mechanism for this? >> > > Strictly OS -- Python provides no control. > I did some research yesterday, and I'd like some clarification. So the OS handles which thread runs and when, however if one of the python processes currently holds the global interpreter lock, when the OS switches to a python thread that does not have this lock, this thread will do nothing. Does this sound right? Ultimately python does control what thread runs by controlling the global interpreter lock although its the underlying OS that handles all the context switching etc. Because of this global interpreter lock does this mean its impossible to get speed up with threading on multiple processor systems? I would think so because only one python thread can execute at any one time. Is there a way to get around this? This isn't something I need to do, I'm just curious at this point. -c -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Question about tuple lengths
From my interpreter prompt:
>>> tuple = ("blah")
>>> len(tuple)
4
>>> tuple2 = ("blah",)
>>> len (tuple2)
1
So why is a tuple containing the string "blah" without the comma of
length four? Is there a good reason for this or is this a bug?
--
Carl J. Van Arsdall
[EMAIL PROTECTED]
Build and Release
MontaVista Software
--
http://mail.python.org/mailman/listinfo/python-list
Re: Why and how "there is only one way to do something"?
[EMAIL PROTECTED] wrote: > >>>> Would you, say, remove "for" loops because they could be written as >>>> "while" loops. Don't forget the word "obvious" that appears in that >>>> catchphrase ... >>>> >>>> >>> If every "for" usage can be done with "while" and that "while" is the >>> preferred way, why not ? As I said, the problem is that "obvious" >>> really is subjective in many case. And if it really is obvious, it >>> really is obvious and I doubt there would be that much disagreement. >>> I think comparing "for" and "while" loops in python is problematic. Although yes a "for" loop could be done with a "while" in python a "for" loop shouldn't be used for general looping, the "obvious" case for a "for" loop is to iterate though a list or something similar to that. I wouldn't typically use "while" loops for that, and although it could be done, if you were familiar with python using a "for" loop would be the most obvious. I think it really is obvious in most cases with python. Although, obvious to whom is a good question. If you don't know the language very little will be obvious to you, however one who is familiar with python (rtfm) would know which cases should obviously use "while" and which cases should obviously use "for" 2cents -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Is this a refrence issue?
KraftDiner wrote: > I understand that everything in python is a refrence > > I have a small problem.. > > I have a list and want to make a copy of it and add an element to the > end of the new list, > but keep the original intact > > so: > tmp = myList > tmp = myList is a shallow copy > tmp.append(something) > print tmp, myList > > should be different... > Therefore it shouldn't be different. What you could do is make a second list tmp = [] And then use the extend method: tmp.extend(myList) tmp.append(someEntry) This would give you what you want, and there's more than one way to do this. -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Abuse of the object-nature of functions?
Bruno Desthuilliers wrote: > Carl J. Van Arsdall wrote: > >> Sybren Stuvel wrote: >> >> >>> Ant enlightened us with: >>> >>> >>> >>>> try: >>>> assertion = callable.is_assertion >>>> except: >>>> pass >>>> >>>> >>> Try to make a habit out of catching only the exceptions you know will >>> be thrown. Catching everything generally is a bad idea. In this case, >>> my bet is that catching AttributeError is enough. >>> >>> >>> >> What about doing exception kind of like a C switch statement with a >> default case: >> >> try: >> do_something() >> except TypeError: >> fix_something() >> except: >> print "Unknown error, you are doomed" >> traceback.print_exc() #something to print the traceback >> exit_gracefully() >> >> Is this frowned upon? You still handle the error and you know where it >> happened and what happened. Anything wrong with this? I don't like the >> idea of my system crashing for any reason. >> > > It may be a good idea to do something like this *at the top level of the > application*. But take time to carefully read the standard exceptions > hierarchy in the fine manual - you'll notice some exception you perhaps > don't want to catch or at least don't want to display (hint: look for > the warnings hierarchy and for SysExit...) > > Hrmms, well, here's an interesting situation. So say we wanna catch most exceptions but we don't necessarily know what they are going to be. For example, I have a framework that executes modules (python functions), the framework wraps each function execution in a try/except block in order to compensate for what *might* happen. Upon coding the framework I really have no idea what types of problems these modules might have but I want to catch these errors so that I can clean up and exit gracefully, not only that but I want to dump the exception to log files so that we can attempt to fix it. So, I have the option of catching all standard exceptions and not list the ones I know I don't want to catch. But what about user defined exceptions? Do I then have to enforce policies on the system stating what types of exceptions can be raised? Is there a way in python to say, "hey, catch everything but these two"? -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Is there a limit to os.popen()?
I'm not sure the proper way to phrase the question, but let me try.
Basically, I'm working with a script where someone wrote:
kr = string.strip(os.popen('make kernelrelease').read())
And then searches kr to match a regular expression.
This seems to have been working, however lately when this line executes
I get a number of messages to stderr after several minutes of execution:
cat: write error: Broken pipe
cat: write error: Broken pipe
cat: write error: Broken pipe
I know the output from this make has been growing (make applies some
patches and the patch list is growing). Does os.popen() have some kind
of read buffer limit that i'm hitting which is causing things to break?
--
Carl J. Van Arsdall
[EMAIL PROTECTED]
Build and Release
MontaVista Software
--
http://mail.python.org/mailman/listinfo/python-list
Re: Is there a limit to os.popen()?
cdecarlo wrote: > Hello, > > I'm not 100% sure on this but to me it looks like there is a problem in > your make file. I would look in there first, see where 'cat' is > executed, I bet your problem will be around there. > > Hope this helps, > > Colin > > Well, running the make on the command line seems to work just fine, no errors at all. I, in fact, get the results I expect with no error messages printed out. I thought maybe something might have used cat as a utility, thanks, i'll keep on it. -c -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Is there a limit to os.popen()?
Lawrence D'Oliveiro wrote: > In article <[EMAIL PROTECTED]>, > "Carl J. Van Arsdall" <[EMAIL PROTECTED]> wrote: > > >> Well, running the make on the command line seems to work just fine, no >> errors at all. >> > > What about running it as > > make kernelrelease | cat > > This way the output goes to a pipe, which is what happens when it's > called from your script. Do you see those "broken pipe" messages after a > few minutes in this case? > Alright, so I tried that line in the interpreter and got the same problem. So let me quickly state what I know, and what I'm starting to infer. Anyhow, running make on the command line gives me no problems. There are cats all over the makefile, I checked, but I don't seem to have any problems on the command line. Now, when attempting to run this make in python things seem to go alright, except I get tons of errors dumped to the screen, its very confusing. I don't know too much about pipes though or how exactly to go about debugging this. Could it be something like, make's output is piped correctly but pipes used by cat within the makefile get screwed up inside python somehow? -c -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Python for Embedded Systems?
Grant Edwards wrote: > On 2006-07-14, Jack <[EMAIL PROTECTED]> wrote: > >> Is there a Python packaging that is specifically for >> embedded systems? ie, very small and configurable so the >> user gets to select what modules to install? >> >> For Linux-based embedded systems in particular? >> >> I'm thinking of running it on the Linksys's Linux-based open >> source router WRT54G. It has 4MB flash and 16MB RAM. I think >> another model has 16MB flash. Any possibilities of running >> Python on these systems? >> > > A few years back there was a "deeply embedded python" project, > but it's been dead for quite a while: > > http://www.tucs.fi/magazin/output.php?ID=2000.N2.LilDeEmPy > http://mail.python.org/pipermail/python-announce-list/1999-August/000157.html > > This raises a good question. Is there a need for python to change somewhat to work better in an embedded profile? Are there many people in the community interested in using python for embedded projects? -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: How to lock files (the easiest/best way)?
[EMAIL PROTECTED] wrote:
>
>> ith a quick look.
>>
>
>
> f = open("/path/to/data/directory/lockfile","r")
> try:
> fcntl.flock(f.fileno(),fcntl.LOCK_EX)
> ...access data freely here...
> finally:
> f.close()
>
> Closing the file should release the lock (unless you have a truly
> horrible operating system).
>
>
I also find that fcntl has problems with NFS (or at least, *I* had
problems using the python fcntl module and nfs - could be that horrible
operating system, but doing things like that over nfs can be tricky).
-carl
--
Carl J. Van Arsdall
[EMAIL PROTECTED]
Build and Release
MontaVista Software
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to force a thread to stop
Dennis Lee Bieber wrote:
> On Sat, 22 Jul 2006 14:47:30 +0200, "Hans" <[EMAIL PROTECTED]> declaimed
> the following in comp.lang.python:
>
>
>> Hi all,
>>
>> Is there a way that the program that created and started a thread also stops
>> it.
>> (My usage is a time-out).
>>
>>
> Hasn't this subject become a FAQ entry yet?
>
> The only reliable way of stopping a thread is from inside the thread
> itself. That is, the thread must, at some point, examine some flag
> variable which, when set, says "stop"
>
> Without using actual code:
>
> class StoppableThread(...):
> def __init__(self, ...):
> #whatever is needed to initialize as a thread
> self.Stop = False
>
> def run(self, ...):
> while not self.Stop:
> #do one cycle of the computation
>
>
>
My problem with the fact that python doesn't have some type of "thread
killer" is that again, the only solution involves some type of polling
loop. I.e. "if your thread of execution can be written so that it
periodically checks for a kill condition". This really sucks, not just
because polling is a ridiculous practice, but it forces programmers in
many situations to go through a lengthy process of organizing operations
into a list. For, say I have threads that share a bunch of common
memory (yea, i'm saying this exclusively to get the procses users off my
back) that executes a series of commands on remote nodes using rsh or
something. So if i've constructed my system using threads I need to
neatly go and dump all operations into some sort of list so that I can
implement a polling mechanism, i.e.
opList = [op1, op2, op3, op4]
for op in opList:
checkMessageQueue()
op()
That works if you can easily create an opList. If you want good
response time this can become quite ugly, especially if you have a lot
going on. Say I have a function I want to run in a thread:
#Just pretend for the sake of arguement that 'op' actually means
something and is a lengthy operation
def func_to_thread():
os.system('op 1')
os.system('op 2')
os.system('op 3')
#In order to make this killable with reasonable response time we have to
organize each of our ops into a function or something equally annoying
op_1():
os.system('op 1')
op_2():
os.system('op 2')
op_3():
os.system('op 3')
opList(op_1, op_2, op_3)
def to_thread():
for op in opList:
checkMessageQueue()
op()
So with this whole "hey mr. nice thread, please die for me" concept gets
ugly quickly in complex situations and doesn't scale well at all.
Furthermore, say you have a complex systems where users can write
pluggable modules. IF a module gets stuck inside of some screwed up
loop and is unable to poll for messages there's no way to kill the
module without killing the whole system. Any of you guys thought of a
way around this scenario?
--
Carl J. Van Arsdall
[EMAIL PROTECTED]
Build and Release
MontaVista Software
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to force a thread to stop
Steve Holden wrote: > Carl J. Van Arsdall wrote: > [... rant ...] > >> So with this whole "hey mr. nice thread, please die for me" concept gets >> ugly quickly in complex situations and doesn't scale well at all. >> Furthermore, say you have a complex systems where users can write >> pluggable modules. IF a module gets stuck inside of some screwed up >> loop and is unable to poll for messages there's no way to kill the >> module without killing the whole system. Any of you guys thought of a >> way around this scenario? >> >> >> > > Communications through Queue.Queue objects can help. But if you research > the history of this design decision in the language you should discover > there are fairly sound rasons for not allowing arbitrary "threadicide". > > > Right, I'm wondering if there was a way to make an interrupt driven communication mechanism for threads? Example: thread receives a message, stops everything, and processes the message. -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: How to force a thread to stop
Jean-Paul Calderone wrote: > On Mon, 24 Jul 2006 11:22:49 -0700, "Carl J. Van Arsdall" <[EMAIL PROTECTED]> > wrote: > >> Steve Holden wrote: >> >>> Carl J. Van Arsdall wrote: >>> [... rant ...] >>> >>> >>>> So with this whole "hey mr. nice thread, please die for me" concept gets >>>> ugly quickly in complex situations and doesn't scale well at all. >>>> Furthermore, say you have a complex systems where users can write >>>> pluggable modules. IF a module gets stuck inside of some screwed up >>>> loop and is unable to poll for messages there's no way to kill the >>>> module without killing the whole system. Any of you guys thought of a >>>> way around this scenario? >>>> >>>> >>>> >>>> >>> Communications through Queue.Queue objects can help. But if you research >>> the history of this design decision in the language you should discover >>> there are fairly sound rasons for not allowing arbitrary "threadicide". >>> >>> >>> >>> >> Right, I'm wondering if there was a way to make an interrupt driven >> communication mechanism for threads? Example: thread receives a >> message, stops everything, and processes the message. >> >> > > And what happens if the thread was halfway through a malloc call and > the data structures used to manage the state of the heap are in an > inconsistent state when the interrupt occurs? > > This has been discussed many many times in the context of many many > languages and threading libraries. If you're really interested, do > the investigation Steve suggested. You'll find plenty of material. > I've been digging around with Queue.Queue and have yet to come across any solution to this problem. Queue.Queue just offers a pretty package for passing around data, it doesn't solve the "polling" problem. I wonder why malloc()'s can't be done in an atomic state (along with other operations that should be atomic, maybe that's a question for OS guys, I dunno). Using Queue.Queue still puts me in a horribly inflexible "polling" scenario. Yea, I understand many of the reasons why we don't have "threadicide", and why it was even removed from java. What I don't understand is why we can't come up with something a bit better. Why couldn't a thread relinquish control when its safe to do so? While the interpreter is busy doing malloc()s a thread receives a control message, the thread waits until it knows its no longer in an atomic state and gives control to the message handler when it can. Its about setting up a large system that is controllable without the painstaking process of putting message polling loops all over the place. Main threads in a python program can setup a signal handler, accept signals, and that signal can happily go ahead and kill a python interpreter. Why can't this concept be taken farther and introduced into threading? There is no system that is completely interruptible, there will always be a state in which it is not safe to interrupt, but many systems work around this just fine with cautious programming. Has anyone considered an event driven approach to sending control messages to threads? -c -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: How to force a thread to stop
Steve Holden wrote: > Carl J. Van Arsdall wrote: > > >> >> > I take this to mean you don't want to do the necessary research? ;-) > Well, i've been looking at this on and off for quite some time now, I almost feel like I've seen it all in regards to the "thread killer" scenario. It could be that i'm not finding or understanding the answers I'm getting from google. But after the last email I sat here with my python book, the python cook book, the python module reference, and google for a few minutes before responding. I swear... i'm not THAT lazy. Just curious and uninformed! Questions of event driven python generally lead me to twisted. I will admit that I haven't read much about that yet. >> There is no system that is completely interruptible, there will always >> be a state in which it is not safe to interrupt, but many systems work >> around this just fine with cautious programming. Has anyone considered >> an event driven approach to sending control messages to threads? >> >> > The big problem (as you'll see when you ...) is providing facilities > that are platform-independent. > > Ah, I could see that. I think before I can make any suggestions on this front I need to start reading python source code. Gracias, Carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: How to force a thread to stop
[EMAIL PROTECTED] wrote:
> Carl J. Van Arsdall wrote:
> [...]
>
>> My problem with the fact that python doesn't have some type of "thread
>> killer" is that again, the only solution involves some type of polling
>> loop.
>>
>
> A polliing loop is neither required nor helpful here.
>
> [...]
>
>> #Just pretend for the sake of arguement that 'op' actually means
>> something and is a lengthy operation
>> def func_to_thread():
>> os.system('op 1')
>> os.system('op 2')
>> os.system('op 3')
>>
>
> What good do you think killing that thread would do? The
> process running 'op n' has no particular binding to the thread
> that called os.system(). If 'op n' hangs, it stays hung.
>
> The problem here is that os.system doesn't give you enough
> control. It doesn't have a timeout and doesn't give you a
> process ID or handle to the spawned process.
>
> Running os.system() in multiple threads strikes me as
> kind of whacked. Won't they all compete to read and write
> stdin/stdout simultaneously?
>
Unfortunately this is due to the nature of the problem I am tasked with
solving. I have a large computing farm, these os.system calls are often
things like ssh that do work on locations remote from the initial python
task. I suppose eventually I'll end up using a framework like twisted
but, as with many projects, I got thrown into this thing and threading
is where we ended up. So now there's the rush to make things work
before we can really look at a proper solution.
>
>> #In order to make this killable with reasonable response time we have to
>> organize each of our ops into a function or something equally annoying
>>
>> op_1():
>> os.system('op 1')
>>
>> op_2():
>> os.system('op 2')
>>
>> op_3():
>> os.system('op 3')
>>
>> opList(op_1, op_2, op_3)
>> def to_thread():
>> for op in opList:
>> checkMessageQueue()
>> op()
>>
>
> Nonsense. If op() hangs, you never get to checkMessageQueue().
>
Yea, understood. At the same time, I can't use a timeout either, I
don't know how long op_1 or op_2 will be. This is why I want something
that is triggered on an event.
> Now suppose op has a timeout. We could write
>
> def opcheck(thing):
> result = op(thing)
> if result == there_was_a_timeout:
> raise some_timeout_exception
>
> How is:
>
> def func_to_thread():
> opcheck('op 1')
> opcheck('op 2')
> opcheck('op 3')
>
> any less managable than your version of func_to_thread?
>
>
Again, the problem I'm trying to solve doesn't work like this. I've
been working on a framework to be run across a large number of
distributed nodes (here's where you throw out the "duh, use a
distributed technology" in my face). The thing is, I'm only writing the
framework, the framework will work with modules, lots of them, which
will be written by other people. Its going to be impossible to get
people to write hundreds of modules that constantly check for status
messages. So, if I want my thread to "give itself up" I have to tell it
to give up. In order to tell it to give up I need some mechanism to
check messages that is not going to piss off a large team of
programmers. At the same time, do I really want to rely on other people
to make things work? Not really, I'd much rather let my framework
handle all control and not leave that up to programmers.
So the problem is, I have something linearly executed a large list of
python functions of various sizes ranging from short to long. Its not
about killing the thread so much as how do I make the thread listen to
control messages without polling.
>> So with this whole "hey mr. nice thread, please die for me" concept gets
>> ugly quickly in complex situations and doesn't scale well at all.
>> Furthermore, say you have a complex systems where users can write
>> pluggable modules. IF a module gets stuck inside of some screwed up
>> loop and is unable to poll for messages there's no way to kill the
>> module without killing the whole system. Any of you guys thought of a
>> way around this scenario?
>>
>
> Threadicide would not solve the problems you actually have, and it
> tends to create other problems. What is the condition that makes
> you want to kill the thread? Make the victim thread respond to that
> condition itself.
>
>
I feel like this is something we'
Re: How to force a thread to stop
Gerhard Fiedler wrote: > On 2006-07-25 13:30:22, Carl J. Van Arsdall wrote: > > >>> Running os.system() in multiple threads strikes me as kind of whacked. >>> Won't they all compete to read and write stdin/stdout simultaneously? >>> >>> >> Unfortunately this is due to the nature of the problem I am tasked with >> solving. I have a large computing farm, these os.system calls are often >> things like ssh that do work on locations remote from the initial python >> task. >> > > [...] > > >> Again, the problem I'm trying to solve doesn't work like this. I've been >> working on a framework to be run across a large number of distributed >> nodes (here's where you throw out the "duh, use a distributed >> technology" in my face). The thing is, I'm only writing the framework, >> the framework will work with modules, lots of them, which will be >> written by other people. Its going to be impossible to get people to >> write hundreds of modules that constantly check for status messages. >> > > Doesn't this sound like a case for using processes instead of threads? > Where you don't have control over the thread, you can use a process and get > the separation you need to be able to kill this task. > > Alternatively you could possibly provide a base class for the threads that > handles the things you need every thread to handle. They'd not have to > write it then; they'd not even have to know too much about it. > > Gerhard > > I'd be all for using processes but setting up communication between processes would be difficult wouldn't it? I mean, threads have shared memory so making sure all threads know the current system state is an easy thing. With processes wouldn't I have to setup some type of server/client design, where one process has the system state and then the other processes constantly probe the host when they need the current system state? -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: How to force a thread to stop
[EMAIL PROTECTED] wrote: > Carl J. Van Arsdall wrote: > > I don't get what threading and Twisted would to do for > you. The problem you actually have is that you sometimes > need terminate these other process running other programs. > Use spawn, fork/exec* or maybe one of the popens. > > > I have a strong need for shared memory space in a large distributed environment. How does spawn, fork/exec allow me to meet that need? I'll look into it, but I was under the impression having shared memory in this situation would be pretty hairy. For example, I could fork of a 50 child processes, but then I would have to setup some kind of communication mechanism between them where the server builds up a queue of requests from child processes and then services them in a FIFO fashion, does that sound about right? > Threads have little to do with what you say you need. > > [...] > >> I feel like this is something we've established multiple times. Yes, we >> want the thread to kill itself. Alright, now that we agree on that, >> what is the best way to do that. >> > > Wrong. In your examples, you want to kill other processes. You > can't run external programs such as ssh as Python threads. Ending > a Python thread has essentially nothing to do with it. > There's more going on than ssh here. Since I want to run multiple processes to multiple devices at one time and still have mass shared memory I need to use threads. There's a mass distributed system that needs to be controlled, that's the problem I'm trying to solve. You can think of each ssh as a lengthy IO process that each gets its own device. I use the threads to allow me to do IO to multiple devices at once, ssh just happens to be the IO. The combination of threads and ssh allowed us to have a *primitive* distributed system (and it works too, so I *can* run external programs in python threads). I didn't say is was the best or the correct solution, but it works and its what I was handed when I was thrown into this project. I'm hoping in fifteen years or when I get an army of monkeys to fix it, it will change. I'm not worried about killing processes, that's easy, I could kill all the sshs or whatever else I want without batting an eye. The threads that were created in order to allow me to do all of this work simultaneously, that's the issue. Granted, I'm heavily looking into a way of doing this with processes, I still don't see how threads are the wrong choice with my present situation. > > Not me. I'm saying work the problem you actually have. > The problem I have is a large distributed system, that's the reality of it. The short summary, I need to use and control 100+ machines in a computing farm. They all need to share memory or to actively communicate with each other via some other mechanism. Without giving any other details, that's the problem I have to solve. Right now I'm working with someone else's code. Without redesigning the system from the ground up, I have to fix it. -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: How to force a thread to stop
Paul Rubin wrote: > "Carl J. Van Arsdall" <[EMAIL PROTECTED]> writes: > >> The problem I have is a large distributed system, that's the reality >> of it. The short summary, I need to use and control 100+ machines in >> a computing farm. They all need to share memory or to actively >> communicate with each other via some other mechanism. Without giving >> any other details, that's the problem I have to solve. >> > > Have you looked at POSH yet? http://poshmodule.sf.net > > There's also an shm module that's older and maybe more reliable. > Or you might be able to just use mmap. > I'm looking at POSH, shm, and stackless right now! :-) Thanks! -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Threads vs Processes
Alright, based a on discussion on this mailing list, I've started to wonder, why use threads vs processes. So, If I have a system that has a large area of shared memory, which would be better? I've been leaning towards threads, I'm going to say why. Processes seem fairly expensive from my research so far. Each fork copies the entire contents of memory into the new process. There's also a more expensive context switch between processes. So if I have a system that would fork 50+ child processes my memory usage would be huge and I burn more cycles that I don't have to. I understand that there are ways of IPC, but aren't these also more expensive? So threads seems faster and more efficient for this scenario. That alone makes me want to stay with threads, but I get the feeling from people on this list that processes are better and that threads are over used. I don't understand why, so can anyone shed any light on this? Thanks, -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: How to force a thread to stop
[EMAIL PROTECTED] wrote: > Carl J. Van Arsdall wrote: > >> [EMAIL PROTECTED] wrote: >> >>> Carl J. Van Arsdall wrote: >>> >>> I don't get what threading and Twisted would to do for >>> you. The problem you actually have is that you sometimes >>> need terminate these other process running other programs. >>> Use spawn, fork/exec* or maybe one of the popens. >>> >>> >> I have a strong need for shared memory space in a large distributed >> environment. >> > > Distributed shared memory is a tough trick; only a few systems simulate > it. > Yea, this I understand, maybe I chose some poor words to describe what I wanted. I think this conversation is getting hairy and confusing so I'm going to try and paint a better picture of what's going on. Maybe this will help you understand exactly what's going on or at least what I'm trying to do, because I feel like we're just running in circles. After the detailed explanation, if threads are the obvious choice or not, it will be much easier to pick apart what I need and probably also easier for me to see your point... so here goes... (sorry its long, but I keep getting dinged for not being thorough enough). So, I have a distributed build system. The system is tasked with building a fairly complex set of packages that form a product. The system needs to build these packages for 50 architectures using cross compilation as well as support for 5 different hosts. Say there are also different versions of this with tweaks for various configurations, so in the end I might be trying to build 200+ different things at once. I have a computing farm of 40 machines to do this for me.. That's the high-level scenario without getting too detailed. There are also subsystems that help us manage the machines and things, I don't want to get into that, I'm going to try to focus on a scenario more abstract than cluster/resource management stuff. Alright, so manually running builds is going to be crazy and unmanageable. So what the people who came before me did to manage this scenario was to fork on thread per build. The threads invoke a series of calls that look like os.system(ssh ) or for more complex operations they would just spawn a process that ran another python script) os.system(ssh
Re: How to force a thread to stop
Paul Rubin wrote: > "Carl J. Van Arsdall" <[EMAIL PROTECTED]> writes: > >> Alright, so manually running builds is going to be crazy and >> unmanageable. So what the people who came before me did to manage >> this scenario was to fork on thread per build. The threads invoke a >> series of calls that look like >> >> os.system(ssh ) >> > > Instead of using os.system, maybe you want to use one of the popens or > the subprocess module. For each ssh, you'd spawn off a process that > does the ssh and communicates back to the control process through a > set of file descriptors (Unix pipe endpoints or whatever). The > control process could use either threads or polling/select to talk to > the pipes and keep track of what the subprocesses were doing. > > I don't think you need anything as complex as shared memory for this. > You're just writing a special purpose chat server. > Sorry for sounding naive, but how is writing a chat server less complex then letting python handle shared memory while I manage a couple of locks? Also, threading's condition and event constructs are used a lot (i talk about it somewhere in that thing I wrote). They are easy to use and nice and ready for me, with a server wouldn't I have to have things poll/wait for messages? -c -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Threads vs Processes
Paul Rubin wrote: > "Carl J. Van Arsdall" <[EMAIL PROTECTED]> writes: > >> Processes seem fairly expensive from my research so far. Each fork >> copies the entire contents of memory into the new process. >> > > No, you get two processes whose address spaces get the data. It's > done with the virtual memory hardware. The data isn't copied. The > page tables of both processes are just set up to point to the same > physical pages. Copying only happens if a process writes to one of > the pages. The OS detects this using a hardware trap from the VM > system. > Ah, alright. So if that's the case, why would you use python threads versus spawning processes? If they both point to the same address space and python threads can't run concurrently due to the GIL what are they good for? -c -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: How to force a thread to stop
Gerhard Fiedler wrote: > On 2006-07-26 19:08:44, Carl J. Van Arsdall wrote: > > >> Also, threading's condition and event constructs are used a lot >> (i talk about it somewhere in that thing I wrote). They are easy to use >> and nice and ready for me, with a server wouldn't I have to have things >> poll/wait for messages? >> > > How would a thread receive a message, unless it polls some kind of queue or > waits for a message from a queue or at a semaphore? You can't just "push" a > message into a thread; the thread has to "pick it up", one way or another. > > Gerhard > > Well, I guess I'm thinking of an event driven mechanism, kinda like setting up signal handlers. I don't necessarily know how it works under the hood, but I don't poll for a signal. I setup a handler, when the signal comes, if it comes, the handler gets thrown into action. That's what I'd be interesting in doing with threads. -c -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Threads vs Processes
[EMAIL PROTECTED] wrote: > Carl J. Van Arsdall wrote: > >> Alright, based a on discussion on this mailing list, I've started to >> wonder, why use threads vs processes. >> > > In many cases, you don't have a choice. If your Python program > is to run other programs, the others get their own processes. > There's no threads option on that. > > If multiple lines of execution need to share Python objects, > then the standard Python distribution supports threads, while > processes would require some heroic extension. Don't confuse > sharing memory, which is now easy, with sharing Python > objects, which is hard. > > Ah, alright, I think I understand, so threading works well for sharing python objects. Would a scenario for this be something like a a job queue (say Queue.Queue) for example. This is a situation in which each process/thread needs access to the Queue to get the next task it must work on. Does that sound right? Would the same apply to multiple threads needed access to a dictionary? list? Now if you are just passing ints and strings around, use processes with some type of IPC, does that sound right as well? Or does the term "shared memory" mean something more low-level like some bits that don't necessarily mean anything to python but might mean something to your application? Sorry if you guys think i'm beating this to death, just really trying to get a firm grasp on what you are telling me and again, thanks for taking the time to explain all of this to me! -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Threads vs Processes
[EMAIL PROTECTED] wrote: > Carl J. Van Arsdall wrote: > >> Ah, alright, I think I understand, so threading works well for sharing >> python objects. Would a scenario for this be something like a a job >> queue (say Queue.Queue) for example. This is a situation in which each >> process/thread needs access to the Queue to get the next task it must >> work on. Does that sound right? >> > > That's a reasonable and popular technique. I'm not sure what "this" > refers to in your question, so I can't say if it solves the > problem of which you are thinking. > > >> Would the same apply to multiple >> threads needed access to a dictionary? list? >> > > The Queue class is popular with threads because it already has > locking around its basic methods. You'll need to serialize your > operations when sharing most kinds of objects. > > Yes yes, of course. I was just making sure we are on the same page, and I think I'm finally getting there. >> Now if you are just passing ints and strings around, use processes with >> some type of IPC, does that sound right as well? >> > > Also reasonable and popular. You can even pass many Python objects > by value using pickle, though you lose some safety. > I actually do use pickle (not for this, but for other things), could you elaborate on the safety issue? > >> Or does the term >> "shared memory" mean something more low-level like some bits that don't >> necessarily mean anything to python but might mean something to your >> application? >> > > Shared memory means the same memory appears in multiple processes, > possibly at different address ranges. What any of them writes to > the memory, they can all read. The standard Python distribution > now offers shared memory via os.mmap(), but lacks cross-process > locks. > > Python doesn't support allocating objects in shared memory, and > doing so would be difficult. That's what the POSH project is > about, but it looks stuck in alpha. > > > -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: How to force a thread to stop
Paul Rubin wrote: > "Paul Boddie" <[EMAIL PROTECTED]> writes: > >> Whether this solves the questioner's problems remains to be seen, but >> issues of handling SSH-based communications streams do seem to be >> addressed. >> > > Actually I don't understand the need for SSH. This is traffic over a > LAN, right? Is all of the LAN traffic encrypted? That's unusual; SSH > is normally used to secure connections over the internet, but the > local network is usually trusted. Hopefully it's not wireless. > The reason for ssh is legacy. I think the person who originally set things up (it was an 8 node farm at the time) just used ssh to execute commands on the remote machine. It was a quick and dirty approach I believe, but at the time, it wasn't worth investing in anything better. Just setup some keys for each node and use ssh (as opposed to rexec or something else, I doubt they put much thought into it). Its not a need as much as I was working with what was handed, and as in many projects, it becomes difficult to change everything at once. So the goal was to change what we had to and abstract the ssh calls away so that we could do something better later. -c -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Threads vs Processes
[EMAIL PROTECTED] wrote: > Carl J. Van Arsdall wrote: > [...] > >> I actually do use pickle (not for this, but for other things), could you >> elaborate on the safety issue? >> > > >From http://docs.python.org/lib/node63.html : > > Warning: The pickle module is not intended to be secure > against erroneous or maliciously constructed data. Never > unpickle data received from an untrusted or unauthenticated > source. > > A corrupted pickle can crash Python. An evil pickle could probably > hijack your process. > > > Ah, i the data is coming from someone else. I understand. Thanks. -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiple Telnet sessions through one script
Well, although you spawn seperate telnet processes there is still only
one thread of control in your pythons script. If you need to do two
things simultaneously you'll need to setup a parallel control
mechanism. For example you could use python threads, each thread spawns
a separate telnet and controls it accordingly. Similarly, you could
fork off other python scripts that control each telnet session.
Alright, so that's for controlling two telnets at once. I think you'll
have another problem, and that's controlling each telnet session
manually. To do this I think you'll need to setup an interface that
provides the two consoles you are after. I'm not exactly sure the best
way to do that. One thought I have is if you used one of the various
GUI toolkits you could have your app open a window that is seperated
into two consoles. Each thread could be bound to one of these consoles
and you could switch between the two by clicking on one side versus the
other. Although if you want to do it all in a shell, or have your
program open multiple shells I'm not sure how to do that, you might
check google. I suppose if you were doing things from a single shell
and wanted to do thing similar to the GUI toolkit I described earlier,
you could try something like ncurses.
I guess I have one final idea, you could use a single shell, buffer
output from each telnet session and have your main control loop give you
the ability to switch back and forth between the two sessions.
Anyhow, hope those ideas help you out a little.
vmalhotra wrote:
> Hi
>
> I am new in python scripting. I want to open a Multiple telnet session
> through once script. In other way i can tell i want to open two linux
> consoles through one script.
>
> I wrote one script, but the issue is I am not able to open multiple
> consoles. The Scripts which i wrote is as follows:
>
> import pexpect
> session = pexpect.spawn("telnet localhost 2601\n")
> session.expect("Password: ")
> session.send("XYZ\n\n")
> session.expect("Router1> ")
> session1 = pexpect.spawn("telnet localhost 2604\n")
> session1.expect("Password: ")
> session1.send("ABCD\n\n")
> session1.expect("ospfd> ")
> #session1.interact()
> session1.interact()
>
> output :
> ospf>
>
> But in this case, i want in one console router one can open and on
> other console ospf should open. But this i want to do is through one
> script only.
>
> Regds
> Vik
>
>
--
Carl J. Van Arsdall
[EMAIL PROTECTED]
Build and Release
MontaVista Software
--
http://mail.python.org/mailman/listinfo/python-list
Get age of a file/dir
I've been looking around the OS module and I haven't found anything useful yet. Does anyone know how to get the age of a file or directory in days? I'm using unix and don't seem to find anything that will help me. The only function that comes close so far is os.path.getctime(path) However this only gets creation time on Windows, on Unix it gets the the time of the last change. Any ideas? Thanks! -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: How to force a thread to stop
Alex Martelli wrote: > H J van Rooyen <[EMAIL PROTECTED]> wrote: > > >> "Paul Rubin" <http://[EMAIL PROTECTED]> Writes: >> >> | "H J van Rooyen" <[EMAIL PROTECTED]> writes: >> | > *grin* - Yes of course - if the WDT was enabled - its something that >> | > I have not seen on PC's yet... >> | >> | They are available for PC's, as plug-in cards, at least for the ISA >> | bus in the old days, and almost certainly for the PCI bus today. >> >> That is cool, I was not aware of this - added to a long running server it >> will >> help to make the system more stable - a hardware solution to hard to find >> bugs >> in Software - (or even stuff like soft errors in hardware - speak to the >> Avionics boys about Neutrons) do you know who sells them and what they are >> called? - >> > > When you're talking about a bunch of (multiprocessing) machines on a > LAN, you can have a "watchdog machine" (or more than one, for > redundancy) periodically checking all others for signs of health -- and, > if needed, rebooting the sick machines via ssh (assuming the sickness is > in userland, of course -- to come back from a kernel panic _would_ > require HW support)... so (in this setting) you _could_ do it in SW, and > save the $100+ per box that you'd have to spend at some shop such as > <http://www.pcwatchdog.com/> or the like... > > > Yea, there are other free solutions you might want to check out, I've been looking at ganglia and nagios. These require constant communication with a server, however they are customizable in that you can have the server take action on various events. Cheers! -c -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Pyro stability
Irmen de Jong wrote: > writeson wrote: > >> Irmen, >> >> Thanks, you're very good about answering Pyro related questions! >> > > Well, I do have an advantage here, being Pyro's author. > And I don't know if you get this enough... but thanks. Pyro is fucking amazing and has been a great help to a couple of our projects. -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Pyro stability
Paul Boddie wrote: > Beliavsky wrote: > >> Carl J. Van Arsdall wrote: >> > > [Enthusiasm for Pyro, not for those with sensitivity to rude words] > > >> You should watch your language in a forum with thousands of readers. >> I think you should find better things to complain about and not worry about language usage so much. -c -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Pyro stability
Steve Holden wrote: > Carl J. Van Arsdall wrote: > >> Paul Boddie wrote: >> >> >>> Beliavsky wrote: >>> >>> >>> >>>> Carl J. Van Arsdall wrote: >>>> >>>> >>> [Enthusiasm for Pyro, not for those with sensitivity to rude words] >>> >>> >>> >>> >>>> You should watch your language in a forum with thousands of readers. >>>> >>>> >> I think you should find better things to complain about and not worry >> about language usage so much. >> >> > It didn't seem like an unreasonable request to me, though I concede that > you aren't (yet :-) known throughout Usenet for your profanity. > > Well, I wouldn't want to be known for it, but some technologies are just so A+mazing that I can't contain myself. Pyro happens to be one of those great things where profanity *is* necessary. I would never want to cheapen my emotion by leaving out colorful language ;) -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: profanity on comp.lang.python (was Re: Pyro stability)
BartlebyScrivener wrote: > Chaz Ginger wrote: > > >>> it is supposed to be about PYTHON. Get it? >>> > > I agree. And Python is an extremely serious matter calling for decorum > and propriety. > Lol, is it really now? And I suppose its your definition of decorum and not mine right? Things like that are always relative. I think decorum would state that you should be an adult and not make a big deal out of nothing. But that's just me, and as I said, its all relative. (and honestly, if you thought the word fuck was bad, you should really be offended by my profanity-free statement above). -c -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: profanity on comp.lang.python (was Re: Pyro stability)
Cliff Wells wrote: > On Wed, 2006-11-08 at 10:12 -0800, Carl J. Van Arsdall wrote: > >> BartlebyScrivener wrote: >> > > >>> I agree. And Python is an extremely serious matter calling for decorum >>> and propriety. >>> >>> >> Lol, is it really now? And I suppose its your definition of decorum and >> not mine right? Things like that are always relative. I think decorum >> would state that you should be an adult and not make a big deal out of >> nothing. But that's just me, and as I said, its all relative. >> > > I think you missed the irony in his statement (or perhaps confused > BartlebyScrivener with Beliavsky, who was the original plaintiff). > > Ah, yea, you are right. My apologies. -c -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: profanity on comp.lang.python (was Re: Pyro stability)
Paddy wrote: > Aahz wrote: > >> In article <[EMAIL PROTECTED]>, >> Beliavsky <[EMAIL PROTECTED]> wrote: >> >>> If this is supposed to justify using bad language in a public forum, >>> it is poorly reasoned. Having heard "f***" does not mean they were not >>> annoyed. 100% of people have seen trash on the street, but that does >>> not justify littering. If a group of people don't mind profanity, there >>> is no harm in their swearing to each other. But Usenet is read by a >>> wide range of people, and needlessly offending some of them is wrong. >>> The OP used "f**" just for emphasis. English is a rich language, >>> and there are better ways of doing that. >>> >> Oh, gimme a f** break. Do a simple Gooja search to find out how >> often people already use "f***" around here. I think you're the one who >> needs to justify your position. >> -- >> Aahz ([EMAIL PROTECTED]) <*> http://www.pythoncraft.com/ >> >> > I too know your wrong Aahz. The written word is not the same as that > spoken. People should make an effort to put across their meaning in a > clear manner. If I were going to an interview I would be very careful > about swearing and most likely not do it. People complain about the > friendliness and tone of groups, and mention it when talking about > programming languages. > > Not everyone swears like Eddy Murphy in Beverley Hills Cop, and a lot > of those that do, would not do so when they want to impress, or > communicate with a stranger. > > The tone of comp.lang.python *is* an asset, I think, to Python that > swearing will diminish. > You are comparing interviews to usenet. I somehow see a disconnect. I don't think many people are going to go to a potential employer and say "hey fuck face, how the fuck are ya?" Yea, its not likely to happen, in most cases people might even dress up to an interview and use all of their professionalisms as to not appear as they would at home. However communicating with people (cause that's what this is, its just people talking to one another about Python and the health of this forum) should be done as people see fit. Although you mentioned impressing people etc, is it really important to impress people here by watching your P's and Q's? What impresses me here is someone's command of the language, I could really give a rats ass how they choose to disseminate their expertise. As its been mentioned before, its one thing for me or anyone else to get in someone's face and be like "listen you little fuck, use a while loop." But that was clearly not the context. Using an expletive as an adjective does not diminish the "friendless" of the group unless you are complete prude. Granted, there are tons of them, I think that the real issue is that people need to learn to ignore things they don't like and not be so *damn* sensitive. Meaning is clearly conveyed, people's sensitivity is their own issue and I think too many people have gotten way to used to the political correctness shoved down our throats by society. Again, that's just my take on it, but those of you who would be offended by my statements and use of colorful language to describe my love of technology should probably just adjust your spam filters to scan for my name or emails that use words you can't handle. Its kind of like not watching tv shows that bother as opposed to raising a stink and having them taken off the air. Hopefully now the count is more like 124. /rant -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: profanity on comp.lang.python (was Re: Pyro stability)
[EMAIL PROTECTED] wrote: > Carl> You are comparing interviews to usenet. I somehow see a > Carl> disconnect. I don't think many people are going to go to a > Carl> potential employer and say "hey fuck face, how the fuck are ya?" > ... > Carl> Although you mentioned impressing people etc, is it really > Carl> important to impress people here by watching your P's and Q's? > Carl> What impresses me here is someone's command of the language, I > Carl> could really give a rats ass how they choose to disseminate their > Carl> expertise. > > If recent news reports are to be believed (*) it would appear that companies > are starting to check out prospective employees online. Granted, most of > the stuff I've seen or heard relates to use of social networking sites like > Facebook and MySpace, but I'm sure savvy employers looking for programming > expertise would know to check out Usenet newsgroups and/or relevant mailing > lists. Anything you post is fair game though. Also, "command of the > language" can extend to the spoken/written word. > > Skip > > (*) > http://www.nytimes.com/2006/06/11/us/11recruit.html?ei=5090&en=ddfbe1e3b386090b&ex=1307678400 > Whereas you are right about employers looking online (its happened to me too), that should remain the issue of the poster of those horrible things. It shouldn't be the issue of "offended code monkey #12". As for your statement about how "command of the language" (in reference to python) extends to written/spoken word, I don't quite understand what you are saying. Yes, there are skills in being able to explain code or concepts to someone, but adjective choice (at least in the way I had used it) hardly hinders one's ability to communicate python ideas and code. Can you explain how the use of profanity affects one's ability to code in python or explain their code? The only thing I really can think of is that someone gets over sensitive and starts complaining instead of taking in the deeper meaning of the "enhanced" statement. -c -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: what are you using python language for?
hacker1017 wrote: > im just asking out of curiosity. > We use python for somewhat complex build system which builds several versions of embedded Linux for multiple architectures (30+). Two particularly interested aspects of this build system involve simple cluster management utilities (utilities to help manage distribution of work across 70 machines) and we are currently developing a distributed execution framework. Some cool stuff if you ask me (I mean, i had better like it, its my job!) -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Killing a thread
Felipe Almeida Lessa wrote: > Em Sex, 2006-06-09 às 13:54 -0700, Manish Marathe escreveu: > >> On 6/9/06, Fredrik Lundh <[EMAIL PROTECTED]> wrote: >> Manish Marathe wrote: >> >> > I am creating threads using my self defined class which >> inherits the >> > threading.Thread class. I want to know how can I kill the >> threads which >> > are being created by the object of my self defined class. >> >> you cannot kill a thread "from the outside"; you have to >> design your >> thread tasks so they can kill themselves, when asked to do >> that. >> >> Thanks for the reply. So can a thread listen to an event i.e. can we >> send an event to the thread indicating to kill itself. >> > > A plain simple boolean flag will certainly do the job. For example > > def run(self): > self.running = True > while self.running: > blah() > > def stop(self): > self.running = False > > Well, this works if your threads are able to poll. If you had a thread doing lengthy IO you could end up waiting a long time until the thread gets an opportunity to kill itself. Are there any plans in the future to add the capability to kill threads from the outside? Better yet, an interruptable thread so instead of using a polling loop you could send a DIE_THREAD_DIE signal or something. I think at present its not possible (or a really bad idea) to put signal handlers in threads. Anyone have thoughts on this? I toyed with another idea (this is a big problem for me), but I noticed that each python thread spawns a new interpreter. If python is doing this, I would think that each thread could be associated with PIDs or something. I haven't thought about it too much, its a little to crazy/brute force for me, but I thought i'd throw it out there so you guys could tell me if that one is a little too far fetched. -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Advanced lockfiles
things*/
void cleanupModule(void)
{
close(fp);
}
/*Init function that python needs to load this as a module*/
void initlock()
{
fp = open("/home/build/bin/resourceManager/lock", O_RDWR);
(void) Py_InitModule("lock", lock_methods);
Py_AtExit((void*)cleanupModule);
}
--
Carl J. Van Arsdall
[EMAIL PROTECTED]
Build and Release
MontaVista Software
--
http://mail.python.org/mailman/listinfo/python-list
Question about the Exception class
So this is probably a fairly basic question, but help me out because I'm just not lining things up and I'm somewhat new to the world of exception handling. What's the benefit to inheriting an exception from and of the available parent exception classes? Does one subclass have benefits over any other? Most of what I see involves making a new class and inheriting from Exception so that one can have an exception class with a name of their choosing. If you didn't care about the name would there be any benefit to making a subclass versus raising StandardError or something else equally vanilla? Are there any difference to library provided exceptions other than their names? -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
need helping tracking down weird bug in cPickle
Hey everyone, cPickle is raising an ImportError that I just don't quite
understand. Before I paste the code, let me explain the application.
Basically the part of the application that failed is a function that
loads a list of objects from a file using cPickle. This list is a queue
of requests. I've done some research and it looks like cPickle tries to
load some modules as some kind of test. From what I can tell the module
that cPickle is trying to load is a concatenation of the module that it
exists in and part of the string that was sent in the orignal request.
It looks like something where memory is getting overwritten, but I'm not
really doing anything too fancy, so i'm not sure how to approach this
problem. I'm using python2.2. Does anyone know how I might go about
trying to troubleshoot something like this? Is there a good tool or a
better method of error handling I can use to try and track this down?
Or is it better to give up troubleshooting and just upgrade to 2.4?
Here's the code where the error occured:
def initQueue():
global masterQueue
try:
queueFPtr = open("/home/build/bin/resourceManager/queue.rm","rb")
except IOError:
raise ResourceError,"Error Opening Queue for Reading"
except:
traceback.print_exc()
raise ResourceError, "Unknown Error Opening Queue"
try:
pickledList = cPickle.load(queueFPtr)
except EOFError:
masterQueue = [[],[],[]] #empty file, set to empty list for construction
#there exist 3 blank lists in the master list, these list refer to
various priorities 0, 1, and 2
queueFPtr.close()
return
except:
traceback.print_exc()
raise ResourceError, "Unknown Error loading Queue"
queueFPtr.close()
masterQueue = pickledList
return
Here's the traceback:
Traceback (most recent call last):
File "/home/build/bin/resourceManager/resourceQueue.py", line 50, in
initQueue
pickledList = cPickle.load(queueFPtr)
ImportError: No module named resourcepmdb'
Traceback (most recent call last):
File "./installimage.py", line 443, in ?
rpmdb = getResource(buildtag,buildid,"dumborpmdb","Installing
installer
rpms")
File "/home/build/bin/resourceManager/resourceManager.py", line 39, in
getResource
resourceQueue.initQueue()
File "/home/build/bin/resourceManager/resourceQueue.py", line 58, in
initQueue
raise ResourceError, "Unknown Error loading Queue"
resourceError.ResourceError: Unknown Error loading Queue
--
Carl J. Van Arsdall
[EMAIL PROTECTED]
Build and Release
MontaVista Software
--
http://mail.python.org/mailman/listinfo/python-list
Re: Quick Question
xkenneth wrote: > I want to be able to cycle through an array and print something in > hexadecimal. Such as this > thisArray = ["AF","0F","5F"] > for x in range(len(thisArray)): >print "\x" + thisArray[x] > > However python chokes on the escaped identifier, how can I get around > this? > > Thanks! > Regards, > Ken > > If you have things in a list you can iterate through the list much easier: thisArray = ["AF","0F","5F"] for item in thisArray: print item In this array you store strings, so if you want to print stuff out in hex you need to give python integers, so let's say you have a list of integers: thisArray = [256,512,1024] for item in thisArray: print "%x"%item That will print those integers in hex. -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: to py or not to py ?
Serge Orlov wrote: > On 6/27/06, Chandrashekhar kaushik <[EMAIL PROTECTED]> wrote: > >> HI all >> I have the following prob. >> I am to write a parallel vis application . >> I wud have by default used C++ for the same but somehow >> thought if py cud help me .. >> It does as in many things that i would otherwise have written down >> already exists ... ( like built in classes for sockets , threading etc ) >> >> I would be doin the following type of tasks .. >> >> 1. sending data across the network >> the data is going to be huge >> >> 2. once data has been sent i will run some vis >> algos parallely on them and get the results >> >> now one thing that i wud req. is serializing my data structures so that >> they can be sent across the net. >> >> pyton does allow this using cPickle , but it bloats the data like anythin >> !!! >> for example a class containing 2 integers which i expect will be 8 bytes >> long .. >> cPickle.dumps returns a string thats 86 bytes wide ( this is the binary >> version protocol 1 ) >> >> anyway to improve serialization ?? >> > > Do it yourself using struct module. > > >> also is it actually a good idea to write high perf applications in python ? >> > > Take a look at Mercurial <http://www.selenic.com/mercurial/> sources. > It's a high performance python application. Or watch Bryan > O'Sullivan's Mercurial presentation > <http://video.google.com/videoplay?docid=-7724296011317502612> he > talks briefly how they made it work fast. > > But writing high performance application in python requires > self-discipline and attention to details, looking at the way you spell > I think it will be a challenge ;) > One more comment would be that running parallel applications on python can be problematic because of the way python does threading (and i'm not presently aware of other threading libraries). Parallelizing in python doesn't do much good unless you spend a lot of time blocking on IO because of the GIL. -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: to py or not to py ?
Robert Kern wrote: >> Carl , what are the problems that could arise with threading ?? >> Because of the GIL only one thread can actually run at a time. So if you are going for speed (say you have an SMP box) and your two requests require significant computation, you'd want each processor to take on a little bit of that work to get things done faster. Now say you want these two to be computed simultaneously and you are using python, you won't get that, one thread will process and then the other will process as the GIL is passed around between the python threads. Now if your bottleneck is IO then you are in good shape for python threads (Your machine only has one network port and you don't have a lot of computation overhead, so your single processor can serve things up faster than your network card can take it). So its not problems that arise, its more like inefficiency. Anyhow, it all depends on what you are trying to do and the needs of your application. -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: to py or not to py ?
Tom Plunket wrote: > Carl J. Van Arsdall wrote: > > >> Because of the GIL only one thread can actually run at a time. >> > > I've recently been wondering about this, since in the work I do, a lot > of time is spent doing disk I/O. So if I want the UI to remain > responsive, I could spawn an IO thread to handle requests, and do a > pretty simple "just whack new requests onto the queue" without locks > since I'm guaranteed to not have the IO thread read at the same time > as the requestor thread? > > ...what exactly constitutes an atomic operation in Python, anyway? > > Well, although only one thread can run at a time due to the GIL you can't accurately predict when the GIL is going to be released and therefore you don't know when another thread is going to pick up and start going (GIL is released on every so many byte instructs - correct me if i'm wrong, certain operations that have a bit to do with IO, and modules you wrote yourself where you manually release the GIL using macros provided in the C API). If you have your own data structure that is shared among threads you can use the threading modules synchronization constructs to get the job done, using locks, conditions, and events. Queue.Queue is also a good way to go about communicating with threads. > e.g. > > class IoThread: ># ... > ># called from the other thread... >def RequestFile(self, name): > self.fileQueue.append(name) > ># called during the IO thread >def GetNextFile(self); > next = self.fileQueue[0] > self.fileQueue.pop(0) > return next > > ? > -tom! > -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Abuse of the object-nature of functions?
Sybren Stuvel wrote: > Ant enlightened us with: > >> try: >> assertion = callable.is_assertion >> except: >> pass >> > > Try to make a habit out of catching only the exceptions you know will > be thrown. Catching everything generally is a bad idea. In this case, > my bet is that catching AttributeError is enough. > > What about doing exception kind of like a C switch statement with a default case: try: do_something() except TypeError: fix_something() except: print "Unknown error, you are doomed" traceback.print_exc() #something to print the traceback exit_gracefully() Is this frowned upon? You still handle the error and you know where it happened and what happened. Anything wrong with this? I don't like the idea of my system crashing for any reason. -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: smtplib needs me to put from/to headers in the message?
Dennis Lee Bieber wrote: > On Wed, 23 Aug 2006 20:23:57 +, Tobiah <[EMAIL PROTECTED]> declaimed > the following in comp.lang.python: > > >> Thank you for the valuable clarification, and the pointers. >> I will look into the RFC's. I wonder though, whether that >> will solve my main question, which is why the python.org >> example mandates that I specify 'from' and 'to' twice. >> >> > > Though it might be useful to extend the library with an optional > parameter that specifies "parse message for addresses to use"... > I actually agree with this, not to say that it wouldn't be difficult to write that function, but I think it would be a definite nicety. I've always thought that the best way to do it would be to construct the message in its entirety, this would include to, from, etc. Then whatever smtp library is available simply takes that message and a server and handles the rest. I don't think that the current functionality should be stripped out or anything, but that this other "interface" exist for a much more intuitive emailing system. -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
trackin down hard to find bugs
Hey all, I have several scripts currently in a production environment. Every two to three weeks or so our systems end up crashing. Unfortunately the traceback's aren't enough for me to debug the problem, so I'm having an issue trying to determine what to do next. The other issue is that these jobs are all run from various cron tasks from a server, so I don't have a console to attach pdb to when this system dies. So far I'm not having any luck reproducing the issue either, so I need to bring in the big guns somehow. When I played with C in college i remember that it was possible to compile the code with debugging support, then when something crashed I could load the core dump to gdb and work on the problem a little more. Is there any way to do this with python? Are there any other suggestions for getting the state of the system after an exception so that I could piece things together and try to solve this thing? TIA! -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Help me use my Dual Core CPU!
Cameron Laird wrote: > In article <[EMAIL PROTECTED]>, > Tim Golden <[EMAIL PROTECTED]> wrote: > . > . > . > >> | I know threads won't help (in CPython at least) so I'm investigating >> | other types of concurrency which I might be able to use. I really like >> | the PyLinda approach >> >> I find it very elegant. I only wish I had some real-world use for it! >> > . > . > . > Me, too. I'd love to talk over Linda with other aficionados, > and/or hunt together for an excuse to use her/it. > Funny you should mention that. I've had PyLinda opened in firefox for a couple days waiting to be read about. I have a large distributed system I'm hoping to use PyLinda for, or at least I hope PyLinda will help me with. -c -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
How to convert a timedelta object to a string?
Basically I used the datetime module and timedelta objects to calculate a difference between two times. Now I'm trying to figure out how I make that time delta a string HH:MM:SS to show elapsed time. I've spent tons of time looking at the module's documentation but I'm not seeing how those methods will help me. Can anyone help me with this? Here's what I'm trying to do, assum that resultsDict is a dictionary of objects that have various pieces of information including a start time and a stop time that are strings which I extracted from a file in an earlier part of the program. I use regEx to split up the hours, minutes, and seconds and create timedelta objects, which I then subtract to get the different. What I need is to get the value in duration as HH:MM:SS as a string: #There's lots of other code above this, but I just need help with this part timeRegex = re.compile(r'(\d\d):(\d\d):(\d\d)') for key in resultsDict.keys(): if resultsDict[key].stop != None: parsedTime = timeRegex.match(resultsDict[key].start) startDelta = datetime.timedelta(seconds=int(parsedTime.group(3)), minutes=int(parsedTime.group(2)), hours=int(parsedTime.group(1))) parsedTime = timeRegex.match(resultsDict[key].stop) stopDelta = datetime.timedelta(seconds=int(parsedTime.group(3)), minutes=int(parsedTime.group(2)), hours=int(parsedTime.group(1))) duration = stopDelta - startDelta Thanks, -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: How to convert a timedelta object to a string?
tobiah wrote: > Carl J. Van Arsdall wrote: > >> Basically I used the datetime module and timedelta objects to calculate >> a difference between two times. Now I'm trying to figure out how I make >> that time delta a string HH:MM:SS >> >> >> > > Oddly enough, str(tdobject) does what you want. > > Well, kinda, although I can work with this, str(tdobject) returns a string that looks like: -1 day, 7:34:32 Granted, I can parse that, I was looking for a way to just get the actual duration. But for now I'll just parse the string, thanks. -c -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Upgrading question
[EMAIL PROTECTED] wrote: > hi > just want to get some opinions > i am working in unix and my Python version is 2.4.1 > I would like to upgrade my version to 2.5 stable. but i have many > scripts already running using 2.4.1. Can i just install 2.5 onto > another directory, and then just change every shebang (#! in first > line) in my script to the new directory? That way, my script will also > run correectly right? > thanks > > Read: http://docs.python.org/dev/whatsnew/section-other.html about code problems other than that change the link in /usr/bin/python to point to 2.5 instead of 2.4 -c -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: a different question: can you earn a living with *just* python?
John Salerno wrote: > It's a nice thought that a person can earn a living programming with > Python, which is fun enough to use just for its own sake. But for > someone like me (i.e. no programming experience) it's always a little > disheartening to see that most (if not all) job descriptions that ask > for Python still require some C/C++ or other language knowledge. I > suppose this isn't an issue if you studied CS in college, because you > would have been exposed to many languages. > > Well, the thing is, once you can program in one language you can pretty much move around to other languages pretty easily (unless they are twisted, but rarely have I been asked to use something crazy). At my job I've written about 30 lines of C code in the 2 years I've been here. The rest has been almost entirely Python. Occasionally other languages come up, you just kinda do what you need to on the fly or get a book when you get stuck. There's so many examples on the web you can go from zero to productive fairly quickly. Of course with anything it takes time to get good with a language, I'm not say you could be an expert but you should be able to look at some C examples and gather most of what you need to construct a simple program. From there hit your references up and go for it. > But what if you are an expert Python program and have zero clue about > other languages? Can you still earn a living that way, or do most/all > companies require multiple language proficiency? > My team mates know strictly Python (and a little shell) and do just fine. I know very few programmers who truly only know a single language. In fact the fact that I know Python has gotten me calls from several recruiters recently. Seems like python is on the up and up. Anyhow, the longer you code the more languages you'll come into contact with. It just usually works out that way. Between school and work I've learned Python, Perl, C, C++, Fortran, and Java. I was only taught C, once I had those concepts down I found usually I just needed to learn the syntax. > (I suppose this isn't exactly a Python problem, either. I'm sure even > companies that don't use Python still use multiple languages. Maybe it > isn't a good idea to focus entirely on a single language, depending on > the job at hand.) > Be familiar with lots of tools, get good at the ones you feel will help you get the job done. No doubt you'll be most proficient in the one you need to use the most, but you'll always need to move around, its definitely a good job skill to have. -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: a different question: can you earn a living with *just* python?
Roy Smith wrote: > In article <[EMAIL PROTECTED]>, > "Michele Simionato" <[EMAIL PROTECTED]> wrote: > > >> John Salerno wrote: >> >>> But what if you are an expert Python program and have zero clue about >>> other languages? >>> >> Python is not a trivial language (think of generators, decorators, >> metaclasses, etc) >> > > This is, unfortunately, a sad commentary on the evolution of the language > over the past few years. One of the things that made Python so attractive > was how simple it was, and yet powerful. > > Things like decorators and metaclasses certainly add power, but they add > complexity too. It's no longer a simple language. > Well, I think a simple language is a language that makes the simple things simple and some of the complex things simple. But I also like a language where, if I need it, I can tap into some raw power and do that really wild stuff. So its simple to use if that's all you need yet offers the complexity to get things done that a purely "simple" language can't do. I'd say its as simple as you want it to be :) -c -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Is anyone using Python for embedded applications?
I'm aware of a couple python projects for embedded systems. I am currently considering using Python on an embedded platform to develop a simple application as a personal project, mostly to see if it will work. I was wondering if anyone here was using python for anything of that nature? For those that are involved in these types of projects, how does development in python differ for embedded projects versus a non-embedded project? Is there a community standard technique or style for this type of development (i.e. heavy in OO design? commonly used patterns?) Are there any good tools to assist for this type of development environment? Oh, and if anyone has opinions/facts on why python should not be used in an embedded platform, I'd like to know that too. I'm somewhat familiar with pythons needs on a system, but there are a number of things I am not aware of. Thanks to everyone for their input! -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release -- http://mail.python.org/mailman/listinfo/python-list
Re: One module per class, bad idea?
Isaac Rodriguez wrote: >> Yes, it would be a bad idea. =) >> > > Saying it is a bad idea and not explaining why will not help anyone. I > would like you to elaborate on why it is a bad idea to have one file > per class. > A module per class makes a lot of sense in some cases, or rather, make your module your class (look at the singleton pattern). I actually like to structure all of my code like this, it helps me keep things organized and separated. I guess i'm not sure why it would ever be a really bad idea, maybe if you had really small classes? -c -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: One module per class, bad idea?
Carl Banks wrote: > Carl J. Van Arsdall wrote: > >> Isaac Rodriguez wrote: >> >>>> Yes, it would be a bad idea. =) >>>> >>>> >>> Saying it is a bad idea and not explaining why will not help anyone. I >>> would like you to elaborate on why it is a bad idea to have one file >>> per class. >>> >>> >> A module per class makes a lot of sense in some cases, or rather, make >> your module your class (look at the singleton pattern). I actually like >> to structure all of my code like this, it helps me keep things organized >> and separated. >> > > I don't understand. Are you saying you organize your code to be a > bunch of modules masquerading as singleton classes? (When I was a > young'n, we called that "procedure oriented programming" :) > Well, when you have a class you want instantiated once and only once (i.e. a singleton) you can do it in python really easily vs a langauge like C++ (and i'll explain). So, in C++ when you want to create a singleton you need to go through a little bit of work to make sure that the constructor (this might be a naive method, but i never claimed to be elite) checks to make sure another class of the same type has never been created. Singletons get more elaborate than this, but I'm just trying to illustrate a point. In python a model can be treated as a singleton class. Think of a module's global variables as the variables of the class and the function in the module that operate on those variables as the class' methods. When you instantiate a module you get one and only one, the global variables more or less stay static throughout your execution (or at least, the way I use it). So say you have a module that acts as a centralized ressource (how about some piece of IO)? In a language like java you will have one and only one interface to that IO (i mean it depends on the IO, bear with me its just to illustrate a point), if a user tries to make a new class it will just return a reference to the singleton class that already exists. So python modules are really useful for this type of construct. I'm not saying you want to make all of your modules singleton classes, but its kind of a hidden benefit of modules. You get a singleton and don't have to really do any extra work :) Again, i've never done anything incredibly complex, but i've found this useful a couple of times. >> I guess i'm not sure why it would ever be a really bad >> idea, maybe if you had really small classes? >> > > Or maybe you have really big classes. > > The potential problem with one module per class is one of missed > opportunity: namely the missed opportunity for a higher-level > organization. > > Classes are rarely, probably never, a good way to organize code at the > highest levels. It's better to organize code into subsystems of some > sort. It's very rare that each class is an independent subsystem unto > itself; most high-level subsystems would encompass many classes (as > well as other code). When you strictly obey a one class per module > rule, you lose the possibility of using the module as a means of > organizing subsystems (and subsubsystems, and so on). > > Now, I think this is the best way to use modules, but you don't need to > use modules to do get higher-level organization; you could use packages > instead. It's a pain if you're working on two different classes in the > same system you have to keep switching files; but I guess some people > prefer to switch files rather than to scroll for some reason. > Yea, you have a good point. I don't have a lot of experience with packages, but I've also never written anything so large that i've had more than 5-10 modules. I'll spend some time looking into it, thanks! > I'd say as long as you use package system, and not just a flat > modulespace, it's not fundamentally disorganized to use a one class per > module rule. In the end, it probably comes down to what you prefer, > but I think most people would prefer not to obey a one class per module > rule. > > > Carl Banks > > -Carl V. -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Is anyone using Python for embedded applications?
Hendrik van Rooyen wrote: > >> >> > > It depends a *lot* on what is meant by "embedded" : > Ha, very true > This definition seems to cover everything from: > - a cut down PC in a non standard box, through > - a processor in a Washing Machine, to > - a bare PIC processor in a Burglar Alarm... > We are considering now are mobile phone and pocket pc-esque devices. I know that several phones with arm processors are running an arm version of linux now, we're thinking how reasonable it might be to run python applications on a phone, and which python might best apply. Is there a good way to determine the "minimum requirements" for a python application? I'd imagine these might be something like the min requirements of python (cpython, pymite, etc) + additional requirements placed by the design of the application. Is there a good way to study a python application and figure that type of thing out? > I think the main hassles are that you need something big enough > to run a reasonable OS in, and it must support being programmed in C, > (which most things do), and it must have some MegaBytes of RAM > loose for the Python. (where more is merrier) > > Trying to run this on say an AVR or 8031 with a 64k address space and > a scarcity of RAM, will, to say the least, be a bit of a challenge... > > As far as the OS goes, Linux is probably the best bet, if you can get it to > fit in your hardware - It has been ported to ARM type processors from > various companies (Atmel springs to mind), and is free, which is a help > in a personal project. You could of course also roll your own kernel, > which will be good practice, as with a limited set of peripherals its not > THAT hard to do, but its a bit far away from Python - :- ) > Yea, we are thinking on the more robust end of the embedded side. So a system capable of running Linux or Windows CE (or something similar) > What display device are you going to use, or is it going to be a webserver > sitting on a power over ethernet link? > > I haven't actually taken the plunge myself yet to put Python on any of the > hardware we make, as it seems to add a lot of overhead to a simple device > - but I come from the bottom up, as it were, and the idea is intriguing, > as I in fact discovered Python because it is embedded in a GPS module > we were evaluating for building into a device - so I will follow your > progress with interest... > > -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: automatically grading small programming assignments
[EMAIL PROTECTED] wrote:
> [EMAIL PROTECTED] wrote:
> Then on your PC you can
>
>> run a script that loads each of such programs, and runs a good series
>> of tests, to test their quality...
>>
> What happens if someone-- perhaps not even someone in the class-- does
> some version of os.system('rm -Rf /') ?
>
>
The system administrator should make sure that student user accounts (or
the auto testing account) doesn't have access to that. Probably should
make sure that user applications only get a limited amount of memory too.
--
Carl J. Van Arsdall
[EMAIL PROTECTED]
Build and Release
MontaVista Software
--
http://mail.python.org/mailman/listinfo/python-list
Re: Problems with import of modules
Ilias Lazaridis wrote:
> I am within a directory
>
> \doc\template\
>
> I launch script.py
>
> within this script.py, I like to import a module from the doc directory.
>
> this here does not work:
>
> form ..\..\module_name import this_one
>
Well, if you are in linux you can do this easily by changing your
PYTHONPATH environment variable, either by changing it explicitely or by
editing it in your .rc files to append the /doc directory.
Although I don't know specifically where this variable might be if you
are using windows, in either case(windows or linux), you can alter this
from python using sys.path
import sys
sys.path.append("/doc")
Hope that helps,
-carl
> how do I go back in the directory hierarchy to import something?
>
> If this is not possible:
>
> How can I modify the python search-path from within the script, thus it
> contains the doc directory?
>
> .
>
>
--
Carl J. Van Arsdall
[EMAIL PROTECTED]
Build and Release
MontaVista Software
--
http://mail.python.org/mailman/listinfo/python-list
C Extended module init/cleanup
I'm extending Python with C, it seems as though initialization is easy enough with an init() function that all C extended python modules must provide that can serve as a sort of "module constructor". Can C extended python modules provide any kind module destructor, so that whenever the module is unloaded some cleanup is run? Thanks, carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: List of files to be opened
yawgmoth7 wrote:
> Hello, I am currently writing a script that requires a few different
> files to be opened, and examined. What I need to be able to do is use
> something like:
>
> filelist = os.system("ls")
>
There are a number of ways you can do this, one method would be to do a loop
for file in os.popen('ls').readlines():
if os.path.isfile(file):
open(file) #or whatever
there's also an os.listdir i believe, check python's documentation
>
> I cannot think of a way to do this, I could put them in a list of
> something of the sort. But that still does not solve the problem of
> opening them one by one.
>
> Thanks for all the advice and help.
> --
> gurusnetwork.org
> Gurus'Network - Are you a guru?
>
--
Carl J. Van Arsdall
[EMAIL PROTECTED]
Build and Release
MontaVista Software
--
http://mail.python.org/mailman/listinfo/python-list
Re: List of files to be opened
Ken Starks wrote:
> yawgmoth7 wrote:
>
>
>> Hello, I am currently writing a script that requires a few different
>> files to be opened, and examined. What I need to be able to do is use
>> something like:
>>
>> filelist = os.system("ls")
>> >
>> I cannot think of a way to do this, I could put them in a list of
>> something of the sort. But that still does not solve the problem of
>> opening them one by one.
>>
>> Thanks for all the advice and help.
>> --
>> gurusnetwork.org
>> Gurus'Network - Are you a guru?
>>
>
> os.walk is your friend. Its has wonderful functionality.
>
>
Don't you mean os.path.walk ?
> The documentation is in the subsection 'Files and Directories' of the os
> module, and there are a couple of examples.
>
> Global Module index --> os --> 'Files and Directories' (Section 6.1.4)
> -->Bottom of page
>
--
Carl J. Van Arsdall
[EMAIL PROTECTED]
Build and Release
MontaVista Software
--
http://mail.python.org/mailman/listinfo/python-list
Re: 2-dimensional data structures
anthonyberet wrote: > Hello again - rather a newbie here... > > I want to work on a sudoku brute-forcer, just for fun. > > I am considering different strategies, but first I need to decide on the > data-structure to use for the progress/solution grid. > > This being a square, I would have used a 9x9 2-dimensional array in my > teenage years back in the 80's, using BASIC. > > What is the equivalent way to store data in python? - It isn't obvious > to me how to do it with lists.] > Well, you could do a list of lists, or a tuple of tuples, or a combination thereof. For example: val = matrix[indexA][indexB] -carl > 'scuse me for being thick - but give me a little pointer and I will do > the rest. > -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Thread imbalance
Tuvas wrote: > I have a program running several threads. One of them must be done > every (Specified time, usually 1 second). The whole point to having a > thread is do this. However, I've noticed the following. When there is > another part of the program that is active, this thread slips into > disuse, ei, it's only ran about once every 4-5 seconds, or perhaps it > waits for a lul in the computing process. How can I ensure that this > does not happen? This thread uses little processing power, so it could > be set to a high priority, if there is a way to do this. Thanks! > > I think that might be difficult using python threads simply based on how python controls the global interpreter lock. One suggestion I might have is to have python release the global interpreter lock more frequently, you can read about the global interpreter lock here: http://docs.python.org/api/threads.html You might also be able to use some timer/condition construct in combination with this, something like Thread A: if watchdogTimer(): conditionVar.aquire() conditionVar.notify(threadB) conditionVar.release() ThreadB: while(1): conditionVar.aquire() conditionVar.wait() functionToDoSomething() This is pseudo python of course, if you need to know about these objects I would suggest consulting the python manual. -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Object Oriented vs Pythonic Code, and Pythonic standards
It seems the more I come to learn about Python as a langauge and the way its used I've come across several discussions where people discuss how to do things using an OO model and then how to design software in a more "Pythonic" way. My question is, should we as python developers be trying to write code that follows more of a python standard or should we try to spend our efforts to stick to a more traditional OO model? For example, in C++ I might make a file that defines a class and all its methods, at which point I create an object and do things with it. My interpretation of what is "Pythonic" would be instead of creating a class I would just define functions and maybe some variables global to a module. At this point, I import the module and just make function calls. There are many similarities here, but the difference is that in python I don't feel as though I would define a class, I would just treat the python module as a class instead, especially if it was a type of object that I would only need a single instance of. The second question that arises from Pythonism is, has the community drafted a standard for quality "Pythonic" code? Thanks, carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Threads and Interpreter death
I've been toying with threads a lot lately and I've noticed that if a scripting error occurs in a thread the thread dies, but not the process that spawned the thread. Is python supposed to behave this way or is this type of behavior accidental? Thanks, Carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: is socket thread safe?
Steve Horsley wrote: > [EMAIL PROTECTED] wrote: > >> thread1: >> while 1: >> buf = s.read() >> process(buf) >> >> thread2: >> while 1: >> buf = getdata() >> s.write(buf) >> >> > > It is safe, but watch out for this gotcha: If thread B calls > s.close() while thread A is blocked in s.read(), thread A will > never return from the read. My preferred solution is to set > socket timeout to a few seconds, and loop checking a status flag > so I know when to quit. > > I think a better thing would be to use something like a condition object to tie the two threads together and not use any polling loops. i.e. consumer goes to sleep while data buffer is empty, producer produces and signals condition object, consumer wakes up and consumes. To take this a step further, you have a status flag that is set to something like QUIT or CONSUME and when the condition is triggered wake up, then examine the status flag to determine if the consumer should then quit, consume, or whatever else you'd want your consumer thread to do. -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: is socket thread safe?
Bryan Olson wrote: > Carl J. Van Arsdall wrote: > >> Steve Horsley wrote: >> >> >>> [EMAIL PROTECTED] wrote: >>> >>> >>> >>>> thread1: >>>> while 1: >>>> buf = s.read() >>>> process(buf) >>>> >>>> thread2: >>>> while 1: >>>> buf = getdata() >>>> s.write(buf) >>>> >>> It is safe, but watch out for this gotcha: If thread B calls s.close() >>> while thread A is blocked in s.read(), thread A will never return from >>> the read. My preferred solution is to set socket timeout to a few >>> seconds, and loop checking a status flag so I know when to quit. >>> > > Certainly one needs timeouts to avoid hanging should the remote > side stop. Sockets don't have a read() method, and hanging on > recv() doesn't seem to have anything to do with close(). > > I didn't find any definitive doc, so I tested using Python > sockets on Linux (Debian/Ubuntu current) and WinXP. A recv() > started before the close() will block/return just as if > close() were never called. The close() neither triggers recv() > to abort, nor prevents it from receiving data and detecting > shutdown. > > > >> I think a better thing would be to use something like a condition object >> to tie the two threads together and not use any polling loops. >> >> i.e. consumer goes to sleep while data buffer is empty, producer >> produces and signals condition object, consumer wakes up and consumes. >> > > I can infer two producer-consumer relationships from the example, > but they don't allow a condition object; the writer's consumer and > the reader's producer are on the remote end of the socket. The > socket will already handle the blocking and wake-up. > > >> To take this a step further, you have a status flag that is set to >> something like QUIT or CONSUME and when the condition is triggered wake >> up, then examine the status flag to determine if the consumer should >> then quit, consume, or whatever else you'd want your consumer thread to do. >> > > What problem are you trying to solve? Normal socket sending, receiving, > and shutdown discipline work fine. When the writer is done writing, it > should call sock.shutdown(socket.SHUT_WR). When the reader gets zero > bytes from recv(nonzero), that means the remote end has finished > writing, so the reader may call sock.shutdown(socket.SHUT_RD). > Doh! I read the word threads and got carried away not even realizing sockets. Well, looks like today i'll just have to remember to drink my coffee :-D -c -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: is socket thread safe?
Jean-Paul Calderone wrote: > On Wed, 15 Feb 2006 12:59:03 -0800, "Carl J. Van Arsdall" <[EMAIL PROTECTED]> > wrote: > >> Steve Horsley wrote: >> >>> [EMAIL PROTECTED] wrote: >>> >>> >>>> thread1: >>>> while 1: >>>> buf = s.read() >>>> process(buf) >>>> >>>> thread2: >>>> while 1: >>>> buf = getdata() >>>> s.write(buf) >>>> >>>> >>>> >>> It is safe, but watch out for this gotcha: If thread B calls >>> s.close() while thread A is blocked in s.read(), thread A will >>> never return from the read. My preferred solution is to set >>> socket timeout to a few seconds, and loop checking a status flag >>> so I know when to quit. >>> >>> >>> >> I think a better thing would be to use something like a condition object >> to tie the two threads together and not use any polling loops. >> >> i.e. consumer goes to sleep while data buffer is empty, producer >> produces and signals condition object, consumer wakes up and consumes. >> > > What makes you think process isn't implemented to notify a condition, and > getdata isn't implemented to wait on one? :) > > I think it was the "loop checking status flag" comment that threw me off *shrug* 8) -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Does python have an internal list/dictionary of functions?
Python Gurus: Let me elaborate a bit more on this question. Basically, I want to know if there is some data structure in python that maps a string function name to an address of a function or something to that nature. If this is confusing, let me describe what I want to do and see if anyone has any ideas. basically we have: >>>def functA(): ... pass >>> functA And what I'd like to do is: >>>__internalFuncDict__['functA'] This is just for a little experimental project of mine, any help or pointers to the proper pages in the manual would be greatly appreciated. Basically, I know that I can create something like this if I have to but I was very much hoping that something already existed somewhere that I could get to via python or by writing a C extension. Thanks in advance! -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Does python have an internal data structure with functions imported from a module?
Alright, I attempted to post this question yesterday but I don't see it as showing up, so I apologize in advance if this is a double post. Python Gurus: Let me elaborate a bit more on this question. Basically, I want to know if there is some data structure in python that maps a string function name to an address of a function or something to that nature. If this is confusing, let me describe what I want to do and see if anyone has any ideas. basically we have: >>>def functA(): ... pass >>> functA And what I'd like to do is: >>>__internalFuncDict__['functA'] This is just for a little experimental project of mine, any help or pointers to the proper pages in the manual would be greatly appreciated. Basically, I know that I can create something like this if I have to but I was very much hoping that something already existed somewhere that I could get to via python or by writing a C extension. Thanks in advance! -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Parallel Python
Just as something to note, but many HPC applications will use a combination of both MPI and threading (OpenMP usually, as for the underlying thread implementation i don't have much to say). Its interesting to see on this message board this huge "anti-threading" mindset, but the HPC community seems to be happy using a little of both depending on their application and the topology of their parallel machine. Although if I was doing HPC applications, I probably would not choose to use Python but I would write things in C or FORTRAN. What I liked about python threads was that they were easy whereas using processes and IPC is a real pain in the butt sometimes. I don't necessarily think this module is the end-all solution to all of our problems but I do think that its a good thing and I will toy with it some in my spare time. I think that any effort to making python threading better is a good thing and I'm happy to see the community attempt to make improvements. It would also be cool if this would be open sourced and I'm not quite sure why its not. -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Tools Designing large/complicated applications
For those of you that work on larger applications but still code in python... do your development teams use any tools to facilitate the design? (i'm not asking about editors here, i'm really asking about software design tools) Are these the same tools you would use to help engineer software in another language? Is there anyone here who is forced to use a tool to design python software that completely hates it? Why do you hate it? Thanks! -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
The reliability of python threads
Hey everyone, I have a question about python threads. Before anyone goes further, this is not a debate about threads vs. processes, just a question. With that, are python threads reliable? Or rather, are they safe? I've had some strange errors in the past, I use threading.lock for my critical sections, but I wonder if that is really good enough. Does anyone have any conclusive evidence that python threads/locks are safe or unsafe? Thanks, Carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: The reliability of python threads
[EMAIL PROTECTED] wrote: > Carl> Does anyone have any conclusive evidence that python threads/locks > Carl> are safe or unsafe? > > In my experience Python threads are generally safer than the programmers > that use them. ;-) > Haha, yea, tell me about it. The whole GIL thing made me nervous about the locking operations happening truly atomically and not getting weird. Thanks for ensuring me that i'm just nuts :) -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: The reliability of python threads
Chris Mellon wrote: > On 24 Jan 2007 18:21:38 GMT, Nick Maclaren <[EMAIL PROTECTED]> wrote: > >> [snip] >> >> > > I'm aware of the issues with the POSIX threading model. I still stand > by my statement - bringing up the problems with the provability of > correctness in the POSIX model amounts to FUD in a discussion of > actual problems with actual code. > > Logic and programming errors in user code are far more likely to be > the cause of random errors in a threaded program than theoretical > (I've never come across a case in practice) issues with the POSIX > standard. > Yea, typically I would think that. The problem I am seeing is incredibly intermittent. Like a simple pyro server that gives me a problem maybe every three or four months. Just something funky will happen to the state of the whole thing, some bad data, i'm having an issue tracking it down and some more experienced programmers mentioned that its most likely a race condition. THe thing is, I'm really not doing anything too crazy, so i'm having difficult tracking it down. I had heard in the past that there may be issues with threads, so I thought to investigate this side of things. It still proves difficult, but reassurance of the threading model helps me focus my efforts. > Emphasizing this means that people will tend to ignore bugs as being > "the fault of POSIX" rather than either auditing their code more > carefully, or avoiding threads entirely (the second being what I > suspect your goal is). > > As a last case, I should point out that while the POSIX memory model > can't be proven safe, concrete implementations do not necessarily > suffer from this problem. > Would you consider the Linux implementation of threads to be concrete? -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: The reliability of python threads
Aahz wrote: > [snip] > > My response is that you're asking the wrong questions here. Our database > server locked up hard Sunday morning, and we still have no idea why (the > machine itself, not just the database app). I think it's more important > to focus on whether you have done all that is reasonable to make your > application reliable -- and then put your efforts into making your app > recoverable. > Well, I assume that I have done all I can to make it reliable. This list is usually my last resort, or a place where I come hoping to find ideas that aren't coming to me naturally. The only other thing I thought to come up with was that there might be network errors. But i've gone back and forth on that, because TCP should handle that for me and I shouldn't have to deal with it directly in pyro, although I've added (and continue to add) checks in places that appear appropriate (and in some cases, checks because I prefer to be paranoid about errors). > I'm particularly making this comment in the context of your later point > about the bug showing up only every three or four months. > > Side note: without knowing what error messages you're getting, there's > not much anybody can say about your programs or the reliability of > threads for your application. > Right, I wasn't coming here to get someone to debug my app, I'm just looking for ideas. I constantly am trying to find new ways to improve my software and new ways to reduce bugs, and when i get really stuck, new ways to track bugs down. The exception won't mean much, but I can say that the error appears to me as bad data. I do checks prior to performing actions on any data, if the data doesn't look like what it should look like, then the system flags an exception. The problem I'm having is determining how the data went bad. In tracking down the problem a couple guys mentioned that problems like that usually are a race condition. From here I examined my code, checked out all the locking stuff, made sure it was good, and wasn't able to find anything. Being that there's one lock and the critical sections are well defined, I'm having difficulty. One idea I have to try and get a better understanding might be to check data before its stored. Again, I still don't know how it would get messed up nor can I reproduce the error on my own. Do any of you think that would be a good practice for trying to track this down? (Check the data after reading it, check the data before saving it) -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Mounting shares with python
Marcpp wrote:
> Hi, when i mount a share with python...
>
> os.system ("mount -t smbfs -o username=nobody ...")
>
> the problem is that I'll to be root.
> Have a comand to send a root password...?
> I've tried
>
> os.system ("su")
> os.system ("the password")
>
> but it doesn't works.
>
>
I do a lot of that type of stuff. Just setup sudo access for that user
and have it state no password.
I also do that stuff on remote machines, similarly, setup ssh such that
you don't need to use a password. It will work :)
-c
--
Carl J. Van Arsdall
[EMAIL PROTECTED]
Build and Release
MontaVista Software
--
http://mail.python.org/mailman/listinfo/python-list
Re: The reliability of python threads
Hendrik van Rooyen wrote: > "Carl J. Van Arsdall" <[EMAIL PROTECTED]> wrote: > > >> [snip] >> > > Are you 100% rock bottom gold plated guaranteed sure that there is > not something else that is also critical that you just haven't realised is? > 100%? No, definitely not. I know myself, as I explore this option and other options, I will of course be going into and out of the code, looking for that small piece I might have missed. But I'm like a modern operating system, I do lots of things at once. So after being unable to solve it the first few times, I thought to pose a question, but as I pose the question that never means that I'm done looking at my code and hoping I missed something. I'd much rather have this be my fault... that means I have a much higher probability of fixing it. But i sought to explore some tips given to me. Ah, but the day I could be 100% sure, that would be a good day (hell, i'd go ask for a raise for being the best coder ever!) > This stuff is never obvious before the fact - and always seems stupid > afterward, when you have found it. Your best (some would say only) > weapon is your imagination, fueled by scepticism... > > Yea, seriously! >> try and get a better understanding might be to check data before its >> stored. Again, I still don't know how it would get messed up nor can I >> reproduce the error on my own. >> >> Do any of you think that would be a good practice for trying to track >> this down? (Check the data after reading it, check the data before >> saving it) >> > > Nothing wrong with doing that to find a bug - not as a general > practice, of course - that would be too pessimistic. > > In hard to find bugs - doing anything to narrow the time and place > of the error down is fair game - the object is to get you to read > some code that you *know works* with new eyes... > > I really like that piece of wisdom, I'll add that to my list of coding mantras. Thanks! > I build in a global boolean variable that I call trace, and when its on > I do all sort of weird stuff, giving a running commentary (either by > print or in some log like file) of what the programme is doing, > like read this, wrote that, received this, done that here, etc. > A bare useful minimum is a "we get here" indicator like the routine > name, but the data helps a lot too. > > Yea, I do some of that too. I use that with conditional print statements to stderr when i'm doing my validation against my test cases. But I could definitely do more of them. The thing will be simulating the failure. In the production server, thousands of printed messages would be bad. I've done short but heavy simulations, but to no avail. For example, I'll have a couple systems infinitely loop and beat on the system. This is a much heavier load than the system will ever normally face, as its hit a lot at once and then idles for a while. The test environment constantly hits it, and I let that run for several days. Maybe a longer run is needed, but how long is reasonable before determining that its something beyond my control? > Compared to an assert, it does not stop the execution, and you > could get lucky by cross correlating such "traces" from different > threads. - or better, if you use a queue or a pipe for the "log", > you might see the timing relationships directly. > Ah, store the logs in a rotating queue of fixed size? That would work pretty well to maintain control on a large run, thanks! > But this in itself is fraught with danger, as you can hit file size > limits, or slow the whole thing down to unusability. > > On the other hand it does not generate the volume that a genuine > trace does, it is easier to read, and you can limit it to the bits that > you are currently suspicious of. > > Programming is such fun... > Yea, I'm one of those guys who really gets a sense of satisfaction out of coding. Thanks for the tips. -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Off-Topic Posts
Yea, that guy sucks. Is there a list mod who can just ban this guy? Sean Schertell wrote: > Hey Genius -- I'm probably further to the left and even more > vehemently opposed to the Bush/Cheney regime than you are. But could > you *please* take your unwelcome ranting elsewhere? You're not > winning any converts here. And you're alienating your ideological > allies to boot. Give it a rest, m'kay? > > Sean > > P.S. Printing letters in all-caps and using extra exclamation points > doesn't necessarily make your argument any more compelling. > > > > > On Jan 27, 2007, at 8:02 AM, [EMAIL PROTECTED] wrote: > > >> What did Dick Faced Cheney told Honorable Senator Patrick Leahy ? >> "Fuck >> yourself". >> So much for politeness and vulgarity at the top level. >> >> Proof: >> http://www.capitolhillblue.com/news2/2007/01/the_madness_of.html >> >> On Jan 26, 2:53 pm, [EMAIL PROTECTED] wrote: >> >>> Yeah, listen to wise counsel of klein. As a member of conquered races >>> and still under occupation, namely Jewish, French, German, Japanese, >>> Korean ... dont mess in the crimes of the anglo-saxon yanks. You >>> should >>> remember the beating you got from the Anglo-Saxon Yanks and just keep >>> quiet ... As for the lunatics off their meds, its also true that 911 >>> was a grand optical illusion and thus within the charter of this >>> group >>> to study ... After all, one of its proponent is the former head of >>> Star >>> Wars, Dr Bob Bowman from MIT and Caltech ... Do you have a degree >>> from >>> those prestigious institutes or did you manage to get any of your >>> kids >>> into one of them ??? As for vulgarity, its the mark of patriotism ... >>> the constitutional amendments guarantee it and Herbert Walker Bush >>> was >>> a strongly suspected pedophile ... there is a video called conspiracy >>> of silence, watch it on video.google.com and download with before the >>> fascist bureau of incompetence bastards (running around like crazy >>> rats >>> to confiscate pentagon videos on that day in destruction of evidence) >>> try to sabotage the google servers ... they are part of the crime and >>> complicit in it. If they were competent, we would see one consistent >>> theory coming out from that bureau to explain all significant >>> events on >>> that day. Now dont forget the SHITTY job done by FEMA and NIST. There >>> are gaping holes in their work. >>> >>> Hey Kinch, if you are jewish, did you forget that England was the >>> first >>> country to expel the Jews. >>> >>> If you forget your status as conquered races, then you are in >>> delusion, >>> not anyone else. >>> >>> The final balance of power provided against the anglos will be by >>> Slavs. Hurray to Vladimir Putin, and Mikhail Khodorkovsky in a >>> Siberian >>> jail. >>> >>> On Jan 26, 1:13 pm, Richard J Kinch <[EMAIL PROTECTED]> wrote: >>> >>> >>>> alex writes: >>>> >>>>> I have only just joined this group, but have noticed a >>>>> significant rise >>>>> in off-topic posts that really detract from what is otherwise an >>>>> excellent place to learn and talk about optics.This has nothing >>>>> to do with sci.optics. Recent days have seen this stuff >>>>> >>>> being broadcast mechanically into all sorts of groups. >>>> >> -- >> http://mail.python.org/mailman/listinfo/python-list >> > > DataFly.Net > Complete Web Services > http://www.datafly.net > > -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: The reliability of python threads
Hendrik van Rooyen wrote: > [snip] >> could definitely do more of them. The thing will be >> > > When I read this - I thought - probably your stuff is working > perfectly - on your test cases - you could try to send it some > random data and to see what happens - seeing as you have a test > server, throw the kitchen sink at it. > > Possibly "random" here means something that "looks like" data > but that is malformed in some way. Kind of try to "trick" the > system to get it to break reliably. > > I'm sorry I can't be more specific - it sounds so weak, and you > probably already have test cases that "must fail" but I don't > know how to put it any better... > Well, sometimes a weak analogy is the best thing because it allows me to fill in the blanks "How can I throw a kitchen sink at it in a way I never have before" And away my mind goes, so thank you. -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: The reliability of python threads
Steve Holden wrote: > [snip] > > Are you using memory with built-in error detection and correction? > > You mean in the hardware? I'm not really sure, I'd assume so but is there any way I can check on this? If the hardware isn't doing that, is there anything I can do with my software to offer more stability? -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: The reliability of python threads
John Nagle wrote: > Aahz wrote: > >> In article <[EMAIL PROTECTED]>, >> Carl J. Van Arsdall <[EMAIL PROTECTED]> wrote: >> My point is that an app that dies only once every few months under load >> is actually pretty damn stable! That is not the kind of problem that >> you are likely to stimulate. >> > > This has all been so vague. How does it die? > Well, before operating on most of the data I perform type checks, if the type check fails, my system flags an exception. Now i'm in the process of finding out how the data went bad. I gotta wait at this point though, so I was investigating possibilities so I could find a new way of throwing the kitchen sink at it. > It would be useful if Python detected obvious deadlock. If all threads > are blocked on mutexes, you're stuck, and at that point, it's time > to abort and do tracebacks on all threads. You shouldn't have to > run under a debugger to detect that. > > Then a timer, so that if the Global Python Lock > stays locked for more than N seconds, you get an abort and a traceback. > That way, if you get stuck in some C library, it gets noticed. > > Those would be some good basic facilities to have in thread support. > I agree. That would be incredibly useful. Although doesn't this spark up the debate on threads killing threads? From what I understand, this is frowned upon (and was removed from java because it was dangerous). Although I think that if there was a master or control thread that watched the state of the system and could intervene, that would be powerful. One way to do this could be to use processes, and each process could catch a kill signal if it appears to be stalled, although I am absolutely sure there is more to it than that. I don't think this could be done at all with python threads though, but as a fan of python threads and their ease of use, it would be a nice and powerful feature to have. -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Any python scripts to do parallel downloading?
Michele Simionato wrote:
> On Jan 31, 5:23 pm, "Frank Potter" <[EMAIL PROTECTED]> wrote:
>
>> I want to find a multithreaded downloading lib in python,
>> can someone recommend one for me, please?
>> Thanks~
>>
>
> Why do you want to use threads for that? Twisted is the
> obvious solution for your problem, but you may use any
> asynchronous framework, as for instance the good ol
>
Well, since it will be io based, why not use threads? They are easy to
use and it would do the job just fine. Then leverage some other
technology on top of that.
You could go as far as using wget via os.system() in a thread, if the
app is simple enough.
def getSite(site):
os.system('wget %s',site)
threadList =[]
for site in websiteList:
threadList.append(threading.Thread( target=getSite,args=(site,)))
for thread in threadList:
thread.start()
for thread in threadList:
thread.join()
> Tkinter:
>
> """
> Example of asynchronous programming with Tkinter. Download 10 times
> the same URL.
> """
>
> import sys, urllib, itertools, Tkinter
>
> URL = 'http://docs.python.org/dev/lib/module-urllib.html'
>
> class Downloader(object):
> chunk = 1024
>
> def __init__(self, urls, frame):
> self.urls = urls
> self.downloads = [self.download(i) for i in range(len(urls))]
> self.tkvars = []
> self.tklabels = []
> for url in urls:
> var = Tkinter.StringVar(frame)
> lbl = Tkinter.Label(frame, textvar=var)
> lbl.pack()
> self.tkvars.append(var)
> self.tklabels.append(lbl)
> frame.pack()
>
> def download(self, i):
> src = urllib.urlopen(self.urls[i])
> size = int(src.info()['Content-Length'])
> for block in itertools.count():
> chunk = src.read(self.chunk)
> if not chunk: break
> percent = block * self.chunk * 100/size
> msg = '%s: downloaded %2d%% of %s K' % (
> self.urls[i], percent, size/1024)
> self.tkvars[i].set(msg)
> yield None
> self.tkvars[i].set('Downloaded %s' % self.urls[i])
>
> if __name__ == '__main__':
> root = Tkinter.Tk()
> frame = Tkinter.Frame(root)
> downloader = Downloader([URL] * 10, frame)
> def next(cycle):
> try:
> cycle.next().next()
> except StopIteration:
> pass
> root.after(50, next, cycle)
> root.after(0, next, itertools.cycle(downloader.downloads))
> root.mainloop()
>
>
> Michele Simionato
>
>
--
Carl J. Van Arsdall
[EMAIL PROTECTED]
Build and Release
MontaVista Software
--
http://mail.python.org/mailman/listinfo/python-list
Re: Any python scripts to do parallel downloading?
Jean-Paul Calderone wrote: > On 31 Jan 2007 12:24:21 -0800, Carl Banks <[EMAIL PROTECTED]> wrote: > >> Michele Simionato wrote: >> >>> On Jan 31, 5:23 pm, "Frank Potter" <[EMAIL PROTECTED]> wrote: >>> >>>> I want to find a multithreaded downloading lib in python, >>>> can someone recommend one for me, please? >>>> Thanks~ >>>> >>> Why do you want to use threads for that? Twisted is the >>> obvious solution for your problem, >>> >> Overkill? Just to download a few web pages? You've got to be >> kidding. >> > > Better "overkill" (whatever that is) than wasting time re-implementing > the same boring thing over and over for no reason. > How is that a waste of time? I wrote the script to do it in 10 lines. What is a waste of time is learning a whole new technology/framework to do a simple task that can be scripted in 4 minutes. -c -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Any python scripts to do parallel downloading?
Jean-Paul Calderone wrote: > [snip] >> >> > > You're right. Learning new things is bad. My mistake. > > Jean-Paul > That isn't what I said at all. You have to look at it from a cost/benefit relationship. Its a waste of time/money to learn something complex to do something simple. For the simple things, use a simple solution. KISS. When he has an application that would require something more complex, it would be at that point he should consider using it for a project. Unless the OP has a desire to learn this technology, then more power to him. I, however, do not believe that would be the best approach for a simple problem. Knowing the appropriate tool for the job is a trait of an good engineer. -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Any python scripts to do parallel downloading?
Jean-Paul Calderone wrote: > On Wed, 31 Jan 2007 15:13:59 -0800, "Carl J. Van Arsdall" <[EMAIL PROTECTED]> > wrote: > >> Jean-Paul Calderone wrote: >> >>> [snip] >>> >>>> >>> You're right. Learning new things is bad. My mistake. >>> >>> Jean-Paul >>> >>> >> That isn't what I said at all. You have to look at it from a >> cost/benefit relationship. Its a waste of time/money to learn something >> complex to do something simple. For the simple things, use a simple >> solution. KISS. When he has an application that would require >> something more complex, it would be at that point he should consider >> using it for a project. Unless the OP has a desire to learn this >> technology, then more power to him. I, however, do not believe that >> would be the best approach for a simple problem. >> >> Knowing the appropriate tool for the job is a trait of an good engineer. >> >> > > You are assuming that he already knows how to use threads, and so there > is no investment required for a threaded solution. In my experience, it's > much safer to assume the opposite. _Even_ (often _especially_ when a > threaded solution is explicitly requested. > I have a bit more confidence in python threads, but that takes us back to the age old debate on this list. So we agree to disagree. -c -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Finding cpu time spent on my program
[EMAIL PROTECTED] wrote: > On Feb 5, 2:37 am, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> > wrote: > >> I am trying to measure the time the processor spends on some >> operation, and I want this measure to not depend on the current load >> of the machine. But doing the following prints different values each >> time a run. >> >> import time >> c1 = time.clock() >> for x in xrange(0xF): pass >> >> c2 = time.clock() >> print "Time spent is %f" % (c2-c1) >> >> Is there a way to measure the number of cpu cycles spent on my program >> alone irrespective of the current load etc of the machine. >> There's an performance testing suite that you might also find useful. Check out TAU: Tuning and Analysis Utilities toolkit (simple google search will take you to the page). You can get some serious numbers using that thing and they have python support as well as some tools for automatically profiling an entire application. Check it out. -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: multithreading concept
Paul Boddie wrote: > [snip] > > Take a look at the Python Wiki for information on parallel processing > with Python: > > http://wiki.python.org/moin/ParallelProcessing > What a great resource! That one is book marked for sure. I was wondering if anyone here had any opinions on some of the technologies listed in there. I've used a couple, but there are some that I've never seen before. In particular, has anyone used rthread before? It looks like something I may use (now orwhen it matures), are there opinions on it? Under the cluster computing section, has anyone tried any of the other technologies? I've only used Pyro and i love it, but I'd like opinions and experiences with other technologies if anyone has anything to say. -c -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: multithreading concept
S.Mohideen wrote: > I would like to add my problem in this thread. > I have a network application in Python which sends and recv using a single > socket. > There is a dictionary on which I store/read data values. I want to seperate > the send and recv functionality on two different processes so that the > parallel execution becomes fast. Is there any way to do so, so that the > Dict's consitency is not lost(able to read & write) and also the performance > improves. I am looking upon the MPI4Py module to see if it does the job for > me. Any ideas would be appreciated. > Well, from your description so far I think that MPI is going to be a bit of overkill. I think you should consider threads or processors with shared memory/objects (POSH). Then take a look at a producer/consumer program to see how it works, that should get you to where you need to go. HTH -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: multithreading concept
Bjoern Schliessmann wrote: > [snip] > What makes you think that'll be faster? > > Remember: > - If you have one CPU, there is no parallelity at all. > - If you do have multiple CPUs but only one network device, there is > no parallel networking. > > Not necessarily, if he's on a full duplex ethernet connection, then there is some parallelity he can take advantage of. He has upstream and downstream. -c -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Re: Java Developer Exploring Python
Jarek Zgoda wrote: > [EMAIL PROTECTED] napisał(a): > > >> Is Python actively developed and supported on Linux? Would it be a >> viable option for cross-platform application development? >> > > Yeas and yeas. Yeas. (That's how we pronounce "yes" here, in Poland, > East Europe). > > >> Can anyone recommend an open source IDE for Python that runs on Linux? >> My favorite IDE: vi > > Komodo. But in most cases you don't need any IDE, just good text editor > would make it go. Like Kate, jEdit or similar. > -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
Conditions vs Events
Le Monde De Python, I've been working a lot with python threads from the threading module. Specifically, when is it better to use a condition object vs an event object? It seems to me that, at least superficially, I can just about use these two mechanisms interchangeably. The only real difference I can see is that I can call isSet() on an event in the case that an event is triggered before wait() is called. Are there any differences? Also, does calling wait reset some type of "notify" in a condition object like it does in an event object? Anyhow, thanks for your help! -carl -- Carl J. Van Arsdall [EMAIL PROTECTED] Build and Release MontaVista Software -- http://mail.python.org/mailman/listinfo/python-list
