Hi Joe, Thanks, those are exactly what we sometimes experience... and yes, the parallel and paralleliterate tags definitely increased the issue. I noticed that early on and switch to sequences or used STAXGlobal copies (including using copy.deepcopy() ) for some of the parallel statements and that mostly got rid of the issues within those particular sections. Now we are dealing with mostly just "randomly" ocurring ones.
Did you actually declare the structure definition as a global? We haven't
tried that. We use various other globals and try to deal with the scoping
issues with globals and actual copies where necessary (since python
defaults to copying by reference), but we didn't globalize the structure
definition itself...
Thanks, good to know we are on the right track!
Paul E.
|------------>
| From: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|Joseph J Veilleux/Lexington/IBM
|
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| To: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|Paul Ellsworth/San Jose/i...@ibmus
|
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Cc: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|staf-users@lists.sourceforge.net
|
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Date: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|03/05/2010 06:11 AM
|
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Subject: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|Re: [staf-users] Python variable scope ... between jobs?
|
>--------------------------------------------------------------------------------------------------------------------------------------------------|
Paul --
Yes, I have seen this kind of error and the pattern of failure seemed to be
just as you observed. In particular the complaint that an object doesn't
have a specific attribute.
My experience suggests that you are on the right track -- the problem is
with variable scoping. There are some very odd scoping rules for python
variables in the context of a STAX job so you have to work through all the
variations of STAX function scope, where your variables are declared,
whether or not you use the STAXGlobal() wrapper class, etc.
This was a few months ago that I was encountering these problems so it's a
little difficult to remember the precise details of the solution but I'm
pretty sure it was to declare/populate my main control structures in my
main STAX function, with the structure declared as a STAXGlobal. Using that
strategy I haven't had any problems recently.
So -- I think you're on the right track.
Oh -- one other thing -- I think your observation about the multiple
threads is on-target too. I found that some of these problems happened when
I was using a <paralleliterate> element to do something simultaneously on
each of four target machines, and in particular if I got a STAX execution
error (e.g. Python syntax error or NameError or something like that) on one
of the threads. I believe under such circumstances I would see the STAX job
hang, as though it wasn't recognizing that all the threads were finished.
I was more focused on finding a solution that would work rather than
thoroughly diagnosing and documenting the problem, so I don't have any
detailed notes on what works and what doesn't... sorry.
----------------------------------------------------------------------------------------
Joe Veilleux
IBM/Lotus Domino Server Quality Engineering
550 King Street
Littleton MA 01460
Email: joeveill...@us.ibm.com
Paul
Ellsworth/San
Jose/i...@ibmus To
staf-users@lists.sourceforge.net
03/04/2010 03:24 cc
PM
Subject
[staf-users] Python variable
scope ... between jobs?
Hello,
I sometimes have strange python (/jython... I will probably refer to it as
python though...) variable issues - e.g., it values from one variable end
up in another variable for whatever reason, or a class somehow gets messed
up and Python complains that an instance of class X does not have attribute
Y when I know for a fact that it does :)
This happens *much* more frequently when many jobs are running (10 or
more... large jobs; if they all completed, they would generate over 100,000
user logs, not sure how many non-user logs, use around 20 XML files that
can have up to 4000 lines in them, etc).
I'm working through possible variable scope issues... question is: is there
a possibility that the python variables/scope/maybe shared memory is
getting mixed up or leaked amongst the many running jobs?
Not necessarily saying this is happening but asking if it is possible while
I explore various reasons it may be happening... :)
It also seems to happen more frequently when multiple jobs are running
through the same XML sections (and thus the same python/jython scripting)
at the same time...
Thanks!
Paul
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
staf-users mailing list
staf-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/staf-users
<<inline: graycol.gif>>
<<inline: ecblank.gif>>
<<inline: pic07471.gif>>
------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev
_______________________________________________ staf-users mailing list staf-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/staf-users
