First, you need to understand that if you return a file using the
RETURNSTDOUT (or RETURNSTDERR/RETURNFILE) options on a PROCESS START
request (or a <process> element in a STAX job), the contents of the file
are read and then sent as a string string. If the file is large, this can
take up a lot of memory and can consure a lot of CPU by the unmarshall
process. Sub-section "8.12,2 Start" that talks about the PROCESS service
in the STAF User's Guide at
http://staf.sourceforge.net/current/STAFUG.htm#HDRPROCSRV contains the
following information that recommends that you don't use the RETURNxxx
options to return large files and talks about how you can specify a
maximum size for a returned file:
Since the entire contents of returned files are stored in the result
string, if you attempt to return the contents of a very large file, you
may run out of memory so it is not recommended that you use the
RETURNSTDOUT, RETURNSTDERR, or RETURNFILE options to return large files.
To help prevent this problem, you can specify a maximum size for a file
returned by this request by setting the MAXRETURNFILESIZE operational
parameter in the STAF configuration file on the machine where the process
is run, or by setting the STAF/MaxReturnFileSize variable in the request
variable pool of the handle that submitted the request. The lowest of
these two values is used as the maximum return file size (not including 0
which indicates no limit).
So, for large files created by processes, instead of returning their
contents this way, you may want to use the FS COPY request to copy the
contents to the STAX service machine after the process completes (if you
really need the file to reside on the STAX service machine) as the FS COPY
request copies the files in "chunks" so it doesn't take up much
CPU/memory. Then you could read the local file using Python via a
<script> element if you needed to. Or, it might be more efficient if you
can leave the process stdout file on the remote machine and if you simply
need to check the contents of the file for some information, you couldrun
another process on the remote machine that does this.
--------------------------------------------------------------
Sharon Lucas
IBM Austin, luc...@us.ibm.com
(512) 286-7313 or Tieline 363-7313
anas ameziane <anas.amezi...@gmail.com>
01/29/2010 08:50 AM
To
staf-users@lists.sourceforge.net
cc
Subject
[staf-users] Stack Overflow ??
Hello to all
I use STAF / STAX to automate some tasks.
I developed a python class that allows me to set the stdout of each
process node in XML.
the problem that it works well, except when stdout is large (7 mega
octects).
java.exe consumes 99.9% of my CPU, and I see no error in the logs (and
staf jvm).
----------------
<script>
proc = Process(procName)
</script>
...
<process name="'copygrep'">
<location>location</location>
<command mode="'shell'">'%s/copygrep.sh'%(dir)</command>
<parms>'%s'%(params)</parms>
<console use="'same'"/>
<stderr mode="'stdout'"/>
<returnstdout/>
</process>
<script>
proc._end([RC,STAXResult[0][1]])
</script>
...
-----------------------------------------------------
<script>
class Process:
def __init__(self,name):
self.name = name
self.stdout = 'NaN'
self.rc ='-1'
...
def _end(self,resultTab):
if resultTab[0] != '' :
self.rc = str(resultTab[0])
if resultTab[1] != '' :
import re
self.stdout = resultTab[1]
...
def getXMLResult(self):
from xml.dom.minidom import Document
xmlDoc = Document()
...
stdout = xmlDoc.createElement("stdout")
stdout.appendChild(xmlDoc.createTextNode(self.stdout))
process.appendChild(stdout)
...
</script>
it is an overflow , a bug or something else ?
any ideas ?
Anas A.
Thanks,
------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the
business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
_______________________________________________
staf-users mailing list
staf-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/staf-users
------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
_______________________________________________
staf-users mailing list
staf-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/staf-users
