Thanks for the additional information. Now I think I see what the problem
is.
Getting error "Error accepting on server socket, RC: 24" means "Too many
open files". This often indicates that you opening lots of files in your
STAX job(s) and not closing them. A side effect of this is once you can't
open any more files (because you've reached the maximum set for your
operating system), you start getting RC 21 submitting a STAF service
request that tries to open a file. You can use the "lsof" command to see
what files are open on Unix machines. You can also list the open files
for the STAX JVM by getting it's PID (Process ID) and then run "lsof -p
<STAX JVM pid>". Note that you can get the PID for the STAX JVM by
looking in the STAX JVM log or you can get it by running "STAF staxmachine
HANDLE LIST HANDLES LONG" and see what the PID is for the handle named
STAF/Service/STAX.
Read the following entries in the STAF/STAX FAQ to help resolve these
issue:
- 3.1.13 Explain "Error accepting on server socket, socket RC: 24" at
http://staf.sourceforge.net/current/STAFFAQ.htm#Error%20accepting%20on%20server%20socket,%20socket%20RC:%2024
- 4.1.7 Explain "RC 21 submitting a STAX EXECUTE request" at
http://staf.sourceforge.net/current/STAFFAQ.htm#RC%2021%20submitting%20a%20STAX%20EXECUTE%20request
I've cut n pasted these entries from the STAF/STAX FAQ here:
3.1.13 Explain "Error accepting on server socket, socket RC: 24"
This error can occur when your system has run out of file descriptors
because too many files have been opened and you've exceeded the maximum
number of open files allowed by your system. Anytime a file is opened, a
file descriptor must be used to allow the program to access the data. When
the STAF connection provider calls accept() to accept a connection on a
socket and it's failing to open the socket and setting errno to 24 (this
is the error code on Unix systems). In /usr/include/asm/errno.h on Linux
systems, errno 24 is defined as:
#define EMFILE 24 /* Too many open files */
EMFILE The per-process limit of open file descriptors has been reached.
To see what files are open on Unix machines, use the lsof command. It
lists the names of the open files and the pids of the processes that
opened them. You can list the open files for a process by getting it's pid
and then run lsof -p <pid>command>.
If you you are using the STAX service on this machine, it's possible that
a STAX job could be responsible for opening files and not closing them.
Refer to Explain "RC 21 submitting a STAX EXECUTE request" for more
information on this.
4.1.7 Explain "RC 21 submitting a STAX EXECUTE request"
You can get this error if your system has exceeded the maximum number of
files that can be open. Check your JVM log and see if you have lots of the
following error logged:
Error accepting on server socket, socket RC: 24
Check your STAX jobs and see if they use Jython's open() function to open
a file in a <script> element but aren't using Jython's close() function to
close the file. If the file is opened within a loop construct (or if the
function this code is in is called many times or the STAX job is run many
times), many files could be left open, even after the STAX job(s)
complete. Unlike Python, Jython requires that you explicitly close files
(otherwise they could be left open for a long, long time until they are
garbage collected).
You can run command lsof -p <STAX JVM PID> to see the names of the files
that were opened by the STAX service and STAX jobs (which can help you
determine which STAX job(s) are leaving files open).
Refer to Explain "Error accepting on server socket, socket RC: 24" for
more information on how to debug this problem by finding out what files
are
--------------------------------------------------------------
Sharon Lucas
IBM Austin, luc...@us.ibm.com
(512) 286-7313 or Tieline 363-7313
Alkesh Shah <salke...@yahoo.com>
02/04/2009 12:31 PM
Please respond to
Alkesh Shah <salke...@yahoo.com>
To
Sharon Lucas/Austin/i...@ibmus
cc
Subject
Re: [staf-users] Max number of STAX threads
Hi Sharon,
Sorry for the delayed response from my end.
First of all Thanks a lot for the information provided.
We were initially using Staf version 3.2.0. So, as suggested, we changed
it to version 3.3.1. However, we are still receiving the same error. Below
is the information that you had requested -
1) On what operating system are you running the STAX service on?
- RHEL 5.2, memory - 4 GB, 32-bit OS
2) What version of STAF are you running the STAX service with? e.g. STAF
staxMachine MISC VERSION. Be sure you're running with STAF V3.2.3 or
later if your operating system is Linux, preferably the current version of
STAF, STAF V3.3.2,
- 3.3.1
3) Provide the line(s) in your STAF.cfg file where you are registering the
STAX service.
- service stax library JSTAF execute
{STAF/Config/STAFRoot}/services/stax/STAX.jar \
OPTION JVMName=TESTSTAX OPTION J2=-Xmx1536m
There is also an inhouse service that uses 1024m for the JVM
4) What is the exact error message that you are seeing?
- 20090203-12:24:46;2358016912;00000100;Caught STAFException in
JSTAF.STAFServiceAcceptRequest(), Endpoint:
tcp://staxmachinehostn...@6500, Service: inhouse, Request: request string,
Exception: STAFConnectionProviderException, Location:
/opt/dev/autobuild/build/src/staf/stafif/STAFConnectionProviderInlImpl.cpp(219),
Text: STAFConnectionProviderConnect: Error performing test read on
connected endpoint: recv() RC=22, Error code: 22
5) Are there any errors in the STAX JVM log? (e.g.
/usr/local/staf/data/STAF/lang/java/jvm/<JVMName>/JVMLog.1 (where
<JVMName> is STAFJVM1 or something else if you used the JVMName option
when registering the STAX service in your STAF.cfg file.
- No errors in the JVM log. The above error was seen on the terminal
from where I started STAF service.
We are using an inhouse service that connects to an api (that is capable
of handling large number of threads). Currently, I am trying out with 300
threads and for this instance of the run, it was able to execute 136
threads properly. And the above mentioned error message is there for 164
times (that is the remaining threads).
However, this is not consistent. I tried out a couple of combinations
STAX Jvm - 1536 MB
INhouse service JVM(with the default memory, no J2 option specified) -
STAF Thread size - 2 MB
No. of threads created - 228
STAX JVM - 1536 MB
Inhouse service JVM - 1024mb
STAF Thread size - 2 MB
No. of threads created - 242
In both the above cases, there was slightly different error noticed -
20090204-10:17:38;13630352;00000100;Error accepting on server socket,
socket RC: 24
20090204-10:17:38;1990794128;00000100;Caught STAFException in
JSTAF.STAFServiceAcceptRequest(), Endpoint:
tcp://staxmachinehostn...@6500, Service: inhouse service, Request:
request, Exception: STAFConnectionProviderException, Location:
/opt/dev/autobuild/build/src/staf/stafif/STAFConnectionProviderInlImpl.cpp(219),
Text: STAFConnectionProviderConnect: Error creating socket: socket()
RC=24, Error code: 22
and in the JVM log there was a message
STAX Log failed with RC 21 and Result level: error logRequest: LOG
MACHINE LOGNAME :10:STAX_Job_1 LEVEL error MESSAGE :250:STAXLogError
signal raised. Continuing job.
There are no other applications running on this RHEL5.2.
Please advise me on this.
Thank you.
Regards,
Alkesh Shah
From: Sharon Lucas <luc...@us.ibm.com>
To: Alkesh Shah <salke...@yahoo.com>
Cc: staf-users@lists.sourceforge.net
Sent: Wednesday, January 28, 2009 3:47:53 PM
Subject: Re: [staf-users] Max number of STAX threads
The maximum number of threads that can be created on a machine depends on
your available memory and workload.
1) On what operating system are you running the STAX service on?
2) What version of STAF are you running the STAX service with? e.g. STAF
staxMachine MISC VERSION. Be sure you're running with STAF V3.2.3 or
later if your operating system is Linux, preferably the current version of
STAF, STAF V3.3.2,
3) Provide the line(s) in your STAF.cfg file where you are registering the
STAX service.
4) What is the exact error message that you are seeing?
5) Are there any errors in the STAX JVM log? (e.g.
/usr/local/staf/data/STAF/lang/java/jvm/<JVMName>/JVMLog.1 (where
<JVMName> is STAFJVM1 or something else if you used the JVMName option
when registering the STAX service in your STAF.cfg file.
In STAF V3.2.3, we decreased default thread stack size for STAF threads on
Linux to help resolve OutOfMemory issues when STAF creates a new thread
(Bug #1688297). We decreased the stack size for Linux to 4M because on
new versions of Linux the default stack size for Linux pthreads is quite
large (e.g. 8M or 10M) and if you create a lot of threads (e.g. by
submitting lots of STAF commands simultaneoulsy and/or recieving process
completion notifications simultaneously), you can eat up a lot of memory
as virtual memory used is determined by:
managed heap + native heap + (number of threads * thread stack size)
Depending on your available memory and workload, you could set it lower.
For example, the following sets the STAF thread stack size to 2M. Note
that you have to restart STAFProc with that environment variable set:
export STAF_THREAD_STACK_SIZE=2048
See
http://sourceforge.net/tracker/index.php?func=detail&aid=1688297&group_id=33142&atid=407381
for more information on this bug fix.
--------------------------------------------------------------
Sharon Lucas
IBM Austin, luc...@us.ibm.com
(512) 286-7313 or Tieline 363-7313
Alkesh Shah <salke...@yahoo.com>
01/21/2009 02:20 PM
To
staf-users@lists.sourceforge.net
cc
Subject
[staf-users] Max number of STAX threads
Hi,
I am using STAX 3.3.1 and wanted to execute staf commands parallely. I
wanted to know the maximum number of threads that can be created for
executing the staf commands.
I have been receiving socket timeout exceptions in STAF beyond approx. 250
threads.
Thanks in advance for the help.
Regards,
Alkesh Shah
------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword
_______________________________________________
staf-users mailing list
staf-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/staf-users
------------------------------------------------------------------------------
Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM)
software. With Adobe AIR, Ajax developers can use existing skills and code to
build responsive, highly engaging applications that combine the power of local
resources and data with the reach of the web. Download the Adobe AIR SDK and
Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com
_______________________________________________
staf-users mailing list
staf-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/staf-users
