thanks, requested info in-line: ----- "Sharon Lucas" <luc...@us.ibm.com> wrote: > Could you provide the following additional information that may give > me some more clues as to what's happening? > > 1) Your STAF.cfg file
# Turn on tracing of internal errors and deprecated options trace enable tracepoints "error deprecated" # Enable TCP/IP connections interface tcp library STAFTCP # Set default local trust trust machine local://local level 5 # Add default service loader serviceloader library STAFDSLS MACHINENICKNAME master TRUST LEVEL 5 MACHINE pernod.office.sproutsys.com TRUST LEVEL 5 MACHINE localhost.localdomain #SERVICE teststafia LIBRARY JSTAF EXECUTE /usr/local/staf/lib/services/teststafia/TestSTAFia.jar OPTION JVMName=stafJVM OPTION JVM=/usr/java/jdk1.6.0_03/bin/java SERVICE teststafia LIBRARY JSTAF EXECUTE /usr/local/staf/lib/services/teststafia/TestSTAFia.jar OPTION JVMName=stafJVM OPTION JVM=/usr/java/jdk1.6.0_16/bin/java SERVICE STAX LIBRARY JSTAF EXECUTE /usr/local/staf/services/stax/STAX.jar OPTION JVM=/usr/java/jdk1.5.0_14/jre/bin/java OPTION JVMName=STAX OPTION J2=-Xmx384m #SERVICE STAX LIBRARY JSTAF EXECUTE /usr/local/staf/services/stax/STAX.jar SERVICE EVENT LIBRARY JSTAF EXECUTE /usr/local/staf/services/stax/STAFEvent.jar SET MAXQUEUESIZE 10000 trust level 5 default > 2) Contents of the /usr/local/staf/install.properties file version=3.3.4 platform=linux-amd64 architecture=64-bit installer=STAFInst file=STAF334-linux-amd64.tar osname=Linux osversion=* osarch=amd64 > 3) Output from the following commands when run on your STAX service > machine: > > STAF local VAR LIST STAF/Config/BootDrive : / STAF/Config/CodePage : UTF-8 STAF/Config/ConfigFile : /usr/local/staf/bin/STAF.c fg STAF/Config/DefaultAuthenticator : none STAF/Config/DefaultInterface : tcp STAF/Config/InstanceName : STAF STAF/Config/Machine : master.colo.sproutsys.com STAF/Config/MachineNickname : master STAF/Config/Mem/Physical/Bytes : 2111148032 STAF/Config/Mem/Physical/KB : 2061668 STAF/Config/Mem/Physical/MB : 2013 STAF/Config/OS/MajorVersion : 2.6.23.1-21.fc7 STAF/Config/OS/MinorVersion : #1 SMP Thu Nov 1 20:28:15 EDT 2007 STAF/Config/OS/Name : Linux STAF/Config/OS/Revision : x86_64 STAF/Config/Processor/NumAvail : 4 STAF/Config/Sep/Command : ; STAF/Config/Sep/File : / STAF/Config/Sep/Line : STAF/Config/Sep/Path : : STAF/Config/STAFRoot : /usr/local/staf STAF/Config/StartupTime : 20091007-19:52:34 STAF/DataDir : /usr/local/staf/data/STAF STAF/Env/_ : /usr/local/staf/bin/STAFPr oc STAF/Env/CLASSPATH : /usr/local/staf/lib/JSTAF. jar:/usr/local/staf/samples/demo/STAFDemo.jar: STAF/Env/HISTSIZE : 1000 STAF/Env/HOME : /home/nparrish STAF/Env/HOSTNAME : master.colo.sproutsys.com STAF/Env/INPUTRC : /etc/inputrc STAF/Env/LANG : en_US.UTF-8 STAF/Env/LD_LIBRARY_PATH : /usr/local/staf/lib: STAF/Env/LOGNAME : root STAF/Env/LS_COLORS : no=00:fi=00:di=00;34:ln=00 ;36:pi=40;33:so=00;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=0 0;32:*.cmd=00;32:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.cs h=00;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31: *.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.c pio=00;31:*.jpg=00;35:*.gif=00;35:*.bmp=00;35:*.xbm=00;35:*.xpm=00;35:*.png=00;3 5:*.tif=00;35: STAF/Env/MAIL : /var/spool/mail/nparrish STAF/Env/PATH : /usr/local/staf/bin:/usr/l ocal/staf/bin:/opt/krobix/tools-3/bin:/usr/local/staf/bin:/usr/local/staf/bin:/u sr/kerberos/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin STAF/Env/PWD : /tmp STAF/Env/SHELL : /bin/bash STAF/Env/SHLVL : 1 STAF/Env/STAF_INSTANCE_NAME : STAF STAF/Env/STAFCONVDIR : /usr/local/staf/codepage STAF/Env/SUDO_COMMAND : /etc/rc.d/init.d/staf star t STAF/Env/SUDO_GID : 9002 STAF/Env/SUDO_UID : 1011 STAF/Env/SUDO_USER : nparrish STAF/Env/TERM : xterm STAF/Env/USER : root STAF/Version : 3.3.4.1 testenvs/betan/client1/hostname : clark testenvs/betan/client2/hostname : clark testenvs/betan/goldDB/dbname : test [....] there are something like a thousand of these testenvs/* variables which are used in our automation infrastructure. > STAF local MISC LIST SETTINGS Connection Attempts : 2 Connect Retry Delay : 1000 Interface Cycling : Enabled Maximum Queue Size : 10000 Maximum Return File Size : 0 Initial Threads : 10 Thread Growth Delta : 1 Data Directory : /usr/local/staf/data/STAF Default Interface : tcp Default Authenticator : none Result Compatibility Mode: Verbose > STAF local SERVICE LIST REQUESTS LONG Request# Source HName H# Date-Time Target Service Request ---------- ------------- ------- --- --------- ------------- ------- ---------- 28 local://local STAF/SE 6 20091007- master.colo.s QUEUE GET WAIT RVICE/E 19:52:38 proutsys.com VENT 3598328 tcp://hefty.c fcsl_w1 345 20091009- master.colo.s sem wait event olo.sproutsys 43 16:02:05 proutsys.com fcsl-clus ....@6500 ter 3598475 tcp://hefty.c fcsl_w2 345 20091009- master.colo.s sem wait event olo.sproutsys 46 16:02:13 proutsys.com fcsl-clus ....@6500 ter 1831657032 tcp://devnode snowbal 235 20091027- master.colo.s sem wait event 4.colo.sprout l3_w1 578 11:26:25 proutsys.com snowball3 sys....@6500 -cluster 1831913759 tcp://devnode snowbal 235 20091027- master.colo.s sem wait event 4.colo.sprout l3_w3 584 11:26:52 proutsys.com snowball3 sys....@6500 -cluster 1832073747 tcp://devnode snowbal 257 20091027- master.colo.s sem wait event 3.colo.sprout l2_w4 905 11:27:09 proutsys.com snowball2 sys....@6500 -cluster 1832099403 local://local STAX/Jo 484 20091027- master.colo.s QUEUE GET WAIT b/4810 7 11:27:12 proutsys.com 1832101422 tcp://devnode snowbal 257 20091027- master.colo.s sem wait event 3.colo.sprout l2_w2 899 11:27:12 proutsys.com snowball2 sys....@6500 -cluster 1832120071 tcp://devnode snowbal 257 20091027- master.colo.s sem wait event 3.colo.sprout l2_w1 896 11:27:14 proutsys.com snowball2 sys....@6500 -cluster 1832264160 tcp://devnode snowbal 235 20091027- master.colo.s sem wait event 4.colo.sprout l3_w4 587 11:27:29 proutsys.com snowball3 sys....@6500 -cluster 1832434108 local://local STAX/Jo 509 20091027- master.colo.s QUEUE GET WAIT b/5053 4 11:27:47 proutsys.com 1832442127 local://local STAX/Jo 507 20091027- master.colo.s QUEUE GET WAIT b/5039 6 11:27:48 proutsys.com 1832442841 tcp://devnode snowbal 218 20091027- master.colo.s respool request po 5.colo.sprout l4_w1 230 11:27:48 proutsys.com ol snowbal sys....@6500 l4-cluster 1832453259 tcp://devnode snowbal 218 20091027- master.colo.s sem wait event 5.colo.sprout l4_w2 233 11:27:49 proutsys.com snowball4 sys....@6500 -cluster 1832460312 local://local STAX/Jo 297 20091027- master.colo.s QUEUE GET WAIT b/2954 3 11:27:50 proutsys.com 1832518328 local://local STAF/Cl 509 20091027- master.colo.s SERVICE LIST REQUE ient 6 11:27:56 proutsys.com STS LONG > STAF local HANDLE LIST HANDLES LONG Handle Handle Name State Last Used Date-Time PID ------ ------------------------------- ---------- ------------------- ----- 1 STAF_Process InProcess 20091027-11:28:42 14481 2 STAF/Service/STAFServiceLoader1 InProcess 20091007-19:59:47 14481 3 STAF/Service/teststafia Registered 20091027-11:28:43 14502 4 STAF/Service/STAX Registered 20091027-11:28:41 14533 5 STAF/Service/LOG InProcess 20091027-11:28:42 14481 6 STAF/SERVICE/EVENT Registered 20091007-19:52:38 14565 14 STAF/Service/RESPOOL InProcess 20091027-11:28:21 14481 2973 STAX/Job/2954 Registered 20091027-11:28:02 14533 4847 STAX/Job/4810 Registered 20091027-11:28:34 14533 5076 STAX/Job/5039 Registered 20091027-11:28:20 14533 5097 STAX/Job/5054 Registered 20091027-11:28:42 14533 5098 STAF/Client Registered 20091027-11:28:43 11273 > STAF local PROCESS LIST H# Command Start Date-Time End Date-Time Return Code -- --------------------------- ----------------- ----------------- ------------ 7 /usr/local/staf/bin/STAFReg 20091007-19:52:38 20091007-19:52:41 0 > STAF local SERVICE LIST Name Library Executable ---------- ---------- ------------------------------------------------------ DELAY <Internal> <None> DIAG <Internal> <None> ECHO <Internal> <None> EVENT JSTAF /usr/local/staf/services/stax/STAFEvent.jar FS <Internal> <None> HANDLE <Internal> <None> HELP <Internal> <None> LIFECYCLE <Internal> <None> LOG STAFLog <None> MISC <Internal> <None> PING <Internal> <None> PROCESS <Internal> <None> QUEUE <Internal> <None> RESPOOL STAFPool <None> SEM <Internal> <None> SERVICE <Internal> <None> SHUTDOWN <Internal> <None> STAX JSTAF /usr/local/staf/services/stax/STAX.jar TESTSTAFIA JSTAF /usr/local/staf/lib/services/teststafia/TestSTAFia.jar TRACE <Internal> <None> TRUST <Internal> <None> VAR <Internal> <None> > STAF local STAX LIST SETTINGS { Event Machine : local Event Service Name : Event Number of Threads : 5 Process Timeout : 60000 File Caching : Enabled Max File Cache Size : 20 Max Machine Cache Size: 20 Max Return File Size : 0 Clear Logs : Disabled Log TC Elapsed Time : Enabled Log TC Num Starts : Enabled Log TC Start/Stop : Disabled Python Output : JobUserLog Python Log Level : Info Extensions : [] Extension File : <None> } > STAF local STAX LIST JOBS Job ID Job Name Start Date-Time Function ------ ------------------------------- ----------------- ---------------- 2954 longevity_3.7.xml-setup39 20091019-14:28:24 longevity 4810 longevity_3.7.xml-fcsburn 20091026-15:18:17 longevity 5039 rep_conduit_long.xml-betan 20091027-10:30:40 rep_conduit_long 5054 elff_killa_qa-884.xml-snowball4 20091027-11:28:41 main > STAF local STAX LIST FILECACHE Machine File Hits Last Hit Added Date-Time -------------- ----------------------- ---- ----------------- ----------------- poet /opt/krobix/qa/setup39/ 1 20091027-11:29:50 20091027-11:29:50 qa-test/testix/stax/ver ify_table_or_db.xml poet /opt/krobix/qa/setup39/ 4 20091027-11:29:50 20091027-10:46:05 qa-test/testix/stax/qa_ stax_lib.xml poet /opt/krobix/qa/setup39/ 1 20091027-11:29:50 20091027-11:29:50 qa-test/testix/stax/sys bench_basic.xml devnode5 /opt/krobix/qa/snowball 10 20091027-11:28:41 20091027-10:16:54 4/qa-test/testix/stax/q a_stax_lib.xml devnode5.colo. /opt/krobix/qa/snowball 1 20091027-11:28:41 20091027-11:28:41 sproutsys.com 4/qa-test/testix/stax/e lff_killa_qa-884.xml devnode3 /opt/krobix/qa/snowball 1 20091027-11:27:34 20091027-11:27:34 2/qa-test/testix/stax/q a_stax_lib.xml devnode3.colo. /opt/krobix/qa/snowball 1 20091027-11:27:33 20091027-11:27:33 sproutsys.com 2/qa-test/testix/stax/s kew_bug661-877.xml devnode5.colo. /opt/krobix/qa/snowball 1 20091027-11:23:04 20091027-11:23:04 sproutsys.com 4/qa-test/testix/stax/b ug443_create_dup-929.xm l devnode3.colo. /opt/krobix/qa/snowball 1 20091027-11:13:28 20091027-11:13:28 sproutsys.com 2/qa-test/testix/stax/S TAXUtil.xml devnode3.colo. /opt/krobix/qa/snowball 1 20091027-11:13:28 20091027-11:13:28 sproutsys.com 2/qa-test/testix/stax/q a_stax_lib.xml devnode3.colo. /opt/krobix/qa/snowball 1 20091027-11:13:27 20091027-11:13:27 sproutsys.com 2/qa-test/testix/stax/c hange_set_user_passwd-9 20.xml devnode5 /opt/krobix/qa/snowball 1 20091027-11:05:50 20091027-11:05:50 4/qa-test/testix/stax/v erify_table_or_db.xml devnode5.colo. /opt/krobix/qa/snowball 1 20091027-11:05:50 20091027-11:05:50 sproutsys.com 4/qa-test/testix/stax/b ug314_queue_replay-940. xml devnode4 /opt/krobix/qa/snowball 2 20091027-11:01:25 20091027-10:44:04 3/qa-test/testix/stax/q a_stax_lib.xml devnode4.colo. /opt/krobix/qa/snowball 1 20091027-11:01:25 20091027-11:01:25 sproutsys.com 3/qa-test/testix/stax/r estart_while_insert-933 .xml handy /data/homes/nparrish/qa 1 20091027-10:56:30 20091027-10:56:30 -test/testix/stax/verif y_table_or_db.xml handy /data/homes/nparrish/qa 2 20091027-10:56:30 20091027-10:52:19 -test/testix/stax/qa_st ax_lib.xml handy /data/homes/nparrish/qa 1 20091027-10:56:29 20091027-10:56:29 -test/testix/stax/sysbe nch_basic.xml devnode5.colo. /opt/krobix/qa/snowball 1 20091027-10:54:41 20091027-10:54:41 sproutsys.com 4/qa-test/testix/stax/v erify_table_or_db.xml devnode5.colo. /opt/krobix/qa/snowball 1 20091027-10:54:41 20091027-10:54:41 sproutsys.com 4/qa-test/testix/stax/q a_stax_lib.xml > STAF local TRACE LIST SETTINGS { Tracing To : Stdout Default Service State: Enabled Trace Points : { Info : Disabled Warning : Disabled Error : Enabled ServiceRequest : Disabled ServiceResult : Disabled ServiceError : Disabled ServiceAccessDenied: Disabled RemoteRequests : Disabled Registration : Disabled Deprecated : Enabled Debug : Disabled ServiceComplete : Disabled } Services : { DELAY : Enabled DIAG : Enabled ECHO : Enabled EVENT : Enabled FS : Enabled HANDLE : Enabled HELP : Enabled LIFECYCLE : Enabled LOG : Enabled MISC : Enabled PING : Enabled PROCESS : Enabled QUEUE : Enabled RESPOOL : Enabled SEM : Enabled SERVICE : Enabled SHUTDOWN : Enabled STAX : Enabled TESTSTAFIA: Enabled TRACE : Enabled TRUST : Enabled VAR : Enabled } } > > You may want to consider upgrading to the latest update for Sun Java > 1.5.0 for Linux x64 in case there are any issues in the JVM related to > this problem that are fixed in a later update. Update 21 for Sun Java > 1.5.0 for Linux x64 is the latest (you're using update 14). I'm not sure why I'm using 1.5.0 for STAX and 1.6.0 for our homebrewed teststafia service; more than likely teststafia required 1.6.0, but I was disinclined to change STAX (if it ain't broke, etc. etc.). any reason not to move STAX to 1.6.0 instead of upgrading 1.5.0? I'm intrigued by the TRACE service; if nothing above suggests a known problem, would it be a reasonable strategy to enabling more verbose tracing on STAX to see if he is doing something in particular to drive CPU load? thanks again, nathan ------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ staf-users mailing list staf-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/staf-users