Hi everyone I ran my test (total of 1k requests, total of 100 threads) against two nodes with default VM settings. I've just set heap size. I had about 15% of errors.
cluster.log - node1 - http://pastebin.com/cpX900Qw cluster.log - node2 - http://pastebin.com/qCSzMaU6 Running for a long time (total of 500k requests, total of 100 threads) I had about 11% of errors. In this case we can see the statistics: Jul 03, 2014 5:53:28 PM org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor report INFO: ThroughputInterceptor Report[ Tx Msg:10000 messages Sent:12.36 MB (total) Sent:12.36 MB (application) Time:7.82 seconds Tx Speed:1.58 MB/sec (total) TxSpeed:1.58 MB/sec (application) Error Msg:0 Rx Msg:10198 messages Rx Speed:0.08 MB/sec (since 1st msg) Received:12.38 MB] All session attributes are Serializable, and it's a session replication issue because if I ran my test with just one node, I had 0% of errors. Regarding "on time", just a correction: 1. first request, pick a random server and store a session object 2. second request, pick *ANY* server (chose by LB based on the cookie - it can be the same, but not necessarily) and ask for the session object. To be more clean, I've been working with a conference system. Each conference should occur in one node. So, the first request can hit any server, and from the second request should hit the node where the conference is. Thanks a lot João 2014-07-03 15:40 GMT-03:00 Filip Hanik <fi...@hanik.com>: > you mention NIO and say maxThreads, that sounds like the <Connector> > configuration, but the BIO receiver is on the cluster, and it a completely > different component that also has an applicable NIO configuration. > > are you confusing the two? > I'm saying that you should use the NIO receiver on the cluster component, > and if you do, what kind of errors do you get? > > > On Thu, Jul 3, 2014 at 12:19 PM, Mark Eggers <its_toas...@yahoo.com.invalid > > > wrote: > > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA1 > > > > João, > > > > This list has a convention of posting either inline or at the end of > > the message you're replying to. > > > > See here for mailing list notes: > > > > http://tomcat.apache.org/lists.html#tomcat-users > > > > On 7/3/2014 10:24 AM, João Sávio wrote: > > > Hello > > > > > > Some points below: > > > > > > ** What is "on time"?* In my application, a group of users should > > > always hit the same node after the first request. So, in first > > > request each group of users will receive an specific cookie, and > > > LB will perform the load balancing based on this cookie. In first > > > request, a user can hit any node, but from the second, he or she > > > should hit the same node. > > > > Hmm, so 'on time' really means that subsequent requests should hit the > > same server. > > > > If you're using sessions, Tomcat has an attribute on the Engine > > element called jvmRoute. So depending on your load balancer (and if > > you use AJP), you can use Tomcat and AJP to route traffic. In that > > case, there's no need to write a special cookie. > > > > At any rate, this doesn't sound like a clustering error per se. > > > > > > > > ** What are the errors? Test result errors?* For this test, I > > > simplified the code of my application: - first request: store one > > > object in session - second request: verify if the object is in > > > session. If it's not -> ERROR > > > > > > > So looking at the information from 'on time', the scenario should be: > > > > 1. first request, pick a random server and store a session object > > 2. second request, pick the SAME server and ask for the session object > > > > Again, I'm not seeing where this is a clustering issue per se. > > > > > ** How big are are the sessions that you're trying to replicate?* > > > - I'm using Spring MVC, and I have 3 additional objects in > > > session. They are not big (15 attributes each one) > > > > > > > And all attributes are serializable? The objects are also marked as > > serializable? > > > > > ** What's the load like on the box when you're running the tests > > > that you get errors on?* - I've experiencing this issue on BIO > > > even without load > > > > > > > I may have not phrased my question carefully. What is the CPU and > > memory situation on your test box while running the 4 Tomcat servers? > > > > I know you've trimmed down your Xms and Xmx (presumably to fit in your > > test environment), but in combination with your other JVM parameters > > could this be causing some issues? > > > > I would follow Dan's recommendation of maybe just setting Xms, Xmx, GC > > logging to see what happens. Ah, I see you're going to do that below. > > > > > ** It is preferred to use the non blocking receiver to be able to > > > grow your cluster without running into thread starvation.* - > > > That's why I've tried NIO first, but I'd like to see if BIO solve > > > my issue and if using BIO my system doesn't get too slow. > > > > I don't think speed is so much an issue here, but scalability is. NIO > > can handle multiple requests per thread, BIO cannot. > > > > > > > > > > > Now, I'll try to run my tests using NIO, default VM configuration > > > and FINER logs. > > > > Post the results when you get them. If the logs are relatively small, > > just cut and paste into the mail message. > > > > I suspect FINER is going to generate LOTS of logging and slow down > > your application. > > > > > > > > Thanks a lot João > > > > . . . . just my two cents > > /mde/ > > > > > > > > > > > 2014-07-03 14:07 GMT-03:00 Mark Eggers > > > <its_toas...@yahoo.com.invalid>: > > > > > > On 7/3/2014 9:12 AM, João Sávio wrote: > > >>>> cluster.log -> http://pastebin.com/c98WhnmG > > >>>> > > >>>> > > >>>> 2014-07-03 13:04 GMT-03:00 João Sávio <joaosa...@gmail.com>: > > >>>> > > >>>>> Hello! > > >>>>> > > >>>>> Using NIO (with channelSendOptions="4", i.e., > > >>>>> synchronous), with lightly load, my tests pass 100%. But, > > >>>>> on heavy load, not all sessions are replicated on time, and > > >>>>> I have about 20% of errors. If I increase maxThreads to > > >>>>> 400, I have about 15% of errors. > > >>>>> > > >>>>> More information: * I am not performing parallel requests > > >>>>> with same session * my cluster has 4 nodes (all in one > > >>>>> machine - for test purpose only) * Java 7 64 bits, Tomcat > > >>>>> 7.0.52, windows 7 64 bits * using default NIO > > >>>>> configuration, but with maxThreads=400 * VM options: > > >>>>> -Xms512M - on real environment this value is 1024 > > >>>>> -Xmx512M - on real environment this value is 1024 > > >>>>> -XX:NewSize=450M -XX:MaxNewSize=450M -XX:PermSize=128M > > >>>>> -XX:MaxPermSize=245M -XX:SurvivorRatio=8 > > >>>>> -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=15 > > >>>>> -XX:+UseBiasedLocking -XX:CMSInitiatingOccupancyFraction=60 > > >>>>> -XX:+UseCMSInitiatingOccupancyOnly > > >>>>> -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC > > >>>>> -XX:+CMSIncrementalMode -XX:+UseParNewGC > > >>>>> -XX:+DisableExplicitGC -XX:+PrintGCDateStamps > > >>>>> -XX:+PrintGCDetails > > >>>>> -Xloggc:%CATALINA_BASE%/logs/tomcat-gc.log > > >>>>> > > >>>>> Moreover, I'm trying to attach the logs again. > > >>>>> > > >>>>> Thanks João > > > > > > João, > > > > > > I took a look at the log. This is the BIO attempt and you do run > > > out of threads. See the following: > > > > > > Jul 03, 2014 11:41:21 AM > > > org.apache.catalina.tribes.transport.bio.BioReceiver listen > > > WARNING: All BIO server replication threads are busy, unable to > > > handle more requests until a thread is freed up. > > > > > > What's the load like on the box when you're running the tests that > > > you get errors on? > > > > > > As Dan asks in his message: > > > > > > What is "on time"? What are the errors? Test result errors? > > > > > > How big are are the sessions that you're trying to replicate? > > > > > > My guess is that something else is going on, since the following > > > log entry doesn't show much in the way of cluster traffic. > > > > > > INFO: ThroughputInterceptor Report[ Tx Msg:1 messages Sent:0.00 MB > > > (total) Sent:0.00 MB (application) Time:0.01 seconds Tx Speed:0.04 > > > MB/sec (total) TxSpeed:0.04 MB/sec (application) Error Msg:0 Rx > > > Msg:13 messages Rx Speed:0.00 MB/sec (since 1st msg) Received:0.00 > > > MB] > > > > > > It would also be interesting to see the logs when you use the NIO > > > connector. According to the documentation: > > > > > > It is preferred to use the non blocking receiver to be able to grow > > > your cluster without running into thread starvation. > > > > > > Also from the documentation: > > > > > > Usually the rule is to use 1 thread per node in the cluster for > > > small clusters, and then depending on your message frequency and > > > your hardware, you'll find an optimal number of threads peak out > > > at a certain number. > > > > > > We might need a little more background on your application and your > > > test environment to figure out why clustering is not behaving for > > > you. > > > > > > . . . just my two cents /mde/ > > > > > > PS - you have some errors in your server.xml (see the log). While > > > they won't impact this problem, it might be a good idea to address > > > them. > > > > > > /mde/ > > >> > > >> --------------------------------------------------------------------- > > >> > > >> > > >> > > >> > > >> > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > >> For additional commands, e-mail: users-h...@tomcat.apache.org > > >> > > >> > > > > > > > > > > -----BEGIN PGP SIGNATURE----- > > Version: GnuPG v1.4.13 (MingW32) > > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > > > iQEcBAEBAgAGBQJTtZ6rAAoJEEFGbsYNeTwtAPsH/jqUHnP5Wag9fRLUYQD582/O > > 7FRoBv+Iq0/VWs2o9Wv0VrAOOazUhtAG38JgCX+v70u2MJNOIcVVpXCuOjZeSJYB > > WRkNIXqRCrVDc3/ZX3nTQoXJheZEfrdvB5cikoARPmBJeb4kOpnxKSs97OSJjHYU > > uCCoXocVfDM3JxtEHXNHyy6BuIYdizvH7DwGSts7shggT/LmKmxA16AzChppwSr4 > > 87p7jCJyxxPJ9MeRNP4uDQpV+Z/1DDhMxzUc9P8VJuSykJ1YUdQOm24AuGezsYyx > > ZQrLkioRnxDcOwpKSoI1o0r/2NgS97YR4GZbU6npzD1DjvPjZm4zimbNKM+l0iE= > > =ftU9 > > -----END PGP SIGNATURE----- > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > > -- http://joaosavio.wordpress.com
