On Thu, Mar 7, 2019 at 2:54 PM Rémy Maucherat <r...@apache.org> wrote:
> On Thu, Mar 7, 2019 at 1:47 PM Mark Thomas <ma...@apache.org> wrote: > >> On 07/03/2019 07:40, Santhosh Kumar wrote: >> > From some of the test cases I can safely say that tomcat is hitting some >> > limits, I have two test cases ran with two diff size of payload and >> without >> > any queryParams. The servlet is a empty servlet just returns after >> > receiving without doing any business side logic >> >> Can you repeat those tests with the NIO connector? It would be helpful >> to know if we should be looking at the HTTP/2 code or the low-level >> connector I/O code. >> > > I was planning to investigate since I'm hunting NIO2 additional issues > after the fix for BZ63182. This one looks simpler to reproduce at least > [assuming there's an issue]. > Ok, so it's a buffer size issue with vectored IO and SSL, the sizes used are too optimized. Rémy > > Rémy > > >> >> Thanks, >> >> Mark >> >> >> > >> > h2load -n100 -c1 -m1 --header="Content-Type:application/json" -d >> > /home/local/santhosh/A-Test/nghttp2/agentRequest.txt >> > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet >> > starting benchmark... >> > spawning thread #0: 1 total client(s). 100 total requests >> > TLS Protocol: TLSv1.3 >> > Cipher: TLS_AES_256_GCM_SHA384 >> > Server Temp Key: X25519 253 bits >> > Application protocol: h2 >> > progress: 10% done >> > progress: 20% done >> > progress: 30% done >> > progress: 40% done >> > progress: 50% done >> > >> > finished in 5.16s, 10.48 req/s, 552B/s >> > requests: 100 total, 55 started, 54 done, 54 succeeded, 46 failed, 46 >> > errored, 0 timeout >> > status codes: 55 2xx, 0 3xx, 0 4xx, 0 5xx >> > traffic: 2.78KB (2846) total, 1.77KB (1815) headers (space savings >> 43.10%), >> > 0B (0) data >> > min max mean sd +/- >> sd >> > time for request: 1.57ms 9.43ms 2.24ms 1.17ms >> 94.44% >> > time for connect: 4.69ms 4.69ms 4.69ms 0us >> 100.00% >> > time to 1st byte: 0us 0us 0us 0us >> 0.00% >> > req/s : 10.48 10.48 10.48 0.00 >> 100.00% >> > >> > This above configuration always returns 54 succeeded, payload size is >> 1200B >> > (1200x54=64800) >> > >> ------------------------------------------------------------------------------------------------------------------------------ >> > Now reduce the payload and trying the same test, >> > >> > h2load -n100 -c1 -m1 --header="Content-Type:application/json" -d >> > /home/local/santhosh/A-Test/nghttp2/agentRequest2.txt >> > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet >> > starting benchmark... >> > spawning thread #0: 1 total client(s). 100 total requests >> > TLS Protocol: TLSv1.3 >> > Cipher: TLS_AES_256_GCM_SHA384 >> > Server Temp Key: X25519 253 bits >> > Application protocol: h2 >> > progress: 10% done >> > progress: 20% done >> > progress: 30% done >> > progress: 40% done >> > progress: 50% done >> > progress: 60% done >> > progress: 70% done >> > progress: 80% done >> > >> > finished in 5.21s, 16.11 req/s, 839B/s >> > requests: 100 total, 85 started, 84 done, 84 succeeded, 16 failed, 16 >> > errored, 0 timeout >> > status codes: 85 2xx, 0 3xx, 0 4xx, 0 5xx >> > traffic: 4.27KB (4376) total, 2.74KB (2805) headers (space savings >> 43.10%), >> > 0B (0) data >> > min max mean sd +/- >> sd >> > time for request: 1.43ms 5.80ms 2.04ms 760us >> 89.29% >> > time for connect: 5.02ms 5.02ms 5.02ms 0us >> 100.00% >> > time to 1st byte: 0us 0us 0us 0us >> 0.00% >> > req/s : 16.11 16.11 16.11 0.00 >> 100.00% >> > >> > This above configuration always returns 84 succeeded, payload size is >> 775B >> > (775x84=65100) >> > >> ------------------------------------------------------------------------------------------------------------------------------ >> > Reducing the payload much smaller, >> > >> > h2load -n200 -c1 -m1 --header="Content-Type:application/json" -d >> > /home/local/santhosh/A-Test/nghttp2/agentRequest3.txt >> > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet >> > starting benchmark... >> > spawning thread #0: 1 total client(s). 200 total requests >> > TLS Protocol: TLSv1.3 >> > Cipher: TLS_AES_256_GCM_SHA384 >> > Server Temp Key: X25519 253 bits >> > Application protocol: h2 >> > progress: 10% done >> > progress: 20% done >> > progress: 30% done >> > progress: 40% done >> > progress: 50% done >> > progress: 60% done >> > progress: 70% done >> > progress: 80% done >> > progress: 90% done >> > >> > finished in 5.41s, 34.40 req/s, 1.73KB/s >> > requests: 200 total, 187 started, 186 done, 186 succeeded, 14 failed, 14 >> > errored, 0 timeout >> > status codes: 187 2xx, 0 3xx, 0 4xx, 0 5xx >> > traffic: 9.35KB (9578) total, 6.03KB (6171) headers (space savings >> 43.10%), >> > 0B (0) data >> > min max mean sd +/- >> sd >> > time for request: 1.18ms 13.49ms 1.91ms 1.13ms >> 95.16% >> > time for connect: 5.93ms 5.93ms 5.93ms 0us >> 100.00% >> > time to 1st byte: 0us 0us 0us 0us >> 0.00% >> > req/s : 34.41 34.41 34.41 0.00 >> 100.00% >> > >> > This above configuration always returns 186 succeeded, payload size is >> > 356 (356x186=66216) >> > >> > On Wed, Mar 6, 2019 at 9:15 PM John Dale <jcdw...@gmail.com> wrote: >> > >> >> When you run your test(s), does it fail after a certain period of >> >> time, or just keep on going under a certain number of requests? >> >> >> >> Also, to confirm: you're sending 1000 Byte + query strings? >> >> >> >> Are you doing anything in the server side component to verify that >> >> your parameters have been received successfully? >> >> >> >> I seems very possible that there is increased overhead parsing the >> >> request (POST) body. That's why I was wondering about the dynamics of >> >> your test case. If you can achieve a steady load state, either some >> >> optimization of the POST request parser could be done, or you could >> >> accept that overhead if it is comparable to other solutions. >> >> >> >> On 3/6/19, Santhosh Kumar <santhosh...@gmail.com> wrote: >> >>> I hope so, I used updated packages/components at the time of >> development. >> >>> few may be outdated like tomcat native as I was using 1.2.18 while >> >>> developing but 1.2.21 got released recently. >> >>> >> >>> On Wed, Mar 6, 2019 at 6:18 PM John Dale <jcdw...@gmail.com> wrote: >> >>> >> >>>> Have you upgraded to the most recent release of your major version? >> >>>> >> >>>> If so, and if this issue still persists, it is something that the >> core >> >>>> development team might want to look at assuming they can replicate >> the >> >>>> issue. >> >>>> >> >>>> On 3/5/19, Santhosh Kumar <santhosh...@gmail.com> wrote: >> >>>>> Sometimes more than 10x >> >>>>> >> >>>>> On Tue, Mar 5, 2019 at 10:00 PM John Dale <jcdw...@gmail.com> >> wrote: >> >>>>> >> >>>>>> How many orders of magnitude slower are the post requests? >> >>>>>> >> >>>>>> On 3/5/19, Santhosh Kumar <santhosh...@gmail.com> wrote: >> >>>>>>> I was testing in the localhost >> >>>>>>> >> >>>>>>> On Tue, Mar 5, 2019 at 9:32 PM John Dale <jcdw...@gmail.com> >> >> wrote: >> >>>>>>> >> >>>>>>>> Are you running your test client (h2load) on the same machine, >> >> same >> >>>>>>>> network, or is it over the net (so, like 20ms latency on each >> >>>>>>>> request)? The reason I ask is that if you are local >> (especially), >> >>>>>>>> it >> >>>>>>>> may queue up too many requests for tomcat to handle in the >> testing >> >>>>>>>> period with its thread pool. Will let you know if I have any >> >> other >> >>>>>>>> ideas. >> >>>>>>>> >> >>>>>>>> On 3/5/19, Santhosh Kumar <santhosh...@gmail.com> wrote: >> >>>>>>>>> Bytes >> >>>>>>>>> >> >>>>>>>>> On Tue, Mar 5, 2019 at 9:28 PM John Dale <jcdw...@gmail.com> >> >>>> wrote: >> >>>>>>>>> >> >>>>>>>>>> 1000-1500 MB or KB? >> >>>>>>>>>> >> >>>>>>>>>> On 3/4/19, Santhosh Kumar <santhosh...@gmail.com> wrote: >> >>>>>>>>>>> As per the documentation, >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>> >> >>>>>> >> >>>> >> >> >> https://tomcat.apache.org/tomcat-9.0-doc/config/http.html#SSL_Support_-_SSLHostConfig >> >>>>>>>>>>> >> >>>>>>>>>>> this connector supports maxPostSize, by default the limit is >> >>>>>>>>>>> set >> >>>>>>>>>>> to >> >>>>>>>> 2MB >> >>>>>>>>>>> >> >>>>>>>>>>> On Tue, Mar 5, 2019 at 5:09 AM John Dale <jcdw...@gmail.com> >> >>>>>> wrote: >> >>>>>>>>>>> >> >>>>>>>>>>>> Does anyone know if this connector supports maxPostSize >> >>>>>>>>>>>> parameter? >> >>>>>>>>>>>> >> >>>>>>>>>>>> On 3/4/19, Santhosh Kumar <santhosh...@gmail.com> wrote: >> >>>>>>>>>>>>> Hi, >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> We have a tomcat instance which is http2 enabled and it >> >>>>>>>>>>>>> needs >> >>>>>>>>>>>>> to >> >>>>>>>>>>>>> serve >> >>>>>>>>>>>>> large number of requests using multiplexing, so we have >> >>>>>>>>>>>>> configured >> >>>>>>>>>>>>> our >> >>>>>>>>>>>>> instance as follows, >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> <Connector port="9191" URIEncoding="UTF-8" >> >>>>>>>>>>>>> sslImplementationName="org.apache.tomcat.util.net >> >>>>>>>>>>>> .openssl.OpenSSLImplementation" >> >>>>>>>>>>>>> protocol="org.apache.coyote.http11.Http11Nio2Protocol" >> >>>>>>>>>>>>> maxThreads="50000" >> >>>>>>>>>>>>> SSLEnabled="true" >> >>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>> >> >>>>>> >> >>>> >> >> >> compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml" >> >>>>>>>>>>>>> compression="on" >> >>>> minSpareThreads="25" >> >>>>>>>>>>>>> noCompressionUserAgents="gozilla, traviata" scheme="https" >> >>>>>>>>>>>>> secure="true" >> >>>>>>>>>>>>> keystoreFile="conf/myfile.keystore" >> >> keystorePass="password" >> >>>>>>>>>>>>> socket.appReadBufSize="81920" >> >>>>>>>>>>>>> socket.appWriteBufSize="81920" socket.rxBufSize="251880" >> >>>>>>>>>>>>> socket.txBufSize="438000"> >> >>>>>>>>>>>>> <UpgradeProtocol compression="on" >> >>>>>>>>>>>>> >> >>>>>> maxConcurrentStreamExecution="200" >> >>>>>>>>>>>>> maxConcurrentStreams="200" >> >>>>>>>>>>>>> className="org.apache.coyote.http2.Http2Protocol"/> >> >>>>>>>>>>>>> </Connector> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> This instance mainly serves concurrent POST request which >> >>>> will >> >>>>>>>>>>>>> have >> >>>>>>>>>>>> payload >> >>>>>>>>>>>>> of size, approx 1000-1500, which can be verified by tomcat >> >>>>>>>>>>>>> logs >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> org.apache.coyote.http2.Http2Parser.validateFrame >> >>>>>>>>>>>>> Connection >> >>>>>> [0], >> >>>>>>>>>>>>> Stream >> >>>>>>>>>>>>> [19], Frame type [DATA], Flags [1], Payload size [*1195*] >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> We tested our server with the help of h2load as follows, >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> h2load -n100 -c1 -m100 https://localhost:9191/ -d >> >>>>>>>>>>>>> '/agentRequest.txt' >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> We are getting this error as follows, >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >> org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch >> >>>>>>>>>>>>> Connection >> >>>>>>>>>>>> [0] >> >>>>>>>>>>>>> java.io.IOException: Unable to unwrap data, invalid >> >> status >> >>>>>>>>>>>>> [BUFFER_OVERFLOW] >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> org.apache.tomcat.util.net >> >>>>>>>>>>>> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1041) >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> org.apache.tomcat.util.net >> >>>>>>>>>>>> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> >> >>>> java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> >> >> java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158) >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> java.base/sun.nio.ch >> >>>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>> >> >>>>>> >> >>>> >> >> >> .UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:552) >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> java.base/sun.nio.ch >> >>>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>> >> >>>>>> >> >>>> >> >> >> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276) >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> java.base/sun.nio.ch >> >>>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>> >> >>>>>> >> >>>> >> >> >> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297) >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> org.apache.tomcat.util.net >> >>>>>>>>>>>> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1027) >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> org.apache.tomcat.util.net >> >>>>>>>>>>>> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> org.apache.tomcat.util.net >> >>>>>>>>>>>> .SecureNio2Channel.read(SecureNio2Channel.java:1067) >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> org.apache.tomcat.util.net >> >>>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>> >> >>>>>> >> >>>> >> >> >> .Nio2Endpoint$Nio2SocketWrapper$VectoredIOCompletionHandler.completed(Nio2Endpoint.java:1153) >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> org.apache.tomcat.util.net >> >>>>>>>>>>>> .Nio2Endpoint$Nio2SocketWrapper.read(Nio2Endpoint.java:1026) >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> org.apache.tomcat.util.net >> >>>>>>>>>>>> .SocketWrapperBase.read(SocketWrapperBase.java:1012) >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>> >> >>>>>> >> >>>> >> >> >> org.apache.coyote.http2.Http2AsyncParser.readFrame(Http2AsyncParser.java:61) >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> >> >>>>>> org.apache.coyote.http2.Http2Parser.readFrame(Http2Parser.java:69) >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>> >> >>>>>> >> >>>> >> >> >> org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2UpgradeHandler.java:322) >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>> >> >>>>>> >> >>>> >> >> >> org.apache.coyote.http2.Http2AsyncUpgradeHandler.upgradeDispatch(Http2AsyncUpgradeHandler.java:37) >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>> >> >>>>>> >> >>>> >> >> >> org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54) >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>> >> >>>>>> >> >>>> >> >> >> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:53) >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>> >> >>>>>> >> >>>> >> >> >> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834) >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> org.apache.tomcat.util.net >> >>>>>>>>>>>> .Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1769) >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> org.apache.tomcat.util.net >> >>>>>>>>>>>> .SocketProcessorBase.run(SocketProcessorBase.java:49) >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> org.apache.tomcat.util.net >> >>>>>>>>>>>> .AbstractEndpoint.processSocket(AbstractEndpoint.java:1048) >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> org.apache.tomcat.util.net >> >>>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>> >> >>>>>> >> >>>> >> >> >> .SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:116) >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> org.apache.tomcat.util.net >> >>>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>> >> >>>>>> >> >>>> >> >> >> .SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:109) >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> >> >>>> java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) >> >>>>>>>>>>>>> at >> >>>>>>>>>>>>> >> >> java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158) >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Why is this error is thrown? How can I configure tomcat to >> >>>>>> handle >> >>>>>>>>>>>>> concurrent POST requests which have a decent payload? >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> We have tried with various java clients like >> >>>>>>>>>>>>> http-client-5-beta, >> >>>>>>>>>>>>> jetty >> >>>>>>>>>>>>> or >> >>>>>>>>>>>>> okhttp3 and spam requests to our tomcat using http2 >> >>>>>>>>>>>>> multiplexing >> >>>>>>>> and >> >>>>>>>>>> we >> >>>>>>>>>>>>> found the time taken to process a requests >> >>>> increases(sometimes >> >>>>>>>>>>>>> even >> >>>>>>>>>>>>> 10x) >> >>>>>>>>>>>>> when load is increased. >> >>>>>>>>>>>>> We have tweaked all common configuration related to http2 >> >>>>>>>>>>>>> on >> >>>>>> both >> >>>>>>>>>>>>> client >> >>>>>>>>>>>>> and server side with no luck. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> But same tomcat configuration can handle 10s of 1000s of >> >>>>>>>>>>>>> get >> >>>>>>>> request >> >>>>>>>>>>>>> concurrently without a problem, its only creating problem >> >>>> with >> >>>>>>>>>>>>> POST >> >>>>>>>>>>>>> requests. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> What is wrong in our configuration? >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Kindly someone shed some light. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Tomcat - 9.0.16 >> >>>>>>>>>>>>> APR-1.2.18 >> >>>>>>>>>>>>> OpenSSL-1.1.1a >> >>>>>>>>>>>>> JDK-10.0.2 >> >>>>>>>>>>>>> OS - Ubuntu/Centos >> >>>>>>>>>>>>> HeapSize - 4GB >> >>>>>>>>>>>>> RAM -16GB >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Kindly help >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> -- >> >>>>>>>>>>>>> *With Regards,* >> >>>>>>>>>>>>> *Santhosh Kumar J* >> >>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>> >> --------------------------------------------------------------------- >> >>>>>>>>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> >>>>>>>>>>>> For additional commands, e-mail: >> >> users-h...@tomcat.apache.org >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> -- >> >>>>>>>>>>> *With Regards,* >> >>>>>>>>>>> *Santhosh Kumar J* >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>> --------------------------------------------------------------------- >> >>>>>>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> >>>>>>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >> >>>>>>>>>> >> >>>>>>>>>> -- >> >>>>>>>>> *With Regards,* >> >>>>>>>>> *Santhosh Kumar J* >> >>>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >> --------------------------------------------------------------------- >> >>>>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> >>>>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >> >>>>>>>> >> >>>>>>>> -- >> >>>>>>> *With Regards,* >> >>>>>>> *Santhosh Kumar J* >> >>>>>>> >> >>>>>> >> >>>>>> >> --------------------------------------------------------------------- >> >>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> >>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >> >>>>>> >> >>>>>> >> >>>>> >> >>>>> -- >> >>>>> *With Regards,* >> >>>>> *Santhosh Kumar J* >> >>>>> >> >>>> >> >>>> --------------------------------------------------------------------- >> >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> >>>> For additional commands, e-mail: users-h...@tomcat.apache.org >> >>>> >> >>>> >> >>> >> >>> -- >> >>> *With Regards,* >> >>> *Santhosh Kumar J* >> >>> >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> >> >> >> > >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >>
