Perhaps I am not wording my question correctly.
Today we have...
[Prxoy 1] | [Proxy 2] ---> [Apache ---> tomcat1]
(HTTPS) (HTTPS) (HTTPS) --> (AJP) -->
So we send the information from the proxies over https to the instance
running the tomcat server.
The SSL is terminated by Apache/HTTPD and handed back to tomcat over AJP.
Tomcat doesn't handle SSL in anyway. It can't, it's not configured to do
so. Is that how you understand the question I asked?
As a test I changed the following line in one of the many tomcat instances
running on the server and bounced it.
Old
<!-- <Connector port="9335" protocol="AJP/1.3" redirectPort="8443"
maxThreads="300" /> -->
New
<Connector port="9335" protocol="org.apache.coyote.ajp.AjpNioProtocol"
redirectPort="8443" maxThreads="300" />
As the docs state I am able to verify that it did in fact switch over to
NIO.
INFO: Starting ProtocolHandler ["ajp-nio-9335"]
Will running NIO and BIO on the same box have a negative impact? I am
thinking they should all be switched to NIO, this was just a test to see if
I was understanding what I was reading. That being said I suspect there
are going to be far more tweaks that needs to be applied as there are none
to date. I also know that the HTTPD server is running in prefork mode.
Which I think leaves me with no control over how many connections can be
handed back from apache on a site by site basis.
Really having hard time explaining to others how BIO could have caused the
connection pool for another use to become exhausted.
Thanks,
TCD
On Wed, Oct 4, 2017 at 1:31 PM, Mark Thomas <ma...@apache.org> wrote:
> On 04/10/17 19:26, TurboChargedDad . wrote:
> > My initial reads about BIO vs NIO seems to involve terminating SSL at
> the
> > tomcat instance. Which we do not do. Am I running off into the weeds
> with
> > that?
>
> Yes. The NIO AJP connector is a drop in replacement for the BIO AJP
> connector.
>
> https://tomcat.apache.org/tomcat-7.0-doc/config/ajp.
> html#Standard_Implementations
>
> Look for the protocol attribute.
>
> Mark
>
>
> >
> > Thanks,
> > TCD
> >
> > On Wed, Oct 4, 2017 at 9:17 AM, Mark Thomas <ma...@apache.org> wrote:
> >
> >> On 04/10/17 13:51, TurboChargedDad . wrote:
> >>> Hello all..
> >>> I am going to do my best to describe my problem. Hopefully someone
> will
> >>> have some sort of insight.
> >>>
> >>> Tomcat 7.0.41 (working on updating that)
> >>> Java 1.6 (Working on getting this updated to the latest minor release)
> >>> RHEL Linux
> >>>
> >>> I inherited an opti-tenant setup. Individual user accounts on the
> system
> >>> each have their own Tomcat instance, each is started using sysinit.
> This
> >>> is done to keep each website in its own permissible world so one
> website
> >>> can't interfere with a others data.
> >>>
> >>> There are two load balanced apache proxies at the edge that point to
> one
> >>> Tomcat server (I know I know but again I inherited this)
> >>>
> >>> Apache lays over the top of tomcat to terminate SSL and uses AJP to
> >>> proxypass to each tomcat instance based on the users assigned port.
> >>>
> >>> Things have run fine for years (so I am being told anyway) until
> >> recently.
> >>> Let me give an example of an outage.
> >>>
> >>> User1, user2 and user3 all use unique databases on a shared database
> >>> server, SQL server 10.
> >>>
> >>> User 4 runs on a windows jboss server and also has a database on shared
> >>> database server 10.
> >>>
> >>> Users 5-50 all run in the mentioned Linux server using tomcat and have
> >>> databases on *other* various shared databases servers but have nothing
> to
> >>> do with database server 10.
> >>>
> >>> User 4 had a stored proc go wild on database server 10 basically
> knocking
> >>> it offline.
> >>>
> >>> Now one would expect sites 1-4 to experience interruption of service
> >>> because they use a shared DBMS platform. However.
> >>>
> >>> Every single site goes down. I monitor the connections for each site
> >> with a
> >>> custom tool. When this happens, the connections start stacking up
> across
> >>> all the components. (Proxies all the way through the stack)
> >>> Looking at the AJP connection pool threads for user 9 shows that user
> has
> >>> exhausted their AJP connection pool threads. They are maxed out at 300
> >> yet
> >>> that user doesn't have high activity at all. The CPU load, memory usage
> >> and
> >>> traffic for everything except SQL server 10 is stable during this
> >> outrage.
> >>> The proxies start consuming more and more memory the longer the outrage
> >>> occurs but that's expected as the connection counts stack up into the
> >>> thousands. After a short time all the sites apache / ssl termination
> >> later
> >>> start throwing AJP timeout errors. Shortly after that the edge proxies
> >>> will naturally also starting throwing timeout errors of their own.
> >>>
> >>> I am only watching user 9 using a tool that allows me to have insight
> >> into
> >>> what's going on using JMX metrics but I suspect that once I get all the
> >>> others instrumented that I will see the same thing. Maxed out AJP
> >>> connection pools.
> >>>
> >>> Aren't those supposed to be unique per user/ JVM? Am I missing
> something
> >> in
> >>> the docs?
> >>>
> >>> Any assistance from the tomcat gods is much appreciated.
> >>
> >> TL;DR - Try switching to the NIO AJP connector on Tomcat.
> >>
> >> Take a look at this session I just uploaded from TomcatCon London last
> >> week. You probably want to start around 35:00 and the topic of thread
> >> exhaustion.
> >>
> >> HTH,
> >>
> >> Mark
> >>
> >> P.S. The other sessions we have are on the way. I plan to update the
> >> site and post links once I have them all uploaded.
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>
> >>
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
