Hi all,
First of all, sorry for the long post, but the reasoning and
architecture requires explanation to get the full understanding of the
situation.
We are currently migrating our website from Oracle 10g to Apache
2/Tomcat 6.0.29. The site itself is a set of pages where customers
select what they want to purchase, then once they're happy, it
switches to SSL to gather their customer details and take payments.
The current setup involves two intelligent load-balancers that
terminates SSL then hands over to Oracle 10g. The 10g stack has
custom Oracle versions of Apache which have configuration options not
found in standard Apache and appear to have made configuration
simpler. For instance, there is a "simulateHttps" directive that you
can give to a virtual host in Oracle Apache which does not exist in
the standard version. Using that option, we simply have 2 virtual
hosts defined in Apache, one for http traffic and one for https
traffic (with simulateHttps set to on). They both forward to OC4J and
it works fine.
With the new architecture we have two load-balancer which round-robin
to two Apache servers. These servers connect to two Tomcat servers
which are configured with a mod_jk loadbalancer using session
affinity. Through config we have separated HTTP & HTTPS traffic. We
have done this because the application needs to identify when it is
secure, which we do by simulating HTTP using settings in the connector
in the Tomcat server.xml. This is detailed further down.
Obviously we want to keep the same functionality on the Apache/Tomcat
stack, and also retain the termination of SSL at the load-balancer,
but are having trouble when the application switches to HTTPS. It
works fine if we have a single Tomcat instance running, but once the
2nd Tomcat instance is enabled, moving to HTTPS fails because a new
session is created.
Our configuration is as follows:
Apache:
We have 2 Apache servers running - they are served in a round-robin
style by the load-balancer. They recieve 2 types of traffic,
non-secure and secure, but the secure traffic is NOT encrypted (SSL
terminated at the load-balancer).
---- httpd.conf ----
JkWorkersFile conf/workers.properties
Listen 5555
Listen 6666
NameVirtualHost *:5555
NameVirtualHost *:6666
<VirtualHost *:5555>
ServerName http://www.mysite.com:80
JkMount /* loadbalancer
</VirtualHost>
<VirtualHost *:6666>
ServerName https://www.mysite.com:443
JkMount /* loadbalancerSSL
</VirtualHost>
---- end httpd.conf ----
We are using mod_jk to connect to Tomcat
---- workers.properties ----
worker.list = loadbalancer, loadbalancerSSL, status
worker.tomcat1.type=ajp13
worker.tomcat1.host=192.168.1.36
worker.tomcat1.port=8009
worker.tomcat2.type=ajp13
worker.tomcat2.host=192.168.1.85
worker.tomcat2.port=8009
worker.tomcatSSL1.type=ajp13
worker.tomcatSSL1.host=192.168.1.36
worker.tomcatSSL1.port=8010
worker.tomcatSSL2.type=ajp13
worker.tomcatSSL2.host=192.168.1.85
worker.tomcatSSL2.port=8010
worker.loadbalancer.type = lb
worker.loadbalancer.sticky_session = 1
worker.loadbalancer.balance_workers = tomcat1,tomcat2
worker.loadbalancer.session_cookie = JSESSIONID
worker.loadbalancerSSL.type = lb
worker.loadbalancerSSL.sticky_session = 1
worker.loadbalancerSSL.balance_workers = tomcatSSL1,tomcatSSL2
worker.loadbalancerSSL.session_cookie = JSESSIONID
worker.status.type = status
---- end workers.properties ----
As you can see, Apache can receive on either 5555 or 6666 dependant on
the source of the traffic. If it's non-secure, it uses the
loadbalancer called "loadbalancer" which uses two Tomcats listening on
port 8009 - this is a connector in Tomcat that is not secure. For
traffic on 6666, "loadbalancerSSL" is used. This is the same two
Tomcats but on port 8010 - this is a connector that believes it is
secure. Tomcat config is detailed below.
Tomcat:
We have 2 Tomcat servers. They too recieve two types of traffic,
non-secure and secure, as passed on from Apache.
---- server.xml Tomcat 1 ----
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009"
protocol="AJP/1.3"
redirectPort="8010"
maxThreads="200"/>
<!-- This is the secure connetor, although its not using SSL, it's
just secure traffic -->
<Connector port="8010"
protocol="AJP/1.3"
connectionTimeout="20000"
scheme="https"
SSLEnabled="true"
secure="true"
maxThreads="200"
proxyPort="443"/>
....
<Engine name="Catalina" defaultHost="localhost" jvmRoute="tomcat1">
---- end server.xml Tomcat 1 ----
---- server.xml Tomcat 2 ----
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009"
protocol="AJP/1.3"
redirectPort="8010"
maxThreads="200"/>
<!-- This is the secure connetor, although its not using SSL, it's
just secure traffic -->
<Connector port="8010"
protocol="AJP/1.3"
connectionTimeout="20000"
scheme="https"
SSLEnabled="true"
secure="true"
maxThreads="200"
proxyPort="443"/>
....
<Engine name="Catalina" defaultHost="localhost" jvmRoute="tomcat2">
---- end server.xml Tomcat 2 ----
As can be seen, the only difference between the two Tomcat configs is
the jvmRoute parameter on the Engine config.
The connector on 8009 is a vanilla connector configuration. On 8010,
SSL related parameters are set so that it believes it is secure.
What's happening:
If I have a single Tomcat running, I can progress all the way through
the site, go secure, retain the session and get to the end of the
process. As soon as the 2nd Tomcat in introduced, when it goes
secure, the site returns to the first page. I believe this is because
sticky sessions fails and the requests ends up on a separate server,
hence creating a new session. The application detects this and sends
the user back to the first page in the process (it needs the data in
the session at this point and as there is none, redirects to the
start).
I think that sticky sessions fails because the process behind it uses
the jvmRoute value from the Tomcat server.xml to append to the end of
the JSESSIONID cookie. When the app goes secure and Apache receives
traffic on 6666, it starts to look for tomcatSSL[1/2] but none of the
tomcats use that jvmRoute and hence it will not be part of the
JSESSIONID, so it will not stick to the same server. The reason that
it works with one server is because it only has a single server to
send it to, therefore will always find the session.
If this is the case, we may have approached this incorrectly.
Separating non-SSL and SSL traffic all the way down to Tomcat may not
be the answer and we may need to use something in Apache to modify
request and response, in the same way that I assume "simulateHttps"
worked.
Has anyone else setup a site using this sort of architecture? I can't
imagine that this is uncommon in the enterprise?
Any help is greatly appreciated!
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
