Hey Baldur,
On Mon, Apr 28, 2014 at 8:00 AM, Baldur <baldur....@gmail.com>
wrote:
Hi all,
I'd like to get some help about my current architecture. The
current scenario uses mod_jk to connect Apache httpd and
Tomcat6. I have two Tomcat instances (using DeltaManager for
session replication and sticky session enabled) in order to
provide high availability and balance load across instances.
Currently Tomcat manages 28 webapps and 7 of them are only web
services. Generally speaking, a webapp usually involves JSF or
Struts while a web services war involves JAX-WS. Both types of
application
have a common stack implemented with Spring and Hibernate. As
a result, each application produces a war of around 40-50 MB.
Here are some questions that can make us better understand your
environment and further the discussion on your choices:
1. What kind of hardware do you run these two instances on
(single box, 2 boxes, how much RAM, etc..)? Do you have resources
to run more Tomcat instances on this(these) box(es)?
2. Do you have HA as requirement for all the apps? Do you have
any specific SLAs (service level agreements) you need to
maintain?
3. Can you live without session replication, and just live with
the sticky sessions? What kind of data do you keep in your
sessions? How big are these sessions?
4. What's the order of magnitude for your concurrent users
(100s, 1000s, 10000s) for these applications? I.e how many
concurrent sessions do you need to maintain?
5. Are your webservices stateless, most of them usually are?
6. Do these applications share any libraries (Hibernate,
Struts2, Spring, etc...)? What is the upgrade/release cycle for
these applications? How do you deal with differences in
versioning, e.g. Hibernate3 vs Hibernate4, or Spring 3.0 vs
Spring 3.2 vs Spring 4.0, etc...
In the ideal world, with infinite amount of resources (hardware,
staff, etc) - I would have one Tomcat instance (or one cluster)
per application, so I can segregate and isolate my application
environments (JVMs). However, given huge number of applications,
and that we don't have that much money to spare - that
segregation might be too extreme, too wasteful - so we typically
organize our applications to co-exist on the Tomcat instance(s),
based on their importance, SLA agreements, release lifecycle,
business operations, etc.
I'd like to ask you several questions to provide better
performance:
* Which approach would be appropriate for this
scenario? All wars in one cluster? Maybe move web services to
other cluster?
It might be useful to move webservices to a separate cluster
that might not need session replication. You might gain some
performance benefit by not having to replicate sessions across
cluster members. Though, having 28 webapps (wars) on the same
instance (clustered instance), my concern is isolation. What
happens if one application trashes one of your JVMs? Then all
other 27 apps are going to suffer and stress your other JVM. If
you truly need HA, consider moving these apps on their own
environment, independent of other apps.
* In order to improve deployments, which technique can
I use to minimize war size? Will be the cause of memory issues?
I have tried to
put
some common jars (spring, apache-commons and so on) in Tomcat
lib but I don't know if there is a better approach by other
means.
Have you observed any issues with the sizing of the apps, e.g.
OutOfMemoryError (permgen space)? Ultimately, if you deploy ton
of applications, and they all have ton of third-party libraries
(think Spring, Hibernate, etc.) - you will end up with larger
PermGen consumption, which might be exhausted after N
applications.
Placing shared libraries in the Tomcat shared folder might help
with memory sizing issues, but then you face the upgrade
lifecycle issue. You will need to coordinate the application
upgrade properly. Also, you might end up with weird errors -
because frameworks might share some objects statically, and
that's not what your intent was, etc. Thus, using shared
libraries need to be carefully planned. Usually, benefits of
shared libraries are not worth the trouble, so we end up
packaging each application separately. Shared libraries can be
very useful when admins want to enforce library versioning, and
force developers to use given environment, rather than them
including what they want/need. It's an architectural decision,
not so much performance optimization decision.
I read as much as I can but I'm stuck trying to find the best
tools to monitor the system and tackle memory issues (such as
the dreaded
PermGen).
I think it's a quite common scenario for a relatively small
production environment but I don't find the best configuration
that suits this type
of
deployment.
Well, you probably want to profile your application(s) and see
how they perform under various configuration options (memory
sizing, connector sizing, etc). That gets much easier when you
have all apps segmented to different environments. Your HTTPD
setup helps a lot, as your clients don't care where HTTPD sends
the traffic in the backend, to two instances or to twenty-eight
instances. There might be minimal or insignificant performance
overhead in maintaining 2 or 28 backend Tomcat instances
connections. However, I would probably want to measure that too
and see how it behaves under real-life like traffic.
Any help would be much appreciated.
Thanks very much in advance
Hope these questions give you something to think about and
revisit and justify your choices.
Cheers! Neven
