Hey Baldur,
On Mon, Apr 28, 2014 at 8:00 AM, Baldur <baldur....@gmail.com> wrote:
Hi all,
I'd like to get some help about my current architecture.
The
current scenario uses mod_jk to connect Apache httpd and Tomcat6. I have
two
Tomcat instances (using DeltaManager for session replication and sticky
session enabled) in order to provide high availability and balance load
across instances. Currently Tomcat manages 28 webapps and 7 of them are
only web services. Generally speaking, a webapp usually involves JSF or
Struts while a web services war involves JAX-WS. Both types of
application
have a common stack implemented with Spring and Hibernate. As a result,
each
application produces a war of around 40-50 MB.
Here are some questions that can make us better understand your environment
and further the discussion on your choices:
1. What kind of hardware do you run these two instances on (single box, 2
boxes, how much RAM, etc..)? Do you have resources to run more Tomcat
instances on this(these) box(es)?
2. Do you have HA as requirement for all the apps? Do you have any specific
SLAs (service level agreements) you need to maintain?
3. Can you live without session replication, and just live with the sticky
sessions? What kind of data do you keep in your sessions? How big are these
sessions?
4. What's the order of magnitude for your concurrent users (100s, 1000s,
10000s) for these applications? I.e how many concurrent sessions do you
need to maintain?
5. Are your webservices stateless, most of them usually are?
6. Do these applications share any libraries (Hibernate, Struts2, Spring,
etc...)? What is the upgrade/release cycle for these applications? How do
you deal with differences in versioning, e.g. Hibernate3 vs Hibernate4, or
Spring 3.0 vs Spring 3.2 vs Spring 4.0, etc...
In the ideal world, with infinite amount of resources (hardware, staff,
etc) - I would have one Tomcat instance (or one cluster) per application,
so I can segregate and isolate my application environments (JVMs). However,
given huge number of applications, and that we don't have that much money
to spare - that segregation might be too extreme, too wasteful - so we
typically organize our applications to co-exist on the Tomcat instance(s),
based on their importance, SLA agreements, release lifecycle, business
operations, etc.
I'd like to ask you several questions to provide better
performance:
* Which approach would be appropriate for this scenario? All wars
in
one cluster? Maybe move web services to other cluster?
It might be useful to move webservices to a separate cluster that might not
need session replication. You might gain some performance benefit by not
having to replicate sessions across cluster members. Though, having 28
webapps (wars) on the same instance (clustered instance), my concern is
isolation. What happens if one application trashes one of your JVMs? Then
all other 27 apps are going to suffer and stress your other JVM. If you
truly need HA, consider moving these apps on their own environment,
independent of other apps.
* In order to improve deployments, which technique can I use to
minimize war size? Will be the cause of memory issues? I have tried to
put
some common jars (spring, apache-commons and so on) in Tomcat lib but I
don't know if there is a better approach by other means.
Have you observed any issues with the sizing of the apps, e.g.
OutOfMemoryError (permgen space)? Ultimately, if you deploy ton of
applications, and they all have ton of third-party libraries (think Spring,
Hibernate, etc.) - you will end up with larger PermGen consumption, which
might be exhausted after N applications.
Placing shared libraries in the Tomcat shared folder might help with memory
sizing issues, but then you face the upgrade lifecycle issue. You will need
to coordinate the application upgrade properly. Also, you might end up with
weird errors - because frameworks might share some objects statically, and
that's not what your intent was, etc. Thus, using shared libraries need to
be carefully planned. Usually, benefits of shared libraries are not worth
the trouble, so we end up packaging each application separately. Shared
libraries can be very useful when admins want to enforce library
versioning, and force developers to use given environment, rather than them
including what they want/need. It's an architectural decision, not so much
performance optimization decision.
I read as much as I can but I'm stuck trying to find the best tools to
monitor the system and tackle memory issues (such as the dreaded
PermGen).
I think it's a quite common scenario for a relatively small production
environment but I don't find the best configuration that suits this type
of
deployment.
Well, you probably want to profile your application(s) and see how they
perform under various configuration options (memory sizing, connector
sizing, etc). That gets much easier when you have all apps segmented to
different environments. Your HTTPD setup helps a lot, as your clients don't
care where HTTPD sends the traffic in the backend, to two instances or to
twenty-eight instances. There might be minimal or insignificant performance
overhead in maintaining 2 or 28 backend Tomcat instances connections.
However, I would probably want to measure that too and see how it behaves
under real-life like traffic.
Any help would be much appreciated.
Thanks very much in advance
Hope these questions give you something to think about and revisit and
justify your choices.
Cheers!
Neven
