[ANN] Apache Tomcat 7.0.73 released

2016-11-15 Thread Violeta Georgieva 
The Apache Tomcat team announces the immediate availability of Apache
Tomcat 7.0.73.

Apache Tomcat is an open source software implementation of the Java
Servlet, JavaServer Pages, Java Expression Language and Java
WebSocket technologies.

This release contains a number of bug fixes and improvements compared to
version 7.0.72. The notable changes since 7.0.72 include:


- Improve handling of I/O errors with async processing
- Fail earlier on invalid HTTP requests


Please refer to the change log for the complete list of changes:
http://tomcat.apache.org/tomcat-7.0-doc/changelog.html

Downloads:
http://tomcat.apache.org/download-70.cgi

Migration guides from Apache Tomcat 5.5.x and 6.0.x:
http://tomcat.apache.org/migration.html

Enjoy

The Apache Tomcat team

[ANN] Apache Tomcat 8.0.39 available

2016-11-15 Thread Violeta Georgieva 
The Apache Tomcat team announces the immediate availability of Apache
Tomcat 8.0.39.

Please note that Tomcat 8.x users should normally be using 8.5.x
releases in preference to 8.0.x releases.

Apache Tomcat 8.0 is an open source software implementation of the Java
Servlet, JavaServer Pages, Java Unified Expression Language and Java
WebSocket technologies.

Apache Tomcat 8.0.39 includes fixes for issues identified in 8.0.38 as
well as other enhancements and changes. The notable changes since 8.0.38
include:

- Improve handling of I/O errors with async processing

- Fail earlier on invalid HTTP requests


Please refer to the change log for the complete list of changes:
http://tomcat.apache.org/tomcat-8.0-doc/changelog.html

Downloads:
http://tomcat.apache.org/download-80.cgi

Migration guides from Apache Tomcat 5.5.x, 6.0.x and 7.0.x:
http://tomcat.apache.org/migration.html

Enjoy!

- The Apache Tomcat team

Please help with Tomcat Garbage Collection

2016-11-15 Thread George I. Develekos 

Hello guys,

We are having problems on a production system with very long "full GC" 
times, as long as1200sec real time (!!!).


We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat 7.0.64.

Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC   
-XX:+CMSIncrementalMode


No other custom memory-related settings are in place.

Looking at the GC log, the last few Full-GC entries are:

1367.020: [Full GC 1367.020: [CMS: 1178831K->527456K(1926784K), 
2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm : 
169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs] [Times: user=1.96 
sys=0.13, real=2.11 secs]


2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-mark: 
2.558/1212.733 secs] [*Times: user=113.05 sys=28.01, real=**1212.49 
**secs] **

*
3539.969: [Full GC 3539.969: [CMS3540.056: [CMS-concurrent-sweep: 
1.571/23.223 secs] [Times: user=6.12 sys=1.36, real=*23.21 secs*]


4070.456: [Full GC 4070.457: [CMS: 1252569K->591200K(1926784K), 
2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm : 
169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs] [Times: user=2.18 
sys=0.14, real=2.34 secs]



What can we do?

Thanks,

George





---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

RE: Apache Tomcat Version 7.0.59 on CentOS 6.8 to handle 2k requests/second

2016-11-15 Thread Hedrick, Brooke - 43 
-Original Message-
From: Kaushal Shriyan [mailto:kaushalshri...@gmail.com] 
Sent: Monday, November 14, 2016 8:46 AM
To: Tomcat Users List 
Subject: Re: Apache Tomcat Version 7.0.59 on CentOS 6.8 to handle 2k 
requests/second

...

>Please suggest me any open source profiling application which i can start 
>with. I will appreciate if you can help me understand the relation between 2k 
>requests/second vs cpu >cores of the server.
>
>Thanks again in advance.
>
>Regards,
>
>Kaushal

A couple I have used: 
1. Apache Bench (ab/abs very basic ) ships with Apache.
2. Apache JMeter.  Much more in-depth and flexible in how you load test and the 
results collected.



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: 8.5.4 to 8.5.5 SSL Issue

2016-11-15 Thread Christopher Schultz 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

William,

On 11/14/16 7:18 PM, William Boyd wrote:
> First, I'd like to thank everyone for the help.
> 
> Is it now safe to say that the behaviour we've been taking
> advantage of is undocumented and will no long be supported?
> 
> Also, for those that hit this thread and need to get HTTPS working
> with a *self-signed* certificate in a dev environment...
> 
> Here's what worked for me: 1. Copy
> %JAVA_HOME%\jre\lib\security\cacerts some place (say C:\keystore) 
> 2. Create a *self-signed* certificate with %JAVA_HOME%\bin\keytool
> -genkeypair -keyalg RSA -alias myAlias -keystore
> "C:\keystore\keystore.jsk" -storepass changeit -validity 360 
> -keysize 2048 -dname
> CN=localhost,OU=OrgUnit,O=Org,L=City,ST=State,C=Country 3. Export
> the myAlias certificate with %JAVA_HOME%\bin\keytool -export -alias
> myAlias -keystore C:\keystore\keystore.jsk -rfc -file
> C:\keystore\myAlias.cer 4. Import the myAlias certificate into your
> copy of cacerts with: %JAVA_HOME%\bin\keytool -import -alias
> myAlias -keystore C:\keystore\cacerts -file
> C:\keystore\myAlias.cer 5. Add this to setclasspath.bat in tomcat: 
> set JAVA_OPTS=%JAVA_OPTS% 
> -Djavax.net.ssl.trustStore="C:\keystore\cacert" 6. In server.xml,
> add these attributes to the Connector element 
> keystoreFile="C:\keystore\keystore.jsk" keystorePass="changeit" 
> keyAlias="myAlias" scheme="https" secure="true" SSLEnabled="true" 
> clientAuth="false" sslProtocol="TLS"
> 
> Now you should be good to go in with HTTPS in Tomcat 8.5.5 +

This is exactly what I suggested, except that you set the trust store
using a system property instead of using truststoreFile in the
. You do not need to duplicate the JRE's trust store. You
only need your own single certificate in your local truststore.

Can you use keystoreFile and truststoreFile separately pointing to
those files and re-check that it works? Perhaps Tomcat chokes when
using the same file for both. That's what I'd like to verify.

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJYK1kHAAoJEBzwKT+lPKRYa48P/jl6hWa1mj5HCnawJZ3mHbjX
ADdXYl45aym/E6QV/n99XYVYG2q+ZN9w0XDVP54lQhQhcgOUtCiPbTHEcYSYdwrU
kLYMc3Ge8Jt7/zDMvem+pKYkHMvyHbspVqSujZ4uJ3Ozr9mYD89hSFgxqG0iYSE+
5c0pvz1nW4Pt1F4A/+WETkL4Y5Xrq1Vn1LSAxAZoYiU/o93nVos7etIBUO9E430+
GihbhvkpS/yBitvrir/YacvWauBxpi30wR++6ZNAhpzlb+j90dk3i6iPcDO6K1f2
SNeqZATJDlXyU1hEksW4UxWLhtUeekqmJEiEqqWCYxNz9lwJG9f4kILUrzsZexlu
FmP2o4IxWTBcgOUs5Km5DlfYwogJmlRhqQoOlg2JOpv+KIb67DX+PuY6bhGomDff
YQ03Y7WQcjNZ/uOIoadAkXxKRaRHmuz2KkPYwgDutOgxtJV1jNxTT3A3znGT1cWN
yekjXHOpe2FdXnaoG0X7mTpvx5AhkHN9mRdW+5/ZBpPzUN0M7zy8oBEpLtZKfrTJ
k40Xz70DnNxBP3XS/1w7DJ1H3/FBxNdatVVbbcJ/+lS/NiS4Gn2kMAZgrCuZrUsn
FdpdyCwq3VLJ2X9LVBR03rJOyPIiybANNjfhPpiEMC9uQu2ENm4A4Hm1p/cXdpo3
2J2O1AQA7tfew10t3F4K
=a+Um
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Apache Tomcat Version 7.0.59 on CentOS 6.8 to handle 2k requests/second

2016-11-15 Thread Christopher Schultz 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Brooke,

On 11/15/16 1:17 PM, Hedrick, Brooke - 43 wrote:
> -Original Message- From: Kaushal Shriyan
> [mailto:kaushalshri...@gmail.com] Sent: Monday, November 14, 2016
> 8:46 AM To: Tomcat Users List  Subject:
> Re: Apache Tomcat Version 7.0.59 on CentOS 6.8 to handle 2k
> requests/second
> 
> ...
> 
>> Please suggest me any open source profiling application which i
>> can start with. I will appreciate if you can help me understand
>> the relation between 2k requests/second vs cpu >cores of the
>> server.
>> 
>> Thanks again in advance.
>> 
>> Regards,
>> 
>> Kaushal
> 
> A couple I have used: 1. Apache Bench (ab/abs very basic ) ships
> with Apache. 2. Apache JMeter.  Much more in-depth and flexible in
> how you load test and the results collected.

Those are load-generators, not profilers.

I have never been happy with any open-source profilers. There are
several good non-free options, though.

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJYK1mSAAoJEBzwKT+lPKRY2PUP/1Uw7C2TiN6x37SejT+pyc1x
7N6n5FoUyB8RdrVIZlkEsbTzaWKHzq5EisxczzblleTip8jNBwSXaTjRQ+CgQWjU
//UinQwf7bhCZCzKuSlzq51HOYSkoIPLQtg2NBNUGtqXJ7LtbwdXkBRbMQxPeiMV
+RSNmY1CoTElvxJeYfqcr4A1R0PRTgoOMkz7CSj2Hf2H7XsccY1bFH/ofiFGbpKh
v0GTRiq9TCNGVBtVhAaU5339KRBuxB5QkTgZF/7KQU719d6jsmsB+u7W1St2gXWc
zaIUg75zC6KP+IxlXkkPldh6dDruw0drG8q1RUEai972JZS9e1/kLnFaGne02pUW
UPSIgCeSXPRiSPGfLE5e1FkQxmrQmuMlWo6imwvbSQ0Rzgy8gKVv2seYsM9jqsko
BzWn2iri4YPVzAunKfZIekOV4xL5Bev08PND4jdBkPIWwTF9Tz1twnBxK33q+Mi1
qEYSezzS4PmFQAtNnwbRBGolqCiJMITtfP4nkO3QsWfzM0/CZo4M587onXnhawbA
yzxlG5tzCHYQhr6PQD4POA+bH0RbWI4rDm4DqlD5RRDjVC0BMkxiejwXH1SIRApP
4vcNVipxr7MzngDSsAwbG+Jz4dUGlNas9o5WjoAu8Jya/PYkaZ4LEv/Dn0DWgTnb
nvYTZz8td79PoGkK2a45
=4KGB
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Please help with Tomcat Garbage Collection

2016-11-15 Thread pina.freder...@gmail.com 
y

Sent from my HTC on T-  .  4G LTE

- Reply message -
From: "George I. Develekos" 
To: 
Subject: Please help with Tomcat Garbage Collection
Date: Tue, Nov 15, 2016 10:46 AM

Hello guys,

We are having problems on a production system with very long "full GC" 
times, as long as1200sec real time (!!!).

We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat 7.0.64.

Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC   
-XX:+CMSIncrementalMode

No other custom memory-related settings are in place.

Looking at the GC log, the last few Full-GC entries are:

1367.020: [Full GC 1367.020: [CMS: 1178831K->527456K(1926784K), 
2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm : 
169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs] [Times: user=1.96 
sys=0.13, real=2.11 secs]

2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-mark: 
2.558/1212.733 secs] [*Times: user=113.05 sys=28.01, real=**1212.49 
**secs] **
*
3539.969: [Full GC 3539.969: [CMS3540.056: [CMS-concurrent-sweep: 
1.571/23.223 secs] [Times: user=6.12 sys=1.36, real=*23.21 secs*]

4070.456: [Full GC 4070.457: [CMS: 1252569K->591200K(1926784K), 
2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm : 
169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs] [Times: user=2.18 
sys=0.14, real=2.34 secs]


What can we do?

Thanks,

George





---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Re: Please help with Tomcat Garbage Collection

2016-11-15 Thread George I. Develekos 

Thanks, but, all I got was this "y"...


On 15-Nov-16 9:48 PM, pina.freder...@gmail.com wrote:

y

Sent from my HTC on T-  .  4G LTE

- Reply message -
From: "George I. Develekos" 
To: 
Subject: Please help with Tomcat Garbage Collection
Date: Tue, Nov 15, 2016 10:46 AM

Hello guys,

We are having problems on a production system with very long "full GC"
times, as long as1200sec real time (!!!).

We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat 7.0.64.

Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode

No other custom memory-related settings are in place.

Looking at the GC log, the last few Full-GC entries are:

1367.020: [Full GC 1367.020: [CMS: 1178831K->527456K(1926784K),
2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm :
169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs] [Times: user=1.96
sys=0.13, real=2.11 secs]

2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-mark:
2.558/1212.733 secs] [*Times: user=113.05 sys=28.01, real=**1212.49
**secs] **
*
3539.969: [Full GC 3539.969: [CMS3540.056: [CMS-concurrent-sweep:
1.571/23.223 secs] [Times: user=6.12 sys=1.36, real=*23.21 secs*]

4070.456: [Full GC 4070.457: [CMS: 1252569K->591200K(1926784K),
2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm :
169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs] [Times: user=2.18
sys=0.14, real=2.34 secs]


What can we do?

Thanks,

George





---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus



---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Please help with Tomcat Garbage Collection

2016-11-15 Thread Zdeněk Henek 
Is there running something else in that server? Real time should be never
bigger than user+sys

Z

On Tue, Nov 15, 2016, 16:46 George I. Develekos 
wrote:

> Hello guys,
>
> We are having problems on a production system with very long "full GC"
> times, as long as1200sec real time (!!!).
>
> We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat 7.0.64.
>
> Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC
> -XX:+CMSIncrementalMode
>
> No other custom memory-related settings are in place.
>
> Looking at the GC log, the last few Full-GC entries are:
>
> 1367.020: [Full GC 1367.020: [CMS: 1178831K->527456K(1926784K),
> 2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm :
> 169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs] [Times: user=1.96
> sys=0.13, real=2.11 secs]
>
> 2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-mark:
> 2.558/1212.733 secs] [*Times: user=113.05 sys=28.01, real=**1212.49
> **secs] **
> *
> 3539.969: [Full GC 3539.969: [CMS3540.056: [CMS-concurrent-sweep:
> 1.571/23.223 secs] [Times: user=6.12 sys=1.36, real=*23.21 secs*]
>
> 4070.456: [Full GC 4070.457: [CMS: 1252569K->591200K(1926784K),
> 2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm :
> 169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs] [Times: user=2.18
> sys=0.14, real=2.34 secs]
>
>
> What can we do?
>
> Thanks,
>
> George
>
>
>
>
>
> ---
> This email has been checked for viruses by Avast antivirus software.
> https://www.avast.com/antivirus
>

Re: Please help with Tomcat Garbage Collection

2016-11-15 Thread Christopher Schultz 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

George,

On 11/15/16 10:46 AM, George I. Develekos wrote:
> Hello guys,
> 
> We are having problems on a production system with very long "full
> GC" times, as long as1200sec real time (!!!).
> 
> We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat
> 7.0.64.
> 
> Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC 
> -XX:+CMSIncrementalMode
> 
> No other custom memory-related settings are in place.
> 
> Looking at the GC log, the last few Full-GC entries are:
> 
> 1367.020: [Full GC 1367.020: [CMS: 1178831K->527456K(1926784K), 
> 2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm : 
> 169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs] [Times:
> user=1.96 sys=0.13, real=2.11 secs]
> 
> 2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-mark: 
> 2.558/1212.733 secs] [*Times: user=113.05 sys=28.01,
> real=**1212.49 **secs] ** * 3539.969: [Full GC 3539.969:
> [CMS3540.056: [CMS-concurrent-sweep: 1.571/23.223 secs] [Times:
> user=6.12 sys=1.36, real=*23.21 secs*]
> 
> 4070.456: [Full GC 4070.457: [CMS: 1252569K->591200K(1926784K), 
> 2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm : 
> 169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs] [Times:
> user=2.18 sys=0.14, real=2.34 secs]
> 
> 
> What can we do?

1367.020 Full GC duration=2.11s
2579.317 Full GC duration=1212.49s

So your full GC immediately started another full GC that took 20 minutes
?

Are you only showing certain FULL GC activity from your log, or is
that everything?

CMS should have a mark and then a sweep each time, but your times
don't seem to add up.

also note that the whole point of CMS is that there isn't any
stop-the-world during the mark portion of the process.

Are you actually experiencing a problem, or are you just suffering
from instrumentor's remorse?

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJYK25oAAoJEBzwKT+lPKRYQNEQAJ8xASbPXtJUHy7xzhdYpwnr
M6wpCnYcls/cihN1p5OgJaV5G4EyNDGzoDky/ijW6o6ywlAkwAw0TOZKiUD6KN3q
LPq/3nHCvUr0769TuzkLpCe+VjYKmD4k4JGq9qqBLrKeMhfEprKnod+BR2ZKVK03
WmI6mqYKfkdPi3iR5GBv95Wo4SXAONGSnG06ISo+mPPOrLqH8QSrGocFH63/+nU8
Vx3yESyWL4FiQxFe+BowEnhu26p5IgA76DtgXs8XV+ojKnuM1liZPd6GXTjT6lE/
HkVg7P30sMzVf9P+0naDzHzZw1Pg19c3L5CymF1+WQPUF7Bk7iaOJNX9ZvVvhtpP
aMmyNVp2GUWVXby2rHIS2cA1h0Nq9ZtUZECx5k8dLCYlMgmRlFdgM5W+Un2nP2qf
WrVh0YdsMc9PrP5JSt8MdgxRhRk/WAADC4gTi78hqS6Fuz6I8R5tkXu2xsQA5ALZ
Cd/dvX2hRY4yJ00A3ooYE9sRciQXoSrQ6VvfWfU1k4kI2ZC4C5mDXASQTOdyCKHx
KtGs9hK56hitZSCBhCrDD8OhzmnIJF1E+FmtKwuTEmkc7cGiCXiTJI+coO+sWzzV
AljVYpG8YIDSD+gHrWJbf4Bs9EzfjI7EwvAtszWBC1SM8pAHBbyguCDhFOAbLlp0
gHTCnOGA7/Qvgd+nLLxC
=UeZj
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Please help with Tomcat Garbage Collection

2016-11-15 Thread George I. Develekos 


On 15-Nov-16 10:22 PM, Christopher Schultz wrote:


George,

On 11/15/16 10:46 AM, George I. Develekos wrote:

Hello guys,

We are having problems on a production system with very long "full
GC" times, as long as1200sec real time (!!!).

We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat
7.0.64.

Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode

No other custom memory-related settings are in place.

Looking at the GC log, the last few Full-GC entries are:

1367.020: [Full GC 1367.020: [CMS: 1178831K->527456K(1926784K),
2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm :
169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs] [Times:
user=1.96 sys=0.13, real=2.11 secs]

2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-mark:
2.558/1212.733 secs] [*Times: user=113.05 sys=28.01,
real=**1212.49 **secs] ** * 3539.969: [Full GC 3539.969:
[CMS3540.056: [CMS-concurrent-sweep: 1.571/23.223 secs] [Times:
user=6.12 sys=1.36, real=*23.21 secs*]

4070.456: [Full GC 4070.457: [CMS: 1252569K->591200K(1926784K),
2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm :
169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs] [Times:
user=2.18 sys=0.14, real=2.34 secs]


What can we do?

1367.020 Full GC duration=2.11s
2579.317 Full GC duration=1212.49s

So your full GC immediately started another full GC that took 20 minutes
?

Are you only showing certain FULL GC activity from your log, or is
that everything?

CMS should have a mark and then a sweep each time, but your times
don't seem to add up.

also note that the whole point of CMS is that there isn't any
stop-the-world during the mark portion of the process.

Are you actually experiencing a problem, or are you just suffering
from instrumentor's remorse?

- -chris



Chris,

What I listed is the result of the command:

grep "Full GC" gc.log

So (obviously) I have skipped other GC activity, i.e. whatever GC 
activity didn't include the "Full GC" string.


Yes we are having app trouble due to the GC delays so this is a real 
problem. Our application has real-time constraints so the GC delays 
cannot be tolerated. I selected those GC options _in order to avoid 
_long GC times.


Additionally, these periods coincide with high CPU for that JVM 
process.  From 5-20% CPU where it is normally, it jumps to 60% ore more. 
Once GC is done, our app rushes to catch up with tasks that had to wait 
for GC to finish.


Answering another question from a member who has kindly responded, yes 
the server is running other stuff. Basically it runs three tomcats, the 
main one being this one. It also runs a DB2 database that has 
close-to-zero activity.


George


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Re: Please help with Tomcat Garbage Collection

2016-11-15 Thread Zdeněk Henek 
I would start with moving this tomcat to its own hw.

Did you check swap? This long pauses could be because part of your heap is
swapped to hdd

Regards,
Zdenek Henek

On Tue, Nov 15, 2016, 21:37 George I. Develekos 
wrote:

>
> On 15-Nov-16 10:22 PM, Christopher Schultz wrote:
> >
> > George,
> >
> > On 11/15/16 10:46 AM, George I. Develekos wrote:
> >> Hello guys,
> >>
> >> We are having problems on a production system with very long "full
> >> GC" times, as long as1200sec real time (!!!).
> >>
> >> We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat
> >> 7.0.64.
> >>
> >> Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC
> >> -XX:+CMSIncrementalMode
> >>
> >> No other custom memory-related settings are in place.
> >>
> >> Looking at the GC log, the last few Full-GC entries are:
> >>
> >> 1367.020: [Full GC 1367.020: [CMS: 1178831K->527456K(1926784K),
> >> 2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm :
> >> 169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs] [Times:
> >> user=1.96 sys=0.13, real=2.11 secs]
> >>
> >> 2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-mark:
> >> 2.558/1212.733 secs] [*Times: user=113.05 sys=28.01,
> >> real=**1212.49 **secs] ** * 3539.969: [Full GC 3539.969:
> >> [CMS3540.056: [CMS-concurrent-sweep: 1.571/23.223 secs] [Times:
> >> user=6.12 sys=1.36, real=*23.21 secs*]
> >>
> >> 4070.456: [Full GC 4070.457: [CMS: 1252569K->591200K(1926784K),
> >> 2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm :
> >> 169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs] [Times:
> >> user=2.18 sys=0.14, real=2.34 secs]
> >>
> >>
> >> What can we do?
> > 1367.020 Full GC duration=2.11s
> > 2579.317 Full GC duration=1212.49s
> >
> > So your full GC immediately started another full GC that took 20 minutes
> > ?
> >
> > Are you only showing certain FULL GC activity from your log, or is
> > that everything?
> >
> > CMS should have a mark and then a sweep each time, but your times
> > don't seem to add up.
> >
> > also note that the whole point of CMS is that there isn't any
> > stop-the-world during the mark portion of the process.
> >
> > Are you actually experiencing a problem, or are you just suffering
> > from instrumentor's remorse?
> >
> > - -chris
> >
>
> Chris,
>
> What I listed is the result of the command:
>
> grep "Full GC" gc.log
>
> So (obviously) I have skipped other GC activity, i.e. whatever GC
> activity didn't include the "Full GC" string.
>
> Yes we are having app trouble due to the GC delays so this is a real
> problem. Our application has real-time constraints so the GC delays
> cannot be tolerated. I selected those GC options _in order to avoid
> _long GC times.
>
> Additionally, these periods coincide with high CPU for that JVM
> process.  From 5-20% CPU where it is normally, it jumps to 60% ore more.
> Once GC is done, our app rushes to catch up with tasks that had to wait
> for GC to finish.
>
> Answering another question from a member who has kindly responded, yes
> the server is running other stuff. Basically it runs three tomcats, the
> main one being this one. It also runs a DB2 database that has
> close-to-zero activity.
>
> George
>
>
> ---
> This email has been checked for viruses by Avast antivirus software.
> https://www.avast.com/antivirus
>

Re: Please help with Tomcat Garbage Collection

2016-11-15 Thread George I. Develekos 
The system does very little swapping, both when it's GC'ing and when 
it's not. Less than 100MB worth of swap is taken.


Giving Tomcat its own HW is not an option at this time, especially as 
there's no guarantee it'll solve the problem. Besides it would be a VM 
anyway, not physical dedicated HW.  The current server is also a VM.



On 15-Nov-16 10:55 PM, Zdeněk Henek wrote:

I would start with moving this tomcat to its own hw.

Did you check swap? This long pauses could be because part of your heap is
swapped to hdd

Regards,
Zdenek Henek

On Tue, Nov 15, 2016, 21:37 George I. Develekos 
wrote:


On 15-Nov-16 10:22 PM, Christopher Schultz wrote:

George,

On 11/15/16 10:46 AM, George I. Develekos wrote:

Hello guys,

We are having problems on a production system with very long "full
GC" times, as long as1200sec real time (!!!).

We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat
7.0.64.

Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode

No other custom memory-related settings are in place.

Looking at the GC log, the last few Full-GC entries are:

1367.020: [Full GC 1367.020: [CMS: 1178831K->527456K(1926784K),
2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm :
169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs] [Times:
user=1.96 sys=0.13, real=2.11 secs]

2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-mark:
2.558/1212.733 secs] [*Times: user=113.05 sys=28.01,
real=**1212.49 **secs] ** * 3539.969: [Full GC 3539.969:
[CMS3540.056: [CMS-concurrent-sweep: 1.571/23.223 secs] [Times:
user=6.12 sys=1.36, real=*23.21 secs*]

4070.456: [Full GC 4070.457: [CMS: 1252569K->591200K(1926784K),
2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm :
169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs] [Times:
user=2.18 sys=0.14, real=2.34 secs]


What can we do?

1367.020 Full GC duration=2.11s
2579.317 Full GC duration=1212.49s

So your full GC immediately started another full GC that took 20 minutes
?

Are you only showing certain FULL GC activity from your log, or is
that everything?

CMS should have a mark and then a sweep each time, but your times
don't seem to add up.

also note that the whole point of CMS is that there isn't any
stop-the-world during the mark portion of the process.

Are you actually experiencing a problem, or are you just suffering
from instrumentor's remorse?

- -chris


Chris,

What I listed is the result of the command:

grep "Full GC" gc.log

So (obviously) I have skipped other GC activity, i.e. whatever GC
activity didn't include the "Full GC" string.

Yes we are having app trouble due to the GC delays so this is a real
problem. Our application has real-time constraints so the GC delays
cannot be tolerated. I selected those GC options _in order to avoid
_long GC times.

Additionally, these periods coincide with high CPU for that JVM
process.  From 5-20% CPU where it is normally, it jumps to 60% ore more.
Once GC is done, our app rushes to catch up with tasks that had to wait
for GC to finish.

Answering another question from a member who has kindly responded, yes
the server is running other stuff. Basically it runs three tomcats, the
main one being this one. It also runs a DB2 database that has
close-to-zero activity.

George


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus




---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Apache Tomcat Version 7.0.59 on CentOS 6.8 to handle 2k requests/second

2016-11-15 Thread Hedrick, Brooke - 43 
Chris,

>-Original Message-
>From: Christopher Schultz [mailto:ch...@christopherschultz.net] 
>Sent: Tuesday, November 15, 2016 12:53 PM
>To: Tomcat Users List 
>Subject: Re: Apache Tomcat Version 7.0.59 on CentOS 6.8 to handle 2k 
>requests/second
>
>-BEGIN PGP SIGNED MESSAGE-
>Hash: SHA256
>
>Brooke,
>
>On 11/15/16 1:17 PM, Hedrick, Brooke - 43 wrote:
>> -Original Message- From: Kaushal Shriyan 
>> [mailto:kaushalshri...@gmail.com] Sent: Monday, November 14, 2016
>> 8:46 AM To: Tomcat Users List  Subject:
>> Re: Apache Tomcat Version 7.0.59 on CentOS 6.8 to handle 2k 
>> requests/second
>> 
>> ...
>> 
>>> Please suggest me any open source profiling application which i can 
>>> start with. I will appreciate if you can help me understand the 
>>> relation between 2k requests/second vs cpu >cores of the server.
>>> 
>>> Thanks again in advance.
>>> 
>>> Regards,
>>> 
>>> Kaushal
>> 
>> A couple I have used: 1. Apache Bench (ab/abs very basic ) ships with 
>> Apache. 2. Apache JMeter.  Much more in-depth and flexible in how you 
>> load test and the results collected.
>
>Those are load-generators, not profilers.

Great point.  You are correct.  Kaushal will need both.  Profiling will help 
find hotspots.  I have used Eclipse and jvmtop, but prefer JVisualVM ( ships 
with JDK ) with the sampling plugin.  Decent, cheap to run, high level numbers 
that help identify hotspots.  But, it will depend on how small of a needle you 
are looking for.

Even once you have profile, you will want to load test.  In fact, you may even 
want to run a load test while you profile - just be aware that profilers have a 
cost that varies from small to molasses behavior inducing.

In any case hopefully we are giving Kaushal something useful to go on.

>
>I have never been happy with any open-source profilers. There are several good 
>non-free options, though.
>
>- -chris
>-BEGIN PGP SIGNATURE-
>Comment: GPGTools - http://gpgtools.org
>Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
>iQIcBAEBCAAGBQJYK1mSAAoJEBzwKT+lPKRY2PUP/1Uw7C2TiN6x37SejT+pyc1x
>7N6n5FoUyB8RdrVIZlkEsbTzaWKHzq5EisxczzblleTip8jNBwSXaTjRQ+CgQWjU
>//UinQwf7bhCZCzKuSlzq51HOYSkoIPLQtg2NBNUGtqXJ7LtbwdXkBRbMQxPeiMV
>+RSNmY1CoTElvxJeYfqcr4A1R0PRTgoOMkz7CSj2Hf2H7XsccY1bFH/ofiFGbpKh
>v0GTRiq9TCNGVBtVhAaU5339KRBuxB5QkTgZF/7KQU719d6jsmsB+u7W1St2gXWc
>zaIUg75zC6KP+IxlXkkPldh6dDruw0drG8q1RUEai972JZS9e1/kLnFaGne02pUW
>UPSIgCeSXPRiSPGfLE5e1FkQxmrQmuMlWo6imwvbSQ0Rzgy8gKVv2seYsM9jqsko
>BzWn2iri4YPVzAunKfZIekOV4xL5Bev08PND4jdBkPIWwTF9Tz1twnBxK33q+Mi1
>qEYSezzS4PmFQAtNnwbRBGolqCiJMITtfP4nkO3QsWfzM0/CZo4M587onXnhawbA
>yzxlG5tzCHYQhr6PQD4POA+bH0RbWI4rDm4DqlD5RRDjVC0BMkxiejwXH1SIRApP
>4vcNVipxr7MzngDSsAwbG+Jz4dUGlNas9o5WjoAu8Jya/PYkaZ4LEv/Dn0DWgTnb
>nvYTZz8td79PoGkK2a45
>=4KGB
>-END PGP SIGNATURE-
>
>-
>To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>For additional commands, e-mail: users-h...@tomcat.apache.org
>


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Tomcat 8.0.39 and tomcat 8.5.8 fails handling requsest

2016-11-15 Thread Zdeněk Henek 
Hi,

we are using tomcat 8.0.30 without problems.

I have tested upgrade to 8.0.38 today and I got this error
More env. details JDK 8, tested on both Linux and Windows using different
JDK 8 updates (71, 111).

15-Nov-2016 17:14:51.189 INFO [http-nio-8080-exec-2]
org.apache.coyote.http11.AbstractHttp11Processor.process Error parsing HTTP
request header
 Note: further occurrences of HTTP header parsing errors will be logged at
DEBUG level.
 java.lang.IllegalArgumentException: Invalid character found in the request
target. The valid characters are defined in RFC 7230 and RFC 3986
at
org.apache.coyote.http11.AbstractNioInputBuffer.parseRequestLine(AbstractNioInputBuffer.java:283)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1017)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1520)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1476)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)


The parameter in the request is this

/list?criteria={%22$type%22:%22Equal%22,%22attr%22:%22id%22,%22value%22:101}


Looks like this commit caused the exception
https://github.com/apache/tomcat80/commit/779d5d34e68e50d2f721897050b147106992f566

The commit message says:
Add additional checks for valid characters to the HTTP request line
parsing so invalid request lines are rejected sooner.

We don't get any error in 8.0.30 using same request.

The state in 8.0.30 was bug or 8.0.38 should process parameter

criteria={%22$type%22:%22Equal%22,%22attr%22:%22id%22,%22value%22:101}

?

Thanks.

Regards,
Zdenek Henek

Re: 8.5.4 to 8.5.5 SSL Issue

2016-11-15 Thread William Boyd 
On Tue, Nov 15, 2016 at 10:50 AM, Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> William,
>
> On 11/14/16 7:18 PM, William Boyd wrote:
> > First, I'd like to thank everyone for the help.
> >
> > Is it now safe to say that the behaviour we've been taking
> > advantage of is undocumented and will no long be supported?
> >
> > Also, for those that hit this thread and need to get HTTPS working
> > with a *self-signed* certificate in a dev environment...
> >
> > Here's what worked for me: 1. Copy
> > %JAVA_HOME%\jre\lib\security\cacerts some place (say C:\keystore)
> > 2. Create a *self-signed* certificate with %JAVA_HOME%\bin\keytool
> > -genkeypair -keyalg RSA -alias myAlias -keystore
> > "C:\keystore\keystore.jsk" -storepass changeit -validity 360
> > -keysize 2048 -dname
> > CN=localhost,OU=OrgUnit,O=Org,L=City,ST=State,C=Country 3. Export
> > the myAlias certificate with %JAVA_HOME%\bin\keytool -export -alias
> > myAlias -keystore C:\keystore\keystore.jsk -rfc -file
> > C:\keystore\myAlias.cer 4. Import the myAlias certificate into your
> > copy of cacerts with: %JAVA_HOME%\bin\keytool -import -alias
> > myAlias -keystore C:\keystore\cacerts -file
> > C:\keystore\myAlias.cer 5. Add this to setclasspath.bat in tomcat:
> > set JAVA_OPTS=%JAVA_OPTS%
> > -Djavax.net.ssl.trustStore="C:\keystore\cacert" 6. In server.xml,
> > add these attributes to the Connector element
> > keystoreFile="C:\keystore\keystore.jsk" keystorePass="changeit"
> > keyAlias="myAlias" scheme="https" secure="true" SSLEnabled="true"
> > clientAuth="false" sslProtocol="TLS"
> >
> > Now you should be good to go in with HTTPS in Tomcat 8.5.5 +
>
> This is exactly what I suggested, except that you set the trust store
> using a system property instead of using truststoreFile in the
> . You do not need to duplicate the JRE's trust store. You
> only need your own single certificate in your local truststore.
>
> Can you use keystoreFile and truststoreFile separately pointing to
> those files and re-check that it works? Perhaps Tomcat chokes when
> using the same file for both. That's what I'd like to verify.
>
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJYK1kHAAoJEBzwKT+lPKRYa48P/jl6hWa1mj5HCnawJZ3mHbjX
> ADdXYl45aym/E6QV/n99XYVYG2q+ZN9w0XDVP54lQhQhcgOUtCiPbTHEcYSYdwrU
> kLYMc3Ge8Jt7/zDMvem+pKYkHMvyHbspVqSujZ4uJ3Ozr9mYD89hSFgxqG0iYSE+
> 5c0pvz1nW4Pt1F4A/+WETkL4Y5Xrq1Vn1LSAxAZoYiU/o93nVos7etIBUO9E430+
> GihbhvkpS/yBitvrir/YacvWauBxpi30wR++6ZNAhpzlb+j90dk3i6iPcDO6K1f2
> SNeqZATJDlXyU1hEksW4UxWLhtUeekqmJEiEqqWCYxNz9lwJG9f4kILUrzsZexlu
> FmP2o4IxWTBcgOUs5Km5DlfYwogJmlRhqQoOlg2JOpv+KIb67DX+PuY6bhGomDff
> YQ03Y7WQcjNZ/uOIoadAkXxKRaRHmuz2KkPYwgDutOgxtJV1jNxTT3A3znGT1cWN
> yekjXHOpe2FdXnaoG0X7mTpvx5AhkHN9mRdW+5/ZBpPzUN0M7zy8oBEpLtZKfrTJ
> k40Xz70DnNxBP3XS/1w7DJ1H3/FBxNdatVVbbcJ/+lS/NiS4Gn2kMAZgrCuZrUsn
> FdpdyCwq3VLJ2X9LVBR03rJOyPIiybANNjfhPpiEMC9uQu2ENm4A4Hm1p/cXdpo3
> 2J2O1AQA7tfew10t3F4K
> =a+Um
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
Hi Christopher,

Thanks, I'm glad to hear I'm heading in the right direction. :)

I tested some more configuration.

Firstly, I removed the system property but retained these Connector
attributes

   keystoreFile="C:\keystore\keystore.jsk" keystorePass="changeit"
keyAlias="myAlias"
   scheme="https" secure="true" SSLEnabled="true" clientAuth="false"
sslProtocol="TLS"

Then I tried the following:

1. adding truststoreFile="C:\keystore\cacerts" to my Connector and got the
following error in an IE11 browser
   Caught Exception (javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException:
   PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException:
   unable to find valid certification path to requested target): ;
   nested exception is: javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException:
   PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException:
   unable to find valid certification path to requested target

2. adding truststoreFile="C:\keystore\myAlias.cer" to my Connector and got
the following error at startup
   15-Nov-2016 12:34:57.379 SEVERE [main]
org.apache.coyote.AbstractProtocol.init
   Failed to initialize end point associated with ProtocolHandler
["https-openssl-nio-8001"]
   java.lang.IllegalArgumentException: java.io.IOException: Invalid
keystore format
at
org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:103)
   :
   Caused by: java.io.IOException: Invalid keystore format
at
sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:658)
at
sun.security.pr

RE: Please help with Tomcat Garbage Collection

2016-11-15 Thread John.E.Gregg 



> -Original Message-
> From: George I. Develekos [mailto:gdevele...@omilia.com]
> Sent: Tuesday, November 15, 2016 3:00 PM
> To: users@tomcat.apache.org
> Subject: Re: Please help with Tomcat Garbage Collection
> 
> The system does very little swapping, both when it's GC'ing and when it's not.
> Less than 100MB worth of swap is taken.
> 
> Giving Tomcat its own HW is not an option at this time, especially as there's 
> no
> guarantee it'll solve the problem. Besides it would be a VM anyway, not 
> physical
> dedicated HW.  The current server is also a VM.
> 
> 
> On 15-Nov-16 10:55 PM, Zdeněk Henek wrote:
> > I would start with moving this tomcat to its own hw.
> >
> > Did you check swap? This long pauses could be because part of your
> > heap is swapped to hdd
> >
> > Regards,
> > Zdenek Henek
> >
> > On Tue, Nov 15, 2016, 21:37 George I. Develekos
> > 
> > wrote:
> >
> >> On 15-Nov-16 10:22 PM, Christopher Schultz wrote:
> >>> George,
> >>>
> >>> On 11/15/16 10:46 AM, George I. Develekos wrote:
>  Hello guys,
> 
>  We are having problems on a production system with very long "full
>  GC" times, as long as1200sec real time (!!!).
> 
>  We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat
>  7.0.64.
> 
>  Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC
>  -XX:+CMSIncrementalMode
> 
>  No other custom memory-related settings are in place.
> 
>  Looking at the GC log, the last few Full-GC entries are:
> 
>  1367.020: [Full GC 1367.020: [CMS: 1178831K->527456K(1926784K),
>  2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm :
>  169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs] [Times:
>  user=1.96 sys=0.13, real=2.11 secs]
> 
>  2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-mark:
>  2.558/1212.733 secs] [*Times: user=113.05 sys=28.01,
>  real=**1212.49 **secs] ** * 3539.969: [Full GC 3539.969:
>  [CMS3540.056: [CMS-concurrent-sweep: 1.571/23.223 secs] [Times:
>  user=6.12 sys=1.36, real=*23.21 secs*]
> 
>  4070.456: [Full GC 4070.457: [CMS: 1252569K->591200K(1926784K),
>  2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm :
>  169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs] [Times:
>  user=2.18 sys=0.14, real=2.34 secs]
> 
> 
>  What can we do?
> >>> 1367.020 Full GC duration=2.11s
> >>> 2579.317 Full GC duration=1212.49s
> >>>
> >>> So your full GC immediately started another full GC that took 20
> >>> minutes ?
> >>>
> >>> Are you only showing certain FULL GC activity from your log, or is
> >>> that everything?
> >>>
> >>> CMS should have a mark and then a sweep each time, but your times
> >>> don't seem to add up.
> >>>
> >>> also note that the whole point of CMS is that there isn't any
> >>> stop-the-world during the mark portion of the process.
> >>>
> >>> Are you actually experiencing a problem, or are you just suffering
> >>> from instrumentor's remorse?
> >>>
> >>> - -chris
> >>>
> >> Chris,
> >>
> >> What I listed is the result of the command:
> >>
> >> grep "Full GC" gc.log
> >>
> >> So (obviously) I have skipped other GC activity, i.e. whatever GC
> >> activity didn't include the "Full GC" string.
> >>
> >> Yes we are having app trouble due to the GC delays so this is a real
> >> problem. Our application has real-time constraints so the GC delays
> >> cannot be tolerated. I selected those GC options _in order to avoid
> >> _long GC times.
> >>
> >> Additionally, these periods coincide with high CPU for that JVM
> >> process.  From 5-20% CPU where it is normally, it jumps to 60% ore more.
> >> Once GC is done, our app rushes to catch up with tasks that had to
> >> wait for GC to finish.
> >>
> >> Answering another question from a member who has kindly responded,
> >> yes the server is running other stuff. Basically it runs three
> >> tomcats, the main one being this one. It also runs a DB2 database
> >> that has close-to-zero activity.
> >>
> >> George
> >>
> >>

It might be helpful if you could post a larger chunk of your GC log, at least 
long enough to cover the start and end of the CMS phases and maybe even more.  
Additionally, try using a tool like GCViewer to analyze the log.

How many CPUs do you have?  60% CPU usage isn't usually a big deal.

Like Chris already said, this is not a stop-the-world phase, so your 
application should continue working in parallel with the garbage collector.

Looks like your young generation is only 150MB (2080128k - 1926784k.)  That's 
very small for a 2-5GB heap.  Are you explicitly setting it somewhere or is the 
JVM choosing that for you?  It's so small that your old generation might be 
filling up faster that it should, leading to more frequent full collections.  
You could try setting the young generation to something like 25-50% of the 
total heap.  You'd get a lot of small pauses as the young gen is collected but 
fewer long ones.

John

Re: Please help with Tomcat Garbage Collection

2016-11-15 Thread George I. Develekos 



On 15-Nov-16 11:45 PM, john.e.gr...@wellsfargo.com wrote:




-Original Message-
From: George I. Develekos [mailto:gdevele...@omilia.com]
Sent: Tuesday, November 15, 2016 3:00 PM
To: users@tomcat.apache.org
Subject: Re: Please help with Tomcat Garbage Collection

The system does very little swapping, both when it's GC'ing and when it's not.
Less than 100MB worth of swap is taken.

Giving Tomcat its own HW is not an option at this time, especially as there's no
guarantee it'll solve the problem. Besides it would be a VM anyway, not physical
dedicated HW.  The current server is also a VM.


On 15-Nov-16 10:55 PM, Zdeněk Henek wrote:

I would start with moving this tomcat to its own hw.

Did you check swap? This long pauses could be because part of your
heap is swapped to hdd

Regards,
Zdenek Henek

On Tue, Nov 15, 2016, 21:37 George I. Develekos

wrote:


On 15-Nov-16 10:22 PM, Christopher Schultz wrote:

George,

On 11/15/16 10:46 AM, George I. Develekos wrote:

Hello guys,

We are having problems on a production system with very long "full
GC" times, as long as1200sec real time (!!!).

We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat
7.0.64.

Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode

No other custom memory-related settings are in place.

Looking at the GC log, the last few Full-GC entries are:

1367.020: [Full GC 1367.020: [CMS: 1178831K->527456K(1926784K),
2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm :
169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs] [Times:
user=1.96 sys=0.13, real=2.11 secs]

2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-mark:
2.558/1212.733 secs] [*Times: user=113.05 sys=28.01,
real=**1212.49 **secs] ** * 3539.969: [Full GC 3539.969:
[CMS3540.056: [CMS-concurrent-sweep: 1.571/23.223 secs] [Times:
user=6.12 sys=1.36, real=*23.21 secs*]

4070.456: [Full GC 4070.457: [CMS: 1252569K->591200K(1926784K),
2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm :
169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs] [Times:
user=2.18 sys=0.14, real=2.34 secs]


What can we do?

1367.020 Full GC duration=2.11s
2579.317 Full GC duration=1212.49s

So your full GC immediately started another full GC that took 20
minutes ?

Are you only showing certain FULL GC activity from your log, or is
that everything?

CMS should have a mark and then a sweep each time, but your times
don't seem to add up.

also note that the whole point of CMS is that there isn't any
stop-the-world during the mark portion of the process.

Are you actually experiencing a problem, or are you just suffering
from instrumentor's remorse?

- -chris


Chris,

What I listed is the result of the command:

grep "Full GC" gc.log

So (obviously) I have skipped other GC activity, i.e. whatever GC
activity didn't include the "Full GC" string.

Yes we are having app trouble due to the GC delays so this is a real
problem. Our application has real-time constraints so the GC delays
cannot be tolerated. I selected those GC options _in order to avoid
_long GC times.

Additionally, these periods coincide with high CPU for that JVM
process.  From 5-20% CPU where it is normally, it jumps to 60% ore more.
Once GC is done, our app rushes to catch up with tasks that had to
wait for GC to finish.

Answering another question from a member who has kindly responded,
yes the server is running other stuff. Basically it runs three
tomcats, the main one being this one. It also runs a DB2 database
that has close-to-zero activity.

George



It might be helpful if you could post a larger chunk of your GC log, at least 
long enough to cover the start and end of the CMS phases and maybe even more.  
Additionally, try using a tool like GCViewer to analyze the log.

How many CPUs do you have?  60% CPU usage isn't usually a big deal.

Like Chris already said, this is not a stop-the-world phase, so your 
application should continue working in parallel with the garbage collector.

Looks like your young generation is only 150MB (2080128k - 1926784k.)  That's 
very small for a 2-5GB heap.  Are you explicitly setting it somewhere or is the 
JVM choosing that for you?  It's so small that your old generation might be 
filling up faster that it should, leading to more frequent full collections.  
You could try setting the young generation to something like 25-50% of the 
total heap.  You'd get a lot of small pauses as the young gen is collected but 
fewer long ones.

John


John,

I have a couple of cores. One would be enough for what the server does.

I'll post a larger log chunk tomorrow.  I'll look into GCViewer, thanks.

I don't set the young gen, It's selected by the JVM. I don't even know 
what that is, to be honest. I do too many things for my company to know 
such esoteric things about each one of them. Server admin, network 
admin, DB admin, all admin


Nevertheless, if setting the Young gen makes GC less frequent it won't 
really help, because still I'll hav

RE: 8.5.4 to 8.5.5 SSL Issue

2016-11-15 Thread John.E.Gregg 


> -Original Message-
> From: William Boyd [mailto:william.b...@gmail.com]
> Sent: Tuesday, November 15, 2016 3:44 PM
> To: Tomcat Users List
> Subject: Re: 8.5.4 to 8.5.5 SSL Issue
> 
> On Tue, Nov 15, 2016 at 10:50 AM, Christopher Schultz <
> ch...@christopherschultz.net> wrote:
> 
> > -BEGIN PGP SIGNED MESSAGE-
> > Hash: SHA256
> >
> > William,
> >
> > On 11/14/16 7:18 PM, William Boyd wrote:
> > > First, I'd like to thank everyone for the help.
> > >
> > > Is it now safe to say that the behaviour we've been taking advantage
> > > of is undocumented and will no long be supported?
> > >
> > > Also, for those that hit this thread and need to get HTTPS working
> > > with a *self-signed* certificate in a dev environment...
> > >
> > > Here's what worked for me: 1. Copy
> > > %JAVA_HOME%\jre\lib\security\cacerts some place (say C:\keystore) 2.
> > > Create a *self-signed* certificate with %JAVA_HOME%\bin\keytool
> > > -genkeypair -keyalg RSA -alias myAlias -keystore
> > > "C:\keystore\keystore.jsk" -storepass changeit -validity 360
> > > -keysize 2048 -dname
> > > CN=localhost,OU=OrgUnit,O=Org,L=City,ST=State,C=Country 3. Export
> > > the myAlias certificate with %JAVA_HOME%\bin\keytool -export -alias
> > > myAlias -keystore C:\keystore\keystore.jsk -rfc -file
> > > C:\keystore\myAlias.cer 4. Import the myAlias certificate into your
> > > copy of cacerts with: %JAVA_HOME%\bin\keytool -import -alias myAlias
> > > -keystore C:\keystore\cacerts -file C:\keystore\myAlias.cer 5. Add
> > > this to setclasspath.bat in tomcat:
> > > set JAVA_OPTS=%JAVA_OPTS%
> > > -Djavax.net.ssl.trustStore="C:\keystore\cacert" 6. In server.xml,
> > > add these attributes to the Connector element
> > > keystoreFile="C:\keystore\keystore.jsk" keystorePass="changeit"
> > > keyAlias="myAlias" scheme="https" secure="true" SSLEnabled="true"
> > > clientAuth="false" sslProtocol="TLS"
> > >
> > > Now you should be good to go in with HTTPS in Tomcat 8.5.5 +
> >
> > This is exactly what I suggested, except that you set the trust store
> > using a system property instead of using truststoreFile in the
> > . You do not need to duplicate the JRE's trust store. You
> > only need your own single certificate in your local truststore.
> >
> > Can you use keystoreFile and truststoreFile separately pointing to
> > those files and re-check that it works? Perhaps Tomcat chokes when
> > using the same file for both. That's what I'd like to verify.
> >
> > - -chris
> > -BEGIN PGP SIGNATURE-
> > Comment: GPGTools - http://gpgtools.org
> > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
> >
> >
> iQIcBAEBCAAGBQJYK1kHAAoJEBzwKT+lPKRYa48P/jl6hWa1mj5HCnawJZ3mHbjX
> >
> ADdXYl45aym/E6QV/n99XYVYG2q+ZN9w0XDVP54lQhQhcgOUtCiPbTHEcYSYdwr
> U
> >
> kLYMc3Ge8Jt7/zDMvem+pKYkHMvyHbspVqSujZ4uJ3Ozr9mYD89hSFgxqG0iYSE+
> > 5c0pvz1nW4Pt1F4A/+WETkL4Y5Xrq1Vn1LSAxAZoYiU/o93nVos7etIBUO9E430+
> > GihbhvkpS/yBitvrir/YacvWauBxpi30wR++6ZNAhpzlb+j90dk3i6iPcDO6K1f2
> > SNeqZATJDlXyU1hEksW4UxWLhtUeekqmJEiEqqWCYxNz9lwJG9f4kILUrzsZexlu
> >
> FmP2o4IxWTBcgOUs5Km5DlfYwogJmlRhqQoOlg2JOpv+KIb67DX+PuY6bhGomDf
> f
> >
> YQ03Y7WQcjNZ/uOIoadAkXxKRaRHmuz2KkPYwgDutOgxtJV1jNxTT3A3znGT1cW
> N
> >
> yekjXHOpe2FdXnaoG0X7mTpvx5AhkHN9mRdW+5/ZBpPzUN0M7zy8oBEpLtZKfrT
> J
> > k40Xz70DnNxBP3XS/1w7DJ1H3/FBxNdatVVbbcJ/+lS/NiS4Gn2kMAZgrCuZrUsn
> >
> FdpdyCwq3VLJ2X9LVBR03rJOyPIiybANNjfhPpiEMC9uQu2ENm4A4Hm1p/cXdpo3
> > 2J2O1AQA7tfew10t3F4K
> > =a+Um
> > -END PGP SIGNATURE-
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
> >
> Hi Christopher,
> 
> Thanks, I'm glad to hear I'm heading in the right direction. :)
> 
> I tested some more configuration.
> 
> Firstly, I removed the system property but retained these Connector attributes
> 
>keystoreFile="C:\keystore\keystore.jsk" keystorePass="changeit"
> keyAlias="myAlias"
>scheme="https" secure="true" SSLEnabled="true" clientAuth="false"
> sslProtocol="TLS"
> 
> Then I tried the following:
> 
> 1. adding truststoreFile="C:\keystore\cacerts" to my Connector and got the
> following error in an IE11 browser
>Caught Exception (javax.net.ssl.SSLHandshakeException:
> sun.security.validator.ValidatorException:
>PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException:
>unable to find valid certification path to requested target): ;
>nested exception is: javax.net.ssl.SSLHandshakeException:
> sun.security.validator.ValidatorException:
>PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException:
>unable to find valid certification path to requested target
> 
> 2. adding truststoreFile="C:\keystore\myAlias.cer" to my Connector and got the
> following error at startup
>15-Nov-2016 12:34:57.379 SEVERE [main]
> org.apache.coyote.AbstractProtocol.init
>Failed to initializ

Problems with SSL configuration

2016-11-15 Thread Steve Willett 
I am trying to set up a stand-alone Tomcat server (apparently 7.0.53).  
When I set up a simple Connector on port 8443 (no specified ciphers, and 
a simple sslProtocol="TLS") using a DigiCert Certificate I can connect.


However, if I test it with QualSys, I get an F rating because of the 
accepted insecure cipher suites.  However, when I try to use "approved" 
suites, the server can't be reached.


Connector configuration;
protocol="org.apache.coyote.http11.Http11Protocol"

   maxThreads="150" SSLEnabled="true" scheme="https" secure="true"
   clientAuth="false" keyAlias="server"
keystoreFile="/usr/share/tomcat7/conf/QA_YOURSPORTSLEAGUE_COM.jks" 
keystorePass=""

   sslEnabledProtocols="TLSv1, TLSv1.1, TLSv1.2"
ciphers="TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384,TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384,TLS_DHE_DSS_WITH_AES_256_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_DSS_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256,TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_RC4_128_SHA,TLS_ECDH_ECDSA_WITH_RC4_128_SHA,TLS_ECDH_RSA_WITH_RC4_128_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384,TLS_DHE_DSS_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_DSS_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,TLS_EMPTY_RENEGOTIATION_INFO_SCSVF"
/>

When I try to connect to the site with Chrome I get:


 This site can’t be reached

*qa.yoursportsleague.com*unexpectedly closed the connection.



I also have configured it to require SSL:




Protected Context
/*



CONFIDENTIAL





Any thoughts?


--
*Steve Willett*
YourSportsLeague.com

RE: Problems with SSL configuration

2016-11-15 Thread John.E.Gregg 
Enable verbose SSL.

Start Tomcat with -Djavax.net.debug=ssl. That will print a lot of info to 
catalina.out.

You could also do the same thing on the client side if you used a java client, 
or something similar with OpenSSL, curl, etc.



-Original Message-
From: Steve Willett 
[st...@yoursportsleague.com]
Sent: Tuesday, November 15, 2016 05:48 PM Central Standard Time
To: users@tomcat.apache.org
Subject: Problems with SSL configuration


I am trying to set up a stand-alone Tomcat server (apparently 7.0.53).
When I set up a simple Connector on port 8443 (no specified ciphers, and
a simple sslProtocol="TLS") using a DigiCert Certificate I can connect.

However, if I test it with QualSys, I get an F rating because of the
accepted insecure cipher suites.  However, when I try to use "approved"
suites, the server can't be reached.

Connector configuration;
 

When I try to connect to the site with Chrome I get:


  This site can’t be reached

*qa.yoursportsleague.com*unexpectedly closed the connection.



I also have configured it to require SSL:




Protected Context
/*



CONFIDENTIAL





Any thoughts?


--
*Steve Willett*
YourSportsLeague.com

Re: Problems with SSL configuration

2016-11-15 Thread Mark Thomas 
On 16/11/2016 00:47, Steve Willett wrote:
> I am trying to set up a stand-alone Tomcat server (apparently 7.0.53). 
> When I set up a simple Connector on port 8443 (no specified ciphers, and
> a simple sslProtocol="TLS") using a DigiCert Certificate I can connect.
> 
> However, if I test it with QualSys, I get an F rating because of the
> accepted insecure cipher suites.  However, when I try to use "approved"
> suites, the server can't be reached.

Are those "approved" cipher suites supported by the JVM you are using?

This might help:
http://people.apache.org/~markt/dev/TLSInfo.java

As might this:
https://wiki.apache.org/tomcat/Security/Ciphers

Mark


> 
> Connector configuration;
>  protocol="org.apache.coyote.http11.Http11Protocol"
>maxThreads="150" SSLEnabled="true" scheme="https" secure="true"
>clientAuth="false" keyAlias="server"
> keystoreFile="/usr/share/tomcat7/conf/QA_YOURSPORTSLEAGUE_COM.jks"
> keystorePass=""
>sslEnabledProtocols="TLSv1, TLSv1.1, TLSv1.2"
> ciphers="TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384,TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384,TLS_DHE_DSS_WITH_AES_256_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_DSS_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256,TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_RC4_128_SHA,TLS_ECDH_ECDSA_WITH_RC4_128_SHA,TLS_ECDH_RSA_WITH_RC4_128_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384,TLS_DHE_DSS_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_DSS_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,TLS_EMPTY_RENEGOTIATION_INFO_SCSVF"
> 
> />
> 
> When I try to connect to the site with Chrome I get:
> 
> 
>  This site can’t be reached
> 
> *qa.yoursportsleague.com*unexpectedly closed the connection.
> 
> 
> 
> I also have configured it to require SSL:
> 
> 
> 
> 
> Protected Context
> /*
> 
> 
> 
> CONFIDENTIAL
> 
> 
> 
> 
> 
> Any thoughts?
> 
>