[
https://issues.apache.org/jira/browse/CXF-7135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dan Salt updated CXF-7135:
--------------------------
Description:
Our platform allows the user to switch transports for a particular proxy based
on configuration. We currently allow HTTP and JMS as possible options.
During recent performance testing, we've hit a bug which causes JAXRS Clients
over JMS to fail with a ConcurrentModificationException:
A test case is attached that reproduces our problem.
Our test executes requests across multiple concurrent threads, switching
between JMS and HTTP for both JAXWS and JAXRS Client Proxies. In the test case:
- JAXWS over both HTTP and JMS works fine, both in single and multi-threaded
tests.
- JAXRS works fine across single and multi-threaded tests over HTTP with the
'ThreadSafe' flag set on the Client Factory.
- JAXRS over JMS fails in both single-threaded and multi-threaded mode with
error (1) below. In multi-threaded, also fails with (2) below.
1) The ConnectionFactoryFeature fails to properly set the JMS Connection
Factory on the Conduit and results in an error at send-time because no
Connection Factory is set. This is because in JAXRS Clients, the
InterceptorProvider is neither a Client or Server, it is a ClientConfiguration,
so the JMS ConnectionFactoryFeature does not fire either of it's methods:
public void initialize(Client client, Bus bus)
public void initialize(Server server, Bus bus)
We have worked around this in our code by sub-classing and overriding the
method:
public void initialize(InterceptorProvider interceptorProvider, Bus bus)
.. and adding code similar to that in the ConnectionFactory feature. This
resolves the missing ConnectionFactory problem and JAXRS-over-JMS works fine in
single-threaded mode.
2) Under multi-threaded conditions, the JAXRS Clients fail with the following
exception:
javax.ws.rs.client.ResponseProcessingException: Problem with reading the data,
class java.lang.String, ContentType: text/plain.
at
org.apache.cxf.jaxrs.impl.ResponseImpl.reportMessageHandlerProblem(ResponseImpl.java:438)
at
org.apache.cxf.jaxrs.impl.ResponseImpl.doReadEntity(ResponseImpl.java:378)
at
org.apache.cxf.jaxrs.client.AbstractClient.readBody(AbstractClient.java:521)
at
org.apache.cxf.jaxrs.client.ClientProxyImpl.handleResponse(ClientProxyImpl.java:817)
at
org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:760)
at
org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:228)
at com.sun.proxy.$Proxy24.doSimpleMethod(Unknown Source)
at
com.ge.testcase.ThreadingTest$ThreadedRSJMSTest.run(ThreadingTest.java:132)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429)
at java.util.HashMap$EntryIterator.next(HashMap.java:1463)
at java.util.HashMap$EntryIterator.next(HashMap.java:1461)
at java.util.HashMap.putMapEntries(HashMap.java:511)
at java.util.HashMap.putAll(HashMap.java:784)
at org.apache.cxf.message.MessageImpl$1.putAll(MessageImpl.java:188)
at
org.apache.cxf.message.MessageImpl.calcContextCache(MessageImpl.java:213)
at
org.apache.cxf.message.MessageImpl.getContextualProperty(MessageImpl.java:174)
at
org.apache.cxf.jaxrs.impl.HttpHeadersImpl.getRequestHeaders(HttpHeadersImpl.java:172)
at
org.apache.cxf.jaxrs.utils.JAXRSUtils.readFromMessageBodyReader(JAXRSUtils.java:1345)
at
org.apache.cxf.jaxrs.impl.ResponseImpl.doReadEntity(ResponseImpl.java:369)
... 7 more
This occurs even though the 'ThreadSafe' property is set via:
rsClientFactory.setThreadSafe(true);
I've done quite a large amount of digging through the code to try and narrow
down the scope of the problem, and the results above is as far as I could get,
unfortunately. My guess/assumption is that there are some JMS-specific objects
that are being added to the Request/Response context which are being
added/removed in a non-thread safe way. Purely a guess, though.
Finally, one important factor is that this error only happens on the latest
3.0.x release (3.0.11). When I switch my test case to 3.1.8, it does NOT occur.
Not sure if this equates to a fix not being back-ported, or a fundamental
difference in functionality. Sadly, we're not able to yet move to 3.1.x,
because of running inside the Karaf container, and our dependencies on Camel
and Spring 3.x.
Thanks. Please let me know if you need any other information.
was:
Our platform allows the user to switch transports for a particular proxy based
on configuration. We currently allow HTTP and JMS as possible options.
During recent performance testing, we've hit a bug which causes JAXRS Clients
over JMS to fail with a ConcurrentModificationException:
A test case is attached that reproduces our problem.
Our test executes requests across multiple concurrent threads, switching
between JMS and HTTP for both JAXWS and JAXRS Client Proxies. In the test case:
- JAXWS over both HTTP and JMS works fine, both in single and multi-threaded
tests.
- JAXRS works fine across single and multi-threaded tests over HTTP with the
'ThreadSafe' flag set on the Client Factory.
- JAXRS over JMS fails. Two problems occur:
1) The ConnectionFactoryFeature fails to properly set the JMS Connection
Factory on the Conduit and results in an error at send-time because no
Connection Factory is set. This is because in JAXRS Clients, the
InterceptorProvider is neither a Client or Server, it is a ClientConfiguration,
so the JMS ConnectionFactoryFeature does not fire either of it's methods:
public void initialize(Client client, Bus bus)
public void initialize(Server server, Bus bus)
We have worked around this in our code by sub-classing and overriding the
method:
public void initialize(InterceptorProvider interceptorProvider, Bus bus)
.. and adding code similar to that in the ConnectionFactory feature. This
resolves the missing ConnectionFactory problem.
2) Under multi-threaded conditions, the JAXRS Clients fail with the following
exception:
javax.ws.rs.client.ResponseProcessingException: Problem with reading the data,
class java.lang.String, ContentType: text/plain.
at
org.apache.cxf.jaxrs.impl.ResponseImpl.reportMessageHandlerProblem(ResponseImpl.java:438)
at
org.apache.cxf.jaxrs.impl.ResponseImpl.doReadEntity(ResponseImpl.java:378)
at
org.apache.cxf.jaxrs.client.AbstractClient.readBody(AbstractClient.java:521)
at
org.apache.cxf.jaxrs.client.ClientProxyImpl.handleResponse(ClientProxyImpl.java:817)
at
org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:760)
at
org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:228)
at com.sun.proxy.$Proxy24.doSimpleMethod(Unknown Source)
at
com.ge.testcase.ThreadingTest$ThreadedRSJMSTest.run(ThreadingTest.java:132)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429)
at java.util.HashMap$EntryIterator.next(HashMap.java:1463)
at java.util.HashMap$EntryIterator.next(HashMap.java:1461)
at java.util.HashMap.putMapEntries(HashMap.java:511)
at java.util.HashMap.putAll(HashMap.java:784)
at org.apache.cxf.message.MessageImpl$1.putAll(MessageImpl.java:188)
at
org.apache.cxf.message.MessageImpl.calcContextCache(MessageImpl.java:213)
at
org.apache.cxf.message.MessageImpl.getContextualProperty(MessageImpl.java:174)
at
org.apache.cxf.jaxrs.impl.HttpHeadersImpl.getRequestHeaders(HttpHeadersImpl.java:172)
at
org.apache.cxf.jaxrs.utils.JAXRSUtils.readFromMessageBodyReader(JAXRSUtils.java:1345)
at
org.apache.cxf.jaxrs.impl.ResponseImpl.doReadEntity(ResponseImpl.java:369)
... 7 more
This occurs even though the 'ThreadSafe' property is set via:
rsClientFactory.setThreadSafe(true);
I've done quite a large amount of digging through the code to try and narrow
down the scope of the problem, and the results above is as far as I could get,
unfortunately. My guess/assumption is that there are some JMS-specific objects
that are being added to the Request/Response context which are being
added/removed in a non-thread safe way. Purely a guess, though.
Finally, one important factor is that this error only happens on the latest
3.0.x release (3.0.11). When I switch my test case to 3.1.8, it does NOT occur.
Not sure if this equates to a fix not being back-ported, or a fundamental
difference in functionality. Sadly, we're not able to yet move to 3.1.x,
because of running inside the Karaf container, and our dependencies on Camel
and Spring 3.x.
Thanks. Please let me know if you need any other information.
> ConcurrentModificationException in MessageImpl.calcContextCache() when using
> JMS Transport and JAXRS Client
> -----------------------------------------------------------------------------------------------------------
>
> Key: CXF-7135
> URL: https://issues.apache.org/jira/browse/CXF-7135
> Project: CXF
> Issue Type: Bug
> Components: JAX-RS, JMS
> Affects Versions: 3.0.11
> Environment: Mac OSX 10.11.6.
> Java JDK 1.8.0_71 (64 bit Server)
> Reporter: Dan Salt
> Attachments: cxf-rsjms-testcase.zip
>
>
> Our platform allows the user to switch transports for a particular proxy
> based on configuration. We currently allow HTTP and JMS as possible options.
> During recent performance testing, we've hit a bug which causes JAXRS Clients
> over JMS to fail with a ConcurrentModificationException:
> A test case is attached that reproduces our problem.
> Our test executes requests across multiple concurrent threads, switching
> between JMS and HTTP for both JAXWS and JAXRS Client Proxies. In the test
> case:
> - JAXWS over both HTTP and JMS works fine, both in single and multi-threaded
> tests.
> - JAXRS works fine across single and multi-threaded tests over HTTP with the
> 'ThreadSafe' flag set on the Client Factory.
> - JAXRS over JMS fails in both single-threaded and multi-threaded mode with
> error (1) below. In multi-threaded, also fails with (2) below.
> 1) The ConnectionFactoryFeature fails to properly set the JMS Connection
> Factory on the Conduit and results in an error at send-time because no
> Connection Factory is set. This is because in JAXRS Clients, the
> InterceptorProvider is neither a Client or Server, it is a
> ClientConfiguration, so the JMS ConnectionFactoryFeature does not fire either
> of it's methods:
> public void initialize(Client client, Bus bus)
> public void initialize(Server server, Bus bus)
> We have worked around this in our code by sub-classing and overriding the
> method:
> public void initialize(InterceptorProvider interceptorProvider, Bus bus)
> .. and adding code similar to that in the ConnectionFactory feature. This
> resolves the missing ConnectionFactory problem and JAXRS-over-JMS works fine
> in single-threaded mode.
> 2) Under multi-threaded conditions, the JAXRS Clients fail with the following
> exception:
> javax.ws.rs.client.ResponseProcessingException: Problem with reading the
> data, class java.lang.String, ContentType: text/plain.
> at
> org.apache.cxf.jaxrs.impl.ResponseImpl.reportMessageHandlerProblem(ResponseImpl.java:438)
> at
> org.apache.cxf.jaxrs.impl.ResponseImpl.doReadEntity(ResponseImpl.java:378)
> at
> org.apache.cxf.jaxrs.client.AbstractClient.readBody(AbstractClient.java:521)
> at
> org.apache.cxf.jaxrs.client.ClientProxyImpl.handleResponse(ClientProxyImpl.java:817)
> at
> org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:760)
> at
> org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:228)
> at com.sun.proxy.$Proxy24.doSimpleMethod(Unknown Source)
> at
> com.ge.testcase.ThreadingTest$ThreadedRSJMSTest.run(ThreadingTest.java:132)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429)
> at java.util.HashMap$EntryIterator.next(HashMap.java:1463)
> at java.util.HashMap$EntryIterator.next(HashMap.java:1461)
> at java.util.HashMap.putMapEntries(HashMap.java:511)
> at java.util.HashMap.putAll(HashMap.java:784)
> at org.apache.cxf.message.MessageImpl$1.putAll(MessageImpl.java:188)
> at
> org.apache.cxf.message.MessageImpl.calcContextCache(MessageImpl.java:213)
> at
> org.apache.cxf.message.MessageImpl.getContextualProperty(MessageImpl.java:174)
> at
> org.apache.cxf.jaxrs.impl.HttpHeadersImpl.getRequestHeaders(HttpHeadersImpl.java:172)
> at
> org.apache.cxf.jaxrs.utils.JAXRSUtils.readFromMessageBodyReader(JAXRSUtils.java:1345)
> at
> org.apache.cxf.jaxrs.impl.ResponseImpl.doReadEntity(ResponseImpl.java:369)
> ... 7 more
> This occurs even though the 'ThreadSafe' property is set via:
> rsClientFactory.setThreadSafe(true);
> I've done quite a large amount of digging through the code to try and narrow
> down the scope of the problem, and the results above is as far as I could
> get, unfortunately. My guess/assumption is that there are some JMS-specific
> objects that are being added to the Request/Response context which are being
> added/removed in a non-thread safe way. Purely a guess, though.
> Finally, one important factor is that this error only happens on the latest
> 3.0.x release (3.0.11). When I switch my test case to 3.1.8, it does NOT
> occur. Not sure if this equates to a fix not being back-ported, or a
> fundamental difference in functionality. Sadly, we're not able to yet move to
> 3.1.x, because of running inside the Karaf container, and our dependencies on
> Camel and Spring 3.x.
> Thanks. Please let me know if you need any other information.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
