[jira] [Commented] (CXF-6674) Infinite loop when handling SOAP responses in split/parallel Camel route
[
https://issues.apache.org/jira/browse/CXF-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15188914#comment-15188914
]
Marc Layer commented on CXF-6674:
-
After disabling the parallel processing feature, the problem has not recurred
so far in our production environment. (There was, however, also a change in the
Dynatrace instrumentation which might have brought in additional
synchronizations that may have helped to suppress the problem for now.)
Still, the busy loop in WrappedMessageContext.copyScoped() to me seems to be an
indicator that there is a slumbering thread synchronization issue somewhere in
the context handling code. Does this ring a bell with somebody? Any help would
be appreciated!
> Infinite loop when handling SOAP responses in split/parallel Camel route
>
>
> Key: CXF-6674
> URL: https://issues.apache.org/jira/browse/CXF-6674
> Project: CXF
> Issue Type: Bug
> Components: JAX-WS Runtime
>Affects Versions: 2.7.10
> Environment: Apache Camel
> 2.12.3
> Apache CXF
> 2.7.10
> App Server
> WebSphere Platform 8.0.0.10 [ND 8.0.0.10 cf101502.03]
> Java Runtime Version
> pxa6460_26sr8fp2ifx-20141114_01 (SR8 FP2)
> Java Vendor
> IBM Corporation
> Java Version
> 1.6.0
> Java Compiler
> j9jit26
> Java VM name
> IBM J9 VM
> Operating System
> Linux
> OS Architecture
> x86_64
> OS Version
> 2.6.32-279.el6.x86_64
> VM Information
> JRE 1.6.0 Linux amd64-64 Compressed References 20141114_222464 (JIT enabled,
> AOT enabled)
> J9VM - R26_Java626_SR8_20141114_1023_B222464
> JIT - r11.b07_20141003_74578.01
> GC - R26_Java626_SR8_20141114_1023_B222464_CMPRSS
> J9CL - 20141114_222464
> VM Vendor
> IBM Corporation
> VM Version
> 2.6
>Reporter: Marc Layer
> Attachments: BackendServiceRoute.java, dynatrace.png
>
>
> We are using Apache Camel 2.12.3 and Apache CXF 2.7.10 to integrate some
> backend SOAP web services. In our production environment we are facing the
> issue that at irregular intervals a CXF worker thread would hang in a busy
> loop, consuming 100% CPU time. The only recovery so far is to restart the
> server.
> The loop seems to occur in
> org.apache.cxf.jaxws.context.WrappedMessageContext.copyScoped():
> {noformat}
> private void copyScoped(Map msg) {
> for (String s : scopes.keySet()) {
> message.put(s, msg.get(s));
> }
> }
> {noformat}
> I guess it might be related to a threading issue that causes the iterator of
> scopes.keySet() to loop infinitely. Using the VM monitoring tool Dynatrace, I
> can see that the iterated "scope" map is an unsynchronized HashMap. I am
> unsure where the "scope" map was created and how it could have leaked into
> another thread. "message" seems to be an instance of MessageImpl, which is
> also derived from HashMap and therefore unsynchronized. I attached a
> Dynatrace screenshot from one such invocation.
> Multi-threading in our case is two-fold:
> - CXF seems to be using separate worker threads even for synchronous
> invocations.
> - Additionally the call in question makes use of Camel's splitter EIP with
> parallel processing which also uses separate threads (RouteBuilder is
> attached), effectively like this:
> {noformat}
> .split(body(), dataAggregator)
> .stopOnException()
>
> .parallelProcessing().executorServiceRef("multicastExecutorService")
> .to("cxf:bean:backendService")
> {noformat}
> I analyzed a number of cases in which two web service calls were dispatched
> at the same time. Dynatrace shows that the backend web services always
> finished within about 80-130ms.
> Unfortunately we were unable to reproduce the error on a testing system where
> we could have attached a remote debugger to the application, not even under
> heavy load. It only occured in the production environment so far. Therefore I
> am uncertain if the problem only occurs when Camel's parallelProcessing() is
> used. We will disable this feature in the application's next release (end of
> next week) to verify this.
> There was another issue, CXF-6160 "AsyncResponse infinite loops", in which an
> infinite loop in MessageImpl was tackled by making ExchangeImpl a subclass of
> ConcurrentHashMap rather than a subclass of StringImpl (which is also derived
> from HashMap). Maybe these issues are related. The fix, however, was
> implemented only for CXF 3.x. Unfortunately we are bound to using Java 1.6 as
> Websphere 8 is only available for IBM JRE 1.6. The latest Camel version
> available for Java 1.6 comes with CXF 2.7.14, later Camel versions are only
> available for Java 1.7. I think we could easily upgrade to a newer 2.7.x
> version of CXF but not to CXF 3.x.
--
This message was sent by Atlass
[jira] [Created] (CXF-6827) OAuthRequestFilter should be able to cache token validation results
Sergey Beryozkin created CXF-6827: - Summary: OAuthRequestFilter should be able to cache token validation results Key: CXF-6827 URL: https://issues.apache.org/jira/browse/CXF-6827 Project: CXF Issue Type: Improvement Components: JAX-RS Security Reporter: Sergey Beryozkin Assignee: Sergey Beryozkin Fix For: 3.2.0, 3.1.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CXF-6826) Cache MediaTypeHeaderProvider.valueOf(String mType) results
[ https://issues.apache.org/jira/browse/CXF-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15188998#comment-15188998 ] Alessio Soldano commented on CXF-6826: -- Thanks for the feedback Sergey. Two comments: 1) the idea of having a non-static map could be good, but the MediaTypeHeaderProvider is statically accessed from the JAXRSUtils (for instance I've seen it being heavily invoked from ProviderFactory.matchesWriterCriterias(...) -> JAXRSUtils.getProviderProduceTypes(MessageBodyWriter provider), it doesn't seem to me the RuntimeDelegateImpl is involved in that case 2) wiping out the concurrent map after it reaches a given size should be easy (but unfair, I'd say), evicting stuff from a deque or use a circular buffer would have the drawback of slowing the get, right? > Cache MediaTypeHeaderProvider.valueOf(String mType) results > --- > > Key: CXF-6826 > URL: https://issues.apache.org/jira/browse/CXF-6826 > Project: CXF > Issue Type: Improvement > Components: JAX-RS >Reporter: Alessio Soldano > > MediaTypeHeaderProvider.valueOf(String mType) ends up being called a lot with > the same parameter values, as those are basically the contents of the > @Produce annotations on resources. Each request ends up into the same > invocation, which is cpu and memory intensive. > It might make sense to add a map in the MediaTypeHeaderProvider to store the > result of the static valueOf method invocation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CXF-6825) Performance bottle neck due to synchronize block on each read
[ https://issues.apache.org/jira/browse/CXF-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15189007#comment-15189007 ] Alessio Soldano commented on CXF-6825: -- Well, to me those tuning could be something addressed as a later improvement to the change here. We could start with sensible values, perhaps similar to the default of the JDK ConcurrentHashMap and then add options for letting the user tune them. > Performance bottle neck due to synchronize block on each read > - > > Key: CXF-6825 > URL: https://issues.apache.org/jira/browse/CXF-6825 > Project: CXF > Issue Type: Bug > Components: Bus, JAX-RS >Affects Versions: 3.1.0, 3.1.1, 3.1.2, 3.1.3, 3.1.4, 3.1.5, 3.2.0, 3.1.6 > Environment: All environments >Reporter: Yotam Madem > Fix For: 3.2.0, 3.1.6 > > Original Estimate: 24h > Remaining Estimate: 24h > > In our (IBM MobileFirst foundation) performance tests it is possible to see > that many threads are stuck at: > "LargeThreadPool-thread-1505" daemon prio=10 tid=0x7f10f020b800 > nid=0x31fc waiting for monitor entry [0x7f12bfffd000] >java.lang.Thread.State: BLOCKED (on object monitor) > at org.apache.cxf.BusFactory.getThreadBusHolder(BusFactory.java:120) > - waiting to lock <0xc6b99020> (a java.util.WeakHashMap) > at > org.apache.cxf.BusFactory.getAndSetThreadDefaultBus(BusFactory.java:202) > > (We use version 3.1.0) > I did a test fix locally and it seems to resolve our problem so I submitted 2 > pull requests: > for 3.1.6: https://github.com/apache/cxf/pull/119 > for 3.2.0: https://github.com/apache/cxf/pull/120 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CXF-6828) [OAuth] AbstractGrant set wrong value(scope) for audience
Bernhard Roider created CXF-6828: Summary: [OAuth] AbstractGrant set wrong value(scope) for audience Key: CXF-6828 URL: https://issues.apache.org/jira/browse/CXF-6828 Project: CXF Issue Type: Bug Components: JAX-RS Security Affects Versions: 3.0.8, 3.1.5, 2.7.18 Reporter: Bernhard Roider Priority: Critical in method toMap(..) the scope value is set to audience if audience is set(copy and past failure) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CXF-6826) Cache MediaTypeHeaderProvider.valueOf(String mType) results
[ https://issues.apache.org/jira/browse/CXF-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15189401#comment-15189401 ] Sergey Beryozkin commented on CXF-6826: --- Hi Alessio, Sorry, forgot it was accessed statically too. The reason was that going via MediaType.valueOf() was causing issues in containers with multiple JAX-RS stacks loaded and we had quite a few such cases :-) Well, I'd like to avoid the uncontrolled growth of this map. It is true in many simple services there will be few media types only in the whole container, but if won't restrict then the users who implement HTTP proxies with CXF or do something else with complex media types will start reporting leaks. Clearing the whole map should it reach some threshold is really a safety net :-). I guess the media type parsing is not really expensive in simple cases, few index checks, no reg expressions, two index based splits. So the map temp going empty would only get back the callers to the current level of the parsing speed which may be not too bad ? Would be interesting to get some basic speed checks, with & without the map :-) thanks > Cache MediaTypeHeaderProvider.valueOf(String mType) results > --- > > Key: CXF-6826 > URL: https://issues.apache.org/jira/browse/CXF-6826 > Project: CXF > Issue Type: Improvement > Components: JAX-RS >Reporter: Alessio Soldano > > MediaTypeHeaderProvider.valueOf(String mType) ends up being called a lot with > the same parameter values, as those are basically the contents of the > @Produce annotations on resources. Each request ends up into the same > invocation, which is cpu and memory intensive. > It might make sense to add a map in the MediaTypeHeaderProvider to store the > result of the static valueOf method invocation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CXF-6828) [OAuth] AbstractGrant set wrong value(scope) for audience
[ https://issues.apache.org/jira/browse/CXF-6828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Beryozkin resolved CXF-6828. --- Resolution: Fixed Assignee: Sergey Beryozkin Fix Version/s: 3.0.9 3.1.6 3.2.0 thanks > [OAuth] AbstractGrant set wrong value(scope) for audience > - > > Key: CXF-6828 > URL: https://issues.apache.org/jira/browse/CXF-6828 > Project: CXF > Issue Type: Bug > Components: JAX-RS Security >Affects Versions: 2.7.18, 3.1.5, 3.0.8 >Reporter: Bernhard Roider >Assignee: Sergey Beryozkin >Priority: Critical > Fix For: 3.2.0, 3.1.6, 3.0.9 > > > in method toMap(..) the scope value is set to audience if audience is > set(copy and past failure) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CXF-6826) Cache MediaTypeHeaderProvider.valueOf(String mType) results
[ https://issues.apache.org/jira/browse/CXF-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15189846#comment-15189846 ] Alessio Soldano commented on CXF-6826: -- Hi Sergey, I've updated the PR with the map max size thing and sent you 2 jfr recordings from some basic test I ran locally. Please let me know what you think. > Cache MediaTypeHeaderProvider.valueOf(String mType) results > --- > > Key: CXF-6826 > URL: https://issues.apache.org/jira/browse/CXF-6826 > Project: CXF > Issue Type: Improvement > Components: JAX-RS >Reporter: Alessio Soldano > > MediaTypeHeaderProvider.valueOf(String mType) ends up being called a lot with > the same parameter values, as those are basically the contents of the > @Produce annotations on resources. Each request ends up into the same > invocation, which is cpu and memory intensive. > It might make sense to add a map in the MediaTypeHeaderProvider to store the > result of the static valueOf method invocation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (DOSGI-229) Refactor to make Remote Service Admin core independent of CXF
[ https://issues.apache.org/jira/browse/DOSGI-229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christian Schneider resolved DOSGI-229. --- Resolution: Fixed > Refactor to make Remote Service Admin core independent of CXF > - > > Key: DOSGI-229 > URL: https://issues.apache.org/jira/browse/DOSGI-229 > Project: CXF Distributed OSGi > Issue Type: Improvement > Components: DSW >Reporter: Christian Schneider >Assignee: Christian Schneider > Fix For: 1.8.0 > > > The real implementation of the Remote Service Admin is in the package > org.apache.cxf.dosgi.dsw.service. It would make sense to be able to use this > implementation together with other providers than CXF. > So the idea is to refactor to make the classes in > org.apache.cxf.dosgi.dsw.service independent of CXF and introduce an api > package org.apache.cxf.dosgi.dsw.api to make the connection. > As a next step the api package and core impl should be moved into separate > bundles. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (DOSGI-230) Create TCP provider
[ https://issues.apache.org/jira/browse/DOSGI-230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christian Schneider resolved DOSGI-230. --- Resolution: Fixed > Create TCP provider > --- > > Key: DOSGI-230 > URL: https://issues.apache.org/jira/browse/DOSGI-230 > Project: CXF Distributed OSGi > Issue Type: New Feature >Reporter: Christian Schneider >Assignee: Christian Schneider > Fix For: 1.8.0 > > > Many people complain that the full DOSGi with CXF is too big. > So the idea is to provide a really small TCP provider that uses Java > Serialization over TCP. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
