Process PID Protocol local address local port Remote
Address State
Tomcat8.exe 8160 TCP imal14-app2 4000
imal14-app1.albaraka.com.sd ESTABLISHED
On Tue, Feb 7, 2017 at 5:46 PM, Fady Haikal <fadyhai...@gmail.com> wrote:
> Yes there is a ESTABLISHED connection, the replication of sessions is
> working fine (port 4000 is for tomcat cluster) but we also faced this
> error on the log file
>
> On Tue, Feb 7, 2017 at 5:44 PM, André Warnier (tomcat) <a...@ice-sa.com>
> wrote:
>> On 07.02.2017 16:24, Fady Haikal wrote:
>>>
>>> Hi,
>>> telnet IP 4000 is working fine, i installed a tool for network
>>> monitoring at the level of IP and Port and i didnt see any
>>> disconnection,
>>
>>
>> but did you see a *connection* ?
>> I mean, on the pinging node, if you use the Windows "netstat" program, for
>> example as
>> netstat -aon -p TCP
>> you should see a list of connections in the ESTABLISHED state, of which one
>> of the IP/ports should be your target IP:4000 (in the "remote" column).
>> And on the pinged node, this port :4000 should be in the "local" column, in
>> LISTEN mode
>> (and also probably one in the ESTABLISHED state, if they agree.)
>>
>> Is that the case ?
>>
>>
>>
>> and yes i'm sure that no firewall is enabled.
>>>
>>>
>>> I saw some strange think on the server that I have tried to ping the
>>> multicast IP (228.0.0.4) and i get reply from different IPs in the
>>> network, i don't know why and how i get those IPs, after checking with
>>> the network team they told me that those IPs are related to the SAN
>>> storage taking into consideration that the Tomcat servers are not
>>> connected in anyway to that SUN storage.
>>>
>>>
>>> On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat) <a...@ice-sa.com>
>>> wrote:
>>>>
>>>> Hi.
>>>>
>>>> This is for the Tomcat/Tribes experts on the list.
>>>>
>>>> I know nothing of Tribes, but the on-line documentation seems to say that
>>>> the communication happens over TCP and that the protocol used is not
>>>> encrypted.
>>>> Fady previously tried a standard "ping" and a "telnet" between the two
>>>> nodes, and that is the base for him mentioning that "there is no network
>>>> disconnection" between the nodes.
>>>> Nevertheless, the calling pinging node seems to say that it times out
>>>> without getting a response fom the target node. There is evidently a
>>>> contradiction there.
>>>> So this could still be some kind of network issue.
>>>>
>>>> Considering that the protocol command for this "ping" should be known by
>>>> someone here, would it not be possible to imagine a little program in
>>>> some
>>>> scripting language (or even java, God forbid), which would open a TCP
>>>> channel with the target node IP/port, send such a "ping" message, wait
>>>> for a
>>>> response and report the result ?
>>>> That would at least confirm/deny that the problem is with the network.
>>>>
>>>> The log below does not for example say if the error happens when opening
>>>> the
>>>> TCP communication channel, or after sending the ping message on it,
>>>> (Of course, testing the TCP open could be done with "telnet IP 4000", but
>>>> I
>>>> don't know if Fady tried this).
>>>> Maybe tribes also already contains some löw-level debugging options ?
>>>> wireshark maybe another option, but it has quite a learning curve.
>>>> And this is on Windows.
>>>>
>>>> By the way Fady, are you sure that your "Windows Firewall with Enhanced
>>>> Security" is not just dropping TCP packets to/from port 40xx (or from
>>>> "java.exe") ? There are some "network policies" there which can have
>>>> wide-ranging side-effects.
>>>>
>>>>
>>>>
>>>>
>>>> On 07.02.2017 14:42, Fady Haikal wrote:
>>>>>
>>>>>
>>>>> Hi, issue still not fixed. Tomcat session replication is not able to
>>>>> replicate the key from node to node, please find below the error,
>>>>> taking into consideration that there is no network disconnection
>>>>> between 2 nodes
>>>>>
>>>>>
>>>>> 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8]
>>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo
>>>>> Unable to replicate backup
>>>>> key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to
>>>>> backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>>>> 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350,
>>>>> securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64 12 -102 -14
>>>>> -85 -87 15 9 -51 -112 }, payload={}, command={}, domain={}, ].
>>>>> Reason:Operation has timed out(3000 ms.).; Faulty members:tcp://{10,
>>>>> 114, 43, 102}:4000;
>>>>> org.apache.catalina.tribes.ChannelException: Operation has timed
>>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 102}:4000;
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo(LazyReplicatedMap.java:170)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1040)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1024)
>>>>> at org.apache.catalina.session.ManagerBase.add(ManagerBase.java:647)
>>>>> at
>>>>>
>>>>> org.apache.catalina.session.StandardSession.setId(StandardSession.java:374)
>>>>> at
>>>>> org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.java:279)
>>>>> at
>>>>>
>>>>> org.apache.catalina.session.ManagerBase.createSession(ManagerBase.java:708)
>>>>> at org.apache.catalina.connector.Request.doGetSession(Request.java:2936)
>>>>> at org.apache.catalina.connector.Request.getSession(Request.java:2260)
>>>>> at
>>>>>
>>>>> org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:895)
>>>>> at
>>>>>
>>>>> javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:231)
>>>>> at
>>>>>
>>>>> org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:568)
>>>>> at
>>>>>
>>>>> org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:513)
>>>>> at
>>>>>
>>>>> org.apache.jasper.runtime.PageContextImpl.initialize(PageContextImpl.java:137)
>>>>> at
>>>>>
>>>>> org.apache.jasper.runtime.JspFactoryImpl.internalGetPageContext(JspFactoryImpl.java:109)
>>>>> at
>>>>>
>>>>> org.apache.jasper.runtime.JspFactoryImpl.getPageContext(JspFactoryImpl.java:60)
>>>>> at
>>>>> org.apache.jsp.WEB_002dINF.jsp._401_jsp._jspService(_401_jsp.java:100)
>>>>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>>>> at
>>>>>
>>>>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:438)
>>>>> at
>>>>> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:396)
>>>>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340)
>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>>>> at
>>>>>
>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
>>>>> at
>>>>>
>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>>> at
>>>>>
>>>>> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:719)
>>>>> at
>>>>>
>>>>> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:467)
>>>>> at
>>>>>
>>>>> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:390)
>>>>> at
>>>>>
>>>>> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:317)
>>>>> at
>>>>>
>>>>> org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:445)
>>>>> at
>>>>>
>>>>> org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:304)
>>>>> at
>>>>>
>>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:181)
>>>>> at
>>>>>
>>>>> org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:194)
>>>>> at
>>>>>
>>>>> org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:318)
>>>>> at
>>>>>
>>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
>>>>> at
>>>>>
>>>>> org.apache.catalina.valves.StuckThreadDetectionValve.invoke(StuckThreadDetectionValve.java:208)
>>>>> at
>>>>>
>>>>> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
>>>>> at
>>>>>
>>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
>>>>> at
>>>>>
>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521)
>>>>> at
>>>>>
>>>>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096)
>>>>> at
>>>>>
>>>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674)
>>>>> at
>>>>>
>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
>>>>> at
>>>>>
>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)
>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>>>>> at
>>>>>
>>>>> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>>>>> at java.lang.Thread.run(Unknown Source)
>>>>>
>>>>> On Tue, Feb 7, 2017 at 1:02 PM, Kreuser, Peter <pkreu...@airplus.com>
>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> Fady,
>>>>>>
>>>>>> Sorry for top posting.
>>>>>>
>>>>>> If I remember correctly, the Cluster Element goes into the Container
>>>>>> and
>>>>>> not the Host.
>>>>>> Plus I see in our (working) case, a DeltaManager and a
>>>>>> JvmRouteSessionIDBinderListener
>>>>>>
>>>>>> <Manager
>>>>>> className="org.apache.catalina.ha.session.DeltaManager"
>>>>>> expireSessionsOnShutdown="false"
>>>>>> notifyListenersOnReplication="true"/>
>>>>>> ...
>>>>>> <ClusterListener
>>>>>>
>>>>>> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
>>>>>>
>>>>>> Besides this, only ports, limits and values are different.
>>>>>>
>>>>>> You may want to filter out the replication for static resources as gifs
>>>>>> jpg or css.
>>>>>> <Valve
>>>>>> className="org.apache.catalina.ha.tcp.ReplicationValve"
>>>>>>
>>>>>> filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;"
>>>>>> />
>>>>>>
>>>>>> Best regards
>>>>>>
>>>>>> Peter
>>>>>>
>>>>>>> below is the server.xml configuration, as mentioened earlier the issue
>>>>>>> is related to the cluster configuration, and as per my research i can
>>>>>>> see that some users are facing the same issue but i didnt found the
>>>>>>> solution of it
>>>>>>>
>>>>>>>
>>>>>>> <?xml version='1.0' encoding='utf-8'?>
>>>>>>> <!--
>>>>>>> Licensed to the Apache Software Foundation (ASF) under one or more
>>>>>>> contributor license agreements. See the NOTICE file distributed
>>>>>>> with
>>>>>>> this work for additional information regarding copyright
>>>>>>> ownership.
>>>>>>> The ASF licenses this file to You under the Apache License,
>>>>>>> Version
>>>>>>> 2.0
>>>>>>> (the "License"); you may not use this file except in compliance
>>>>>>> with
>>>>>>> the License. You may obtain a copy of the License at
>>>>>>>
>>>>>>> http://www.apache.org/licenses/LICENSE-2.0
>>>>>>>
>>>>>>> Unless required by applicable law or agreed to in writing,
>>>>>>> software
>>>>>>> distributed under the License is distributed on an "AS IS" BASIS,
>>>>>>> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>>>>>>> implied.
>>>>>>> See the License for the specific language governing permissions
>>>>>>> and
>>>>>>> limitations under the License.
>>>>>>> -->
>>>>>>> <!-- Note: A "Server" is not itself a "Container", so you may not
>>>>>>> define subcomponents such as "Valves" at this level.
>>>>>>> Documentation at /docs/config/server.html
>>>>>>> -->
>>>>>>> <Server port="8005" shutdown="SHUTDOWN">
>>>>>>> <Listener
>>>>>>> className="org.apache.catalina.startup.VersionLoggerListener" />
>>>>>>> <!-- Security listener. Documentation at
>>>>>>> /docs/config/listeners.html
>>>>>>> <Listener
>>>>>>> className="org.apache.catalina.security.SecurityListener"
>>>>>>> />
>>>>>>> -->
>>>>>>> <!--APR library loader. Documentation at /docs/apr.html -->
>>>>>>> <Listener
>>>>>>> className="org.apache.catalina.core.AprLifecycleListener"
>>>>>>> SSLEngine="on" />
>>>>>>> <!-- Prevent memory leaks due to use of particular java/javax
>>>>>>> APIs-->
>>>>>>> <Listener
>>>>>>> className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
>>>>>>> />
>>>>>>> <Listener
>>>>>>>
>>>>>>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
>>>>>>> />
>>>>>>> <Listener
>>>>>>> className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"
>>>>>>> />
>>>>>>>
>>>>>>> <!-- Global JNDI resources
>>>>>>> Documentation at /docs/jndi-resources-howto.html
>>>>>>> -->
>>>>>>> <GlobalNamingResources>
>>>>>>> <!-- Editable user database that can also be used by
>>>>>>> UserDatabaseRealm to authenticate users
>>>>>>> -->
>>>>>>> <Resource name="UserDatabase" auth="Container"
>>>>>>> type="org.apache.catalina.UserDatabase"
>>>>>>> description="User database that can be updated and
>>>>>>> saved"
>>>>>>>
>>>>>>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>>>>>>> pathname="conf/tomcat-users.xml" />
>>>>>>> </GlobalNamingResources>
>>>>>>>
>>>>>>> <!-- A "Service" is a collection of one or more "Connectors" that
>>>>>>> share
>>>>>>> a single "Container" Note: A "Service" is not itself a
>>>>>>> "Container",
>>>>>>> so you may not define subcomponents such as "Valves" at this
>>>>>>> level.
>>>>>>> Documentation at /docs/config/service.html
>>>>>>> -->
>>>>>>> <Service name="Catalina">
>>>>>>>
>>>>>>> <!--The connectors can use a shared executor, you can define one
>>>>>>> or more named thread pools-->
>>>>>>> <!--
>>>>>>> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
>>>>>>> maxThreads="150" minSpareThreads="4"/>
>>>>>>> -->
>>>>>>>
>>>>>>>
>>>>>>> <!-- A "Connector" represents an endpoint by which requests are
>>>>>>> received
>>>>>>> and responses are returned. Documentation at :
>>>>>>> Java HTTP Connector: /docs/config/http.html (blocking &
>>>>>>> non-blocking)
>>>>>>> Java AJP Connector: /docs/config/ajp.html
>>>>>>> APR (HTTP/AJP) Connector: /docs/apr.html
>>>>>>> Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
>>>>>>> -->
>>>>>>> <Connector port="8080" protocol="HTTP/1.1"
>>>>>>> connectionTimeout="60000" maxThreads="500"
>>>>>>> minSpareThreads="25" maxSpareThreads="75" enableLookups="false"
>>>>>>> disableUploadTimeout="true" acceptCount="100" redirectPort="8443" />
>>>>>>> <!-- A "Connector" using the shared thread pool-->
>>>>>>> <!--
>>>>>>> <Connector executor="tomcatThreadPool"
>>>>>>> port="8080" protocol="HTTP/1.1"
>>>>>>> connectionTimeout="60000"
>>>>>>> redirectPort="8443" />
>>>>>>> -->
>>>>>>> <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
>>>>>>> This connector uses the NIO implementation that requires
>>>>>>> the
>>>>>>> JSSE
>>>>>>> style configuration. When using the APR/native
>>>>>>> implementation,
>>>>>>> the
>>>>>>> OpenSSL style configuration is required as described in the
>>>>>>> APR/native
>>>>>>> documentation -->
>>>>>>> <!--
>>>>>>> <Connector port="8443"
>>>>>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>>>>>>> maxThreads="150" SSLEnabled="true" scheme="https"
>>>>>>> secure="true"
>>>>>>> clientAuth="false" sslProtocol="TLS" />
>>>>>>> -->
>>>>>>>
>>>>>>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>>>>>>> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
>>>>>>>
>>>>>>>
>>>>>>> <!-- An Engine represents the entry point (within Catalina) that
>>>>>>> processes
>>>>>>> every request. The Engine implementation for Tomcat stand
>>>>>>> alone
>>>>>>> analyzes the HTTP headers included with the request, and
>>>>>>> passes them
>>>>>>> on to the appropriate Host (virtual host).
>>>>>>> Documentation at /docs/config/engine.html -->
>>>>>>>
>>>>>>> <!-- You should set jvmRoute to support load-balancing via AJP
>>>>>>> ie :
>>>>>>>
>>>>>>> -->
>>>>>>> <Engine name="Catalina" defaultHost="localhost"
>>>>>>> jvmRoute="TomcatNode1">
>>>>>>> <!-- Use the LockOutRealm to prevent attempts to guess user
>>>>>>> passwords
>>>>>>> via a brute-force attack -->
>>>>>>> <Realm className="org.apache.catalina.realm.LockOutRealm">
>>>>>>> <!-- This Realm uses the UserDatabase configured in the
>>>>>>> global
>>>>>>> JNDI
>>>>>>> resources under the key "UserDatabase". Any edits
>>>>>>> that are performed against this UserDatabase are
>>>>>>> immediately
>>>>>>> available for use by the Realm. -->
>>>>>>> <Realm
>>>>>>> className="org.apache.catalina.realm.UserDatabaseRealm"
>>>>>>> resourceName="UserDatabase"/>
>>>>>>> </Realm>
>>>>>>>
>>>>>>> <Host name="localhost" appBase="webapps"
>>>>>>> unpackWARs="true" autoDeploy="true"
>>>>>>> startStopThreads="0">
>>>>>>> <!--For clustering, please take a look at documentation at:
>>>>>>> /docs/cluster-howto.html (simple how to)
>>>>>>> /docs/config/cluster.html (reference documentation) -->
>>>>>>> <!--
>>>>>>> <Cluster
>>>>>>> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
>>>>>>> -->
>>>>>>>
>>>>>>> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>>>>>>> channelSendOptions="4">
>>>>>>> <Channel
>>>>>>> className="org.apache.catalina.tribes.group.GroupChannel">
>>>>>>> <Membership
>>>>>>> className="org.apache.catalina.tribes.membership.McastService"
>>>>>>> address="228.0.0.4"
>>>>>>> port="45564"
>>>>>>> frequency="500"
>>>>>>> dropTime="9000"/>
>>>>>>> <Receiver
>>>>>>> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>>>>>>> address="auto"
>>>>>>> port="4000"
>>>>>>> autoBind="100"
>>>>>>> selectorTimeout="5000"
>>>>>>> maxThreads="6"/>
>>>>>>>
>>>>>>> <Sender
>>>>>>>
>>>>>>> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
>>>>>>> <Transport
>>>>>>>
>>>>>>>
>>>>>>> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
>>>>>>> </Sender>
>>>>>>> <Interceptor
>>>>>>>
>>>>>>>
>>>>>>> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
>>>>>>> <Interceptor
>>>>>>>
>>>>>>>
>>>>>>> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
>>>>>>> </Channel>
>>>>>>>
>>>>>>> <Valve
>>>>>>> className="org.apache.catalina.ha.tcp.ReplicationValve"
>>>>>>> filter=""/>
>>>>>>> <Valve
>>>>>>> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>>>>>>>
>>>>>>> <Deployer
>>>>>>> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
>>>>>>> tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
>>>>>>>
>>>>>>> deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/"
>>>>>>>
>>>>>>> watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/"
>>>>>>> watchEnabled="false"/>
>>>>>>>
>>>>>>> <ClusterListener
>>>>>>> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>>>>>>> </Cluster>
>>>>>>> <!-- SingleSignOn valve, share authentication between web
>>>>>>> applications
>>>>>>> Documentation at: /docs/config/valve.html -->
>>>>>>> <!--
>>>>>>> <Valve
>>>>>>> className="org.apache.catalina.authenticator.SingleSignOn" />
>>>>>>> -->
>>>>>>>
>>>>>>> <!-- Access log processes all example.
>>>>>>> Documentation at: /docs/config/valve.html
>>>>>>> Note: The pattern used is equivalent to using
>>>>>>> pattern="common" -->
>>>>>>> <Valve className="org.apache.catalina.valves.AccessLogValve"
>>>>>>> directory="logs"
>>>>>>> prefix="localhost_access_log" suffix=".txt"
>>>>>>> pattern="%h %l %u %t "%r" %s %b" />
>>>>>>> <Valve
>>>>>>> className="org.apache.catalina.valves.StuckThreadDetectionValve"
>>>>>>> threshold="900" />
>>>>>>>
>>>>>>> </Host>
>>>>>>> </Engine>
>>>>>>> </Service>
>>>>>>> </Server>
>>>>>>>
>>>>>>> On Mon, Feb 6, 2017 at 6:51 PM, André Warnier (tomcat) <a...@ice-sa.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 06.02.2017 17:45, Fady Haikal wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> What is the host OS ? Windows Server 2012
>>>>>>>>> What is the Tomcat version ? Apache Tomcat/8.0.30
>>>>>>>>>
>>>>>>>>> Is this problem new ? was this working before ? how long ? Since
>>>>>>>>> cluster implementation
>>>>>>>>>
>>>>>>>>
>>>>>>>> I still don't know tribes, but then my non-educated guess at this
>>>>>>>> point
>>>>>>>> would be that there is something wrong in your configuration.
>>>>>>>> Can you copy/paste it here ? (remove sensible things like passwords,
>>>>>>>> public
>>>>>>>> IP addresses etc..)(but not to the point of making it uncheckable).
>>>>>>>>
>>>>>>>> Then maybe some tribes-specialist can take over ?
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Is there actually something listening on that address/port ? Tomcat
>>>>>>>>> cluster
>>>>>>>>>
>>>>>>>>> the Port 4000 is listening and there is no disconnection between 2
>>>>>>>>> nodes ping and telnet are OK
>>>>>>>>>
>>>>>>>>> On Mon, Feb 6, 2017 at 6:42 PM, André Warnier (tomcat)
>>>>>>>>> <a...@ice-sa.com>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 06.02.2017 17:24, Fady Haikal wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Plz can i get some help here?
>>>>>>>>>>> This issue is still occurring and it's filling the log file in the
>>>>>>>>>>> Production server
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Fady
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi.
>>>>>>>>>> If you want quick answers, you should provide more information.
>>>>>>>>>> What is the host OS ?
>>>>>>>>>> What is the Tomcat version ?
>>>>>>>>>> Is this problem new ? was this working before ? how long ?
>>>>>>>>>>
>>>>>>>>>> I do not know tribes at all, but according to the logfile below, it
>>>>>>>>>> seems
>>>>>>>>>> that something is trying to "ping" the address 10.114.43.103, port
>>>>>>>>>> 4000,
>>>>>>>>>> and
>>>>>>>>>> never getting a response (or at least not within 3000ms).
>>>>>>>>>> Is there actually something listening on that address/port ?
>>>>>>>>>> The "netstat" command (available both on Linux and Windows) can
>>>>>>>>>> tell
>>>>>>>>>> you.
>>>>>>>>>> If there is something listening there, can it respond to whatever
>>>>>>>>>> is
>>>>>>>>>> ping-ing it ?
>>>>>>>>>> (routing, firewall, ..)
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikal <fadyhai...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Guys, we are facing the below errors in Tomcat cluster, please
>>>>>>>>>>>> advise
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
>>>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>>>>>>>>>>>> Unable to send AbstractReplicatedMap.ping message
>>>>>>>>>>>> org.apache.catalina.tribes.ChannelException: Operation has
>>>>>>>>>>>> timed
>>>>>>>>>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:267)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:161)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
>>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen
>>>>>>>>>>>> Unable
>>>>>>>>>>>> to
>>>>>>>>>>>> process request in NioReceiver
>>>>>>>>>>>> java.io.IOException: A non-blocking socket operation could
>>>>>>>>>>>> not
>>>>>>>>>>>> be
>>>>>>>>>>>> completed immediately
>>>>>>>>>>>> at sun.nio.ch.SocketDispatcher.close0(Native Method)
>>>>>>>>>>>> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
>>>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
>>>>>>>>>>>> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
>>>>>>>>>>>> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(Unknown Source)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:272)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:425)
>>>>>>>>>>>> at java.lang.Thread.run(Unknown Source)
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Fady
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>>>>>>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>>>>>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>>>>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>>>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>>>>>
>>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
