gp456 wrote:
thanks for your reply.
first the fix you mentioned is not in the current tomcat release (6.0.16).
Since I have to build/patch the tomcat for a productive environment what
whould you recommend: patch the 6.0.16 sources  or build tomcat from the
newest trunk?
either or, they haven't changed that much

Filip
We've ran another test and encountered a network error (packet got no ACK)
on the multicast interfaces. The wireshark log (part of) is attached below.
Perhaps you've an idea why this is happening!?
Thanks for your help!
Guido

--------------------- wireshark log start --------------------
No.     Time        Source                Destination           Protocol
Info
37433 1543.747634 223.99.114.20 223.99.114.21 TCP 36615 > 14022 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=568786762 TSER=0 WS=2

Frame 37433 (74 bytes on wire, 74 bytes captured)
    Arrival Time: Feb 12, 2008 22:04:28.652917000
    [Time delta from previous captured frame: 0.071229000 seconds]
    [Time delta from previous displayed frame: 1543.747634000 seconds]
    [Time since reference or first frame: 1543.747634000 seconds]
    Frame Number: 37433
    Frame Length: 74 bytes
    Capture Length: 74 bytes
    [Frame is marked: False]
    [Protocols in frame: eth:ip:tcp]
    [Coloring Rule Name: TCP SYN/FIN]
    [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: Broadcom_11:2a:c1 (00:10:18:11:2a:c1), Dst:
Broadcom_17:7f:8c (00:10:18:17:7f:8c)
    Destination: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
        Address: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
    Source: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
        Address: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
    Type: IP (0x0800)
Internet Protocol, Src: 223.99.114.20 (223.99.114.20), Dst: 223.99.114.21
(223.99.114.21)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 60
    Identification: 0x926a (37482)
    Flags: 0x04 (Don't Fragment)
        0... = Reserved bit: Not set
        .1.. = Don't fragment: Set
        ..0. = More fragments: Not set
    Fragment offset: 0
    Time to live: 64
    Protocol: TCP (0x06)
    Header checksum: 0x0561 [correct]
        [Good: True]
        [Bad : False]
    Source: 223.99.114.20 (223.99.114.20)
    Destination: 223.99.114.21 (223.99.114.21)
Transmission Control Protocol, Src Port: 36615 (36615), Dst Port: 14022
(14022), Seq: 0, Len: 0
    Source port: 36615 (36615)
    Destination port: 14022 (14022)
    Sequence number: 0    (relative sequence number)
    Header length: 40 bytes
    Flags: 0x02 (SYN)
        0... .... = Congestion Window Reduced (CWR): Not set
        .0.. .... = ECN-Echo: Not set
        ..0. .... = Urgent: Not set
        ...0 .... = Acknowledgment: Not set
        .... 0... = Push: Not set
        .... .0.. = Reset: Not set
        .... ..1. = Syn: Set
        .... ...0 = Fin: Not set
    Window size: 5840
    Checksum: 0x7df2 [correct]
        [Good Checksum: True]
        [Bad Checksum: False]
    Options: (20 bytes)
        Maximum segment size: 1460 bytes
        SACK permitted
        Timestamps: TSval 568786762, TSecr 0
        NOP
        Window scale: 2 (multiply by 4)

No.     Time        Source                Destination           Protocol
Info
137308 1547.148341 223.99.114.20 223.99.114.21 TCP 36615 > 14022 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=568786762 TSER=0 WS=2

Frame 137308 (74 bytes on wire, 74 bytes captured)
    Arrival Time: Feb 12, 2008 22:04:32.053624000
    [Time delta from previous captured frame: 0.071236000 seconds]
    [Time delta from previous displayed frame: 3.400707000 seconds]
    [Time since reference or first frame: 1547.148341000 seconds]
    Frame Number: 137308
    Frame Length: 74 bytes
    Capture Length: 74 bytes
    [Frame is marked: False]
    [Protocols in frame: eth:ip:tcp]
    [Coloring Rule Name: TCP SYN/FIN]
    [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: Broadcom_11:2a:c1 (00:10:18:11:2a:c1), Dst:
Broadcom_17:7f:8c (00:10:18:17:7f:8c)
    Destination: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
        Address: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
    Source: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
        Address: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
    Type: IP (0x0800)
Internet Protocol, Src: 223.99.114.20 (223.99.114.20), Dst: 223.99.114.21
(223.99.114.21)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 60
    Identification: 0x926a (37482)
    Flags: 0x04 (Don't Fragment)
        0... = Reserved bit: Not set
        .1.. = Don't fragment: Set
        ..0. = More fragments: Not set
    Fragment offset: 0
    Time to live: 64
    Protocol: TCP (0x06)
    Header checksum: 0x0561 [correct]
        [Good: True]
        [Bad : False]
    Source: 223.99.114.20 (223.99.114.20)
    Destination: 223.99.114.21 (223.99.114.21)
Transmission Control Protocol, Src Port: 36615 (36615), Dst Port: 14022
(14022), Seq: 0, Len: 0
    Source port: 36615 (36615)
    Destination port: 14022 (14022)
    Sequence number: 0    (relative sequence number)
    Header length: 40 bytes
    Flags: 0x02 (SYN)
        0... .... = Congestion Window Reduced (CWR): Not set
        .0.. .... = ECN-Echo: Not set
        ..0. .... = Urgent: Not set
        ...0 .... = Acknowledgment: Not set
        .... 0... = Push: Not set
        .... .0.. = Reset: Not set
        .... ..1. = Syn: Set
        .... ...0 = Fin: Not set
    Window size: 5840
    Checksum: 0x7df2 [correct]
        [Good Checksum: True]
        [Bad Checksum: False]
    Options: (20 bytes)
        Maximum segment size: 1460 bytes
        SACK permitted
        Timestamps: TSval 568786762, TSecr 0
        NOP
        Window scale: 2 (multiply by 4)

No.     Time        Source                Destination           Protocol
Info
37451 1544.749177 223.99.114.20 223.99.114.21 TCP 36616 > 14022 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=568787763 TSER=0 WS=2

Frame 37451 (74 bytes on wire, 74 bytes captured)
    Arrival Time: Feb 12, 2008 22:04:29.654460000
    [Time delta from previous captured frame: 0.076938000 seconds]
    [Time delta from previous displayed frame: 1544.749177000 seconds]
    [Time since reference or first frame: 1544.749177000 seconds]
    Frame Number: 37451
    Frame Length: 74 bytes
    Capture Length: 74 bytes
    [Frame is marked: False]
    [Protocols in frame: eth:ip:tcp]
    [Coloring Rule Name: TCP SYN/FIN]
    [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: Broadcom_11:2a:c1 (00:10:18:11:2a:c1), Dst:
Broadcom_17:7f:8c (00:10:18:17:7f:8c)
    Destination: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
        Address: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
    Source: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
        Address: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
    Type: IP (0x0800)
Internet Protocol, Src: 223.99.114.20 (223.99.114.20), Dst: 223.99.114.21
(223.99.114.21)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 60
    Identification: 0xbb3b (47931)
    Flags: 0x04 (Don't Fragment)
        0... = Reserved bit: Not set
        .1.. = Don't fragment: Set
        ..0. = More fragments: Not set
    Fragment offset: 0
    Time to live: 64
    Protocol: TCP (0x06)
    Header checksum: 0xdc8f [correct]
        [Good: True]
        [Bad : False]
    Source: 223.99.114.20 (223.99.114.20)
    Destination: 223.99.114.21 (223.99.114.21)
Transmission Control Protocol, Src Port: 36616 (36616), Dst Port: 14022
(14022), Seq: 0, Len: 0
    Source port: 36616 (36616)
    Destination port: 14022 (14022)
    Sequence number: 0    (relative sequence number)
    Header length: 40 bytes
    Flags: 0x02 (SYN)
        0... .... = Congestion Window Reduced (CWR): Not set
        .0.. .... = ECN-Echo: Not set
        ..0. .... = Urgent: Not set
        ...0 .... = Acknowledgment: Not set
        .... 0... = Push: Not set
        .... .0.. = Reset: Not set
        .... ..1. = Syn: Set
        .... ...0 = Fin: Not set
    Window size: 5840
    Checksum: 0x2e75 [correct]
        [Good Checksum: True]
        [Bad Checksum: False]
    Options: (20 bytes)
        Maximum segment size: 1460 bytes
        SACK permitted
        Timestamps: TSval 568787763, TSecr 0
        NOP
        Window scale: 2 (multiply by 4)

No.     Time        Source                Destination           Protocol
Info
137326 1548.149884 223.99.114.20 223.99.114.21 TCP 36616 > 14022 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=568787763 TSER=0 WS=2

Frame 137326 (74 bytes on wire, 74 bytes captured)
    Arrival Time: Feb 12, 2008 22:04:33.055167000
    [Time delta from previous captured frame: 0.077082000 seconds]
    [Time delta from previous displayed frame: 3.400707000 seconds]
    [Time since reference or first frame: 1548.149884000 seconds]
    Frame Number: 137326
    Frame Length: 74 bytes
    Capture Length: 74 bytes
    [Frame is marked: False]
    [Protocols in frame: eth:ip:tcp]
    [Coloring Rule Name: TCP SYN/FIN]
    [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: Broadcom_11:2a:c1 (00:10:18:11:2a:c1), Dst:
Broadcom_17:7f:8c (00:10:18:17:7f:8c)
    Destination: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
        Address: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
    Source: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
        Address: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
    Type: IP (0x0800)
Internet Protocol, Src: 223.99.114.20 (223.99.114.20), Dst: 223.99.114.21
(223.99.114.21)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 60
    Identification: 0xbb3b (47931)
    Flags: 0x04 (Don't Fragment)
        0... = Reserved bit: Not set
        .1.. = Don't fragment: Set
        ..0. = More fragments: Not set
    Fragment offset: 0
    Time to live: 64
    Protocol: TCP (0x06)
    Header checksum: 0xdc8f [correct]
        [Good: True]
        [Bad : False]
    Source: 223.99.114.20 (223.99.114.20)
    Destination: 223.99.114.21 (223.99.114.21)
Transmission Control Protocol, Src Port: 36616 (36616), Dst Port: 14022
(14022), Seq: 0, Len: 0
    Source port: 36616 (36616)
    Destination port: 14022 (14022)
    Sequence number: 0    (relative sequence number)
    Header length: 40 bytes
    Flags: 0x02 (SYN)
        0... .... = Congestion Window Reduced (CWR): Not set
        .0.. .... = ECN-Echo: Not set
        ..0. .... = Urgent: Not set
        ...0 .... = Acknowledgment: Not set
        .... 0... = Push: Not set
        .... .0.. = Reset: Not set
        .... ..1. = Syn: Set
        .... ...0 = Fin: Not set
    Window size: 5840
    Checksum: 0x2e75 [correct]
        [Good Checksum: True]
        [Bad Checksum: False]
    Options: (20 bytes)
        Maximum segment size: 1460 bytes
        SACK permitted
        Timestamps: TSval 568787763, TSecr 0
        NOP
        Window scale: 2 (multiply by 4)
--------------------- wireshark log end --------------------


Filip Hanik - Dev Lists wrote:
[CLUSTERLOG] NioReceiver - WARN - 02/12/2008 17:34:23,895 - NioReceiver -
Channel key is registered, but has had no interest ops for the last 3000
ms.
(canc
elled:false):[EMAIL PROTECTED] last access:2008-02-12
17:34:17.957
and finally:

this message most likely indicates that the thread pool handling incoming request is busy.
at this point do a thread dump to see the state of the threads.
There was a deadlock fix in relation to Raul's error, and was fixed in this SVN commit
http://svn.apache.org/viewvc?view=rev&revision=618823

it has been proposed to be merged back into the 6.0.x branch

Filip

gp456 wrote:
Hello,

we have a similar problem as Raul Garcia has posted in http://www.nabble.com/Tomcat-6---Cluster-error.-to14921385.html#a14921385
this  message, but we're receiving the error of a disappearing member
after
appr. 12 hours!!
Our load is also similar: 1 message/sec.

We are using a hardware loadbalancer in front of two apache/tomcat pairs.
The apache are connected via ajp to the local tomcat or (if the local tomcat is down) to the tomcat on the
other node.
Our config:
Node1/2:
Apache 2.0.59
Tomcat 6.0.13

Tomcat conf Node 1:

...
 <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
                channelSendOptions="8">

          <Manager
className="org.apache.catalina.ha.session.DeltaManager"
                   expireSessionsOnShutdown="false"
                   notifyListenersOnReplication="true"/>

          <Channel
className="org.apache.catalina.tribes.group.GroupChannel">
            <Membership
className="org.apache.catalina.tribes.membership.McastService"
                        address="239.252.100.12"
                        port="14022"
                        frequency="1000"
                        dropTime="30000"/>
            <Receiver
className="org.apache.catalina.tribes.transport.nio.NioReceiver"
                      address="223.99.114.21"
                      port="14022"
                      autoBind="100"
                      selectorTimeout="5000"
                      maxThreads="6"/>

            <Sender
className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
              <Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
timeout="60000" maxRetryAttempts="10" keepAliveCount="0" />
            </Sender>
            <Interceptor
className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
            <Interceptor
className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
            <Interceptor
className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"
interval="1000" />
          </Channel>

          <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
                 filter=""/>
          <Valve
className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>

          <!-- Deployer
className="org.apache.catalina.ha.deploy.FarmWarDeployer"
                    tempDir="/tmp/war-temp/"
                    deployDir="/tmp/war-deploy/"
                    watchDir="/tmp/war-listen/"
                    watchEnabled="false"/ -->

          <ClusterListener
className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
          <ClusterListener
className="org.apache.catalina.ha.session.ClusterSessionListener"/>
        </Cluster>
...

Tomcat conf Node 2:
...
       <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
                channelSendOptions="8">

          <Manager
className="org.apache.catalina.ha.session.DeltaManager"
                   expireSessionsOnShutdown="false"
                   notifyListenersOnReplication="true"/>

          <Channel
className="org.apache.catalina.tribes.group.GroupChannel">
            <Membership
className="org.apache.catalina.tribes.membership.McastService"
                        address="239.252.100.12"
                        port="14022"
                        frequency="1000"
                        dropTime="30000"/>
            <Receiver
className="org.apache.catalina.tribes.transport.nio.NioReceiver"
                      address="223.99.114.20"
                      port="14022"
                      autoBind="100"
                      selectorTimeout="5000"
                      maxThreads="6"/>

            <Sender
className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
              <Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
timeout="60000" maxRetryAttempts="10" keepAliveCount="0" />
            </Sender>
            <Interceptor
className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
            <Interceptor
className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
            <Interceptor
className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"
interval="1000" />
          </Channel>

          <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
                 filter=""/>
          <Valve
className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>

          <!-- Deployer
className="org.apache.catalina.ha.deploy.FarmWarDeployer"
                    tempDir="/tmp/war-temp/"
                    deployDir="/tmp/war-deploy/"
                    watchDir="/tmp/war-listen/"
                    watchEnabled="false"/ -->

          <ClusterListener
className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
          <ClusterListener
className="org.apache.catalina.ha.session.ClusterSessionListener"/>
        </Cluster>
...

Log On Node 1:

...
[CLUSTERLOG] main - INFO - 02/12/2008 10:05:15,875 - McastService -
Sleeping
for 2000 milliseconds to establish cluster membership, start level:8
[CLUSTERLOG] pool-2-thread-1 - INFO - 02/12/2008 10:05:15,908 -
BufferPool -
Created a buffer pool with max size:104857600 bytes of
type:org.apache.catalina
.tribes.io.BufferPool15Impl
[CLUSTERLOG] main - INFO - 02/12/2008 10:05:17,877 - McastService - Done
sleeping, membership established, start level:8
[CLUSTERLOG] main - INFO - 02/12/2008 10:05:17,884 - JvmRouteBinderValve
-
JvmRouteBinderValve started
[CLUSTERLOG] pool-2-thread-3 - WARN - 02/12/2008 10:05:19,328 -
ClusterListener - Context manager doesn't exist:
[CLUSTERLOG] main - INFO - 02/12/2008 10:05:20,015 - DeltaManager -
Register
manager  to cluster element Host with name localhost
[CLUSTERLOG] main - INFO - 02/12/2008 10:05:20,015 - DeltaManager -
Starting
clustering manager at
[CLUSTERLOG] main - WARN - 02/12/2008 10:05:20,026 - DeltaManager -
Manager
[], requesting session state from
org.apache.catalina.tribes.membership.MemberIm
pl[tcp://223.99.114.21:14022,223.99.114.21,14022, alive=167423,id={10 -86
53
5 6 -125 71 -7 -109 5 -36 4 48 -46 45 65 }, payload={}, command={},
domain={},
]. This operation will timeout if no session state has been received
within
60 seconds.
[CLUSTERLOG] pool-1-thread-1 - INFO - 02/12/2008 10:05:20,045 -
ThroughputInterceptor - ThroughputInterceptor Report[
        Tx Msg:1 messages
        Sent:0,00 MB (total)
        Sent:0,00 MB (application)
        Time:0,02 seconds
        Tx Speed:0,03 MB/sec (total)
        TxSpeed:0,03 MB/sec (application)
        Error Msg:0
        Rx Msg:2 messages
        Rx Speed:0,00 MB/sec (since 1st msg)
        Received:0,00 MB]
...
after some time we're getting this entry:

[CLUSTERLOG] pool-2-thread-4 - WARN - 02/12/2008 15:00:52,044 -
ClusterListener - Context manager doesn't exist:
[CLUSTERLOG] pool-2-thread-5 - WARN - 02/12/2008 15:00:52,910 -
ClusterListener - Context manager doesn't exist:
[CLUSTERLOG] pool-2-thread-2 - WARN - 02/12/2008 15:00:54,299 -
ClusterListener - Context manager doesn't exist:
[CLUSTERLOG] pool-2-thread-6 - WARN - 02/12/2008 15:00:54,707 -
ClusterListener - Context manager doesn't exist:
[CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
INFO -
02/12/2008 15:00:56,011 - DeltaManager - Register manager  to cluster
element H
ost with name localhost
[CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
INFO -
02/12/2008 15:00:56,011 - DeltaManager - Starting clustering manager at
[CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
WARN -
02/12/2008 15:00:56,012 - DeltaManager - Manager [], requesting session
state f
rom
org.apache.catalina.tribes.membership.MemberImpl[tcp://223.99.114.21:14022,223.99.114.21,14022,
alive=17902419,id={10 -86 53 5 6 -125 71 -7 -109 5 -36 4
 48 -46 45 65 }, payload={}, command={}, domain={}, ]. This operation
will
timeout if no session state has been received within 60 seconds.
[CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
INFO -
02/12/2008 15:01:02,245 - DeltaManager - Manager []; session state send
at
12.0
2.08 15:00 received in 6.233 ms.
[CLUSTERLOG] pool-2-thread-2 - INFO - 02/12/2008 15:01:13,178 -
ThroughputInterceptor - ThroughputInterceptor Report[
        Tx Msg:20003 messages
        Sent:143,22 MB (total)
        Sent:143,22 MB (application)
        Time:54,66 seconds
        Tx Speed:2,62 MB/sec (total)
        TxSpeed:2,62 MB/sec (application)
        Error Msg:0
        Rx Msg:20000 messages
        Rx Speed:0,01 MB/sec (since 1st msg)
        Received:143,04 MB]

[CLUSTERLOG] pool-2-thread-5 - INFO - 02/12/2008 15:23:49,278 -
ThroughputInterceptor - ThroughputInterceptor Report[
        Tx Msg:20606 messages
        Sent:156,13 MB (total)
        Sent:156,13 MB (application)
        Time:61,55 seconds
        Tx Speed:2,54 MB/sec (total)
        TxSpeed:2,54 MB/sec (application)
        Error Msg:9
        Rx Msg:21000 messages
        Rx Speed:0,01 MB/sec (since 1st msg)
        Received:150,14 MB]

and now we're receiving and sending message but also Error messages....
and then:

[CLUSTERLOG] NioReceiver - WARN - 02/12/2008 17:34:23,895 - NioReceiver -
Channel key is registered, but has had no interest ops for the last 3000
ms.
(canc
elled:false):[EMAIL PROTECTED] last access:2008-02-12
17:34:17.957
and finally:

[CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:28,652 - TcpFailureDetector -
Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://
223.99.114.21:14022,223.99.114.21,14022, alive=43315450,id={10 -86 53 5 6
-125 71 -7 -109 5 -36 4 48 -46 45 65 }, payload={}, command={},
domain={},
]] mess
age. Will verify.
[CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:29,654 - TcpFailureDetector -
Verification complete. Member
disappeared[org.apache.catalina.tribes.membership.Me
mberImpl[tcp://223.99.114.21:14022,223.99.114.21,14022,
alive=43316452,id={10 -86 53 5 6 -125 71 -7 -109 5 -36 4 48 -46 45 65 },
payload={}, command={}, dom
ain={}, ]]
[CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:29,654 - SimpleTcpCluster -
Received member
disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp://2
23.99.114.21:14022,223.99.114.21,14022, alive=43316452,id={10 -86 53 5 6
-125 71 -7 -109 5 -36 4 48 -46 45 65 }, payload={}, command={},
domain={}, ]
[CLUSTERLOG] 4564 - ERROR - 02/12/2008 22:04:29,654 - SimpleTcpCluster -
Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed
out(60000
ms.).; Faulty members:tcp://223.99.114.21:14022;
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:97)
        at
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
        at
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
        at
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
        at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
        at
org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.sendMessage(ThroughputInterceptor.java:61)
        at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
        at
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:60)
        at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
        at
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
        at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
        at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
        at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
        at
org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:835)
        at
org.apache.catalina.ha.tcp.SimpleTcpCluster.sendClusterDomain(SimpleTcpCluster.java:814)
        at
org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:551)
        at
org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:535)
        at
org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:517)
        at
org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:428)
        at
org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:362)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:261)
        at
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
        at
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
        at
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
        at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
        at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:686)
        at java.lang.Thread.run(Thread.java:595)
[CLUSTERLOG] 4575 - ERROR - 02/12/2008 22:04:34,920 - SimpleTcpCluster -
Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: No destination given; No
faulty
members identified.
        at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:194)
        at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
...
the error messages are repeating...


Log On Node 2:

[CLUSTERLOG] main - WARN - 02/12/2008 10:02:35,175 - SimpleTcpCluster -
Dynamic setProperty(className,value) has been disabled, please use
explicit
properties
 for the element you are trying to identify
[CLUSTERLOG] main - INFO - 02/12/2008 10:02:35,954 - SimpleTcpCluster -
Cluster is about to start
[CLUSTERLOG] main - INFO - 02/12/2008 10:02:35,973 - ReceiverBase -
Receiver
Server Socket bound to:/223.99.114.21:14022
[CLUSTERLOG] main - INFO - 02/12/2008 10:02:36,025 - McastService -
Setting
cluster mcast soTimeout to 1000
[CLUSTERLOG] main - INFO - 02/12/2008 10:02:36,037 - McastService -
Sleeping
for 2000 milliseconds to establish cluster membership, start level:4
[CLUSTERLOG] Thread-3 - INFO - 02/12/2008 10:02:36,913 - SimpleTcpCluster
-
Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://22
3.99.114.20:14022,223.99.114.20,14022, alive=322404622,id={109 -10 62 -21
19
102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={}, command={},
domain={}, ]
[CLUSTERLOG] main - INFO - 02/12/2008 10:02:38,039 - McastService - Done
sleeping, membership established, start level:4
[CLUSTERLOG] main - INFO - 02/12/2008 10:02:38,044 - McastService -
Sleeping
for 2000 milliseconds to establish cluster membership, start level:8
[CLUSTERLOG] pool-2-thread-1 - INFO - 02/12/2008 10:02:38,074 -
BufferPool -
Created a buffer pool with max size:104857600 bytes of
type:org.apache.catalina.t
ribes.io.BufferPool15Impl
[CLUSTERLOG] pool-2-thread-3 - WARN - 02/12/2008 10:02:38,312 -
ClusterListener - Context manager doesn't exist:
[CLUSTERLOG] pool-2-thread-4 - WARN - 02/12/2008 10:02:38,816 -
ClusterListener - Context manager doesn't exist:
[CLUSTERLOG] pool-2-thread-5 - WARN - 02/12/2008 10:02:39,277 -
ClusterListener - Context manager doesn't exist:
[CLUSTERLOG] pool-2-thread-6 - WARN - 02/12/2008 10:02:39,337 -
ClusterListener - Context manager doesn't exist:
[CLUSTERLOG] main - INFO - 02/12/2008 10:02:40,046 - McastService - Done
sleeping, membership established, start level:8
[CLUSTERLOG] main - INFO - 02/12/2008 10:02:40,053 - JvmRouteBinderValve
-
JvmRouteBinderValve started
[CLUSTERLOG] pool-2-thread-1 - WARN - 02/12/2008 10:02:42,816 -
ClusterListener - Context manager doesn't exist:
[CLUSTERLOG] main - INFO - 02/12/2008 10:02:43,093 - DeltaManager -
Register
manager  to cluster element Host with name localhost
[CLUSTERLOG] main - INFO - 02/12/2008 10:02:43,093 - DeltaManager -
Starting
clustering manager at
[CLUSTERLOG] main - WARN - 02/12/2008 10:02:43,100 - DeltaManager -
Manager
[], requesting session state from
org.apache.catalina.tribes.membership.MemberImpl
[tcp://223.99.114.20:14022,223.99.114.20,14022, alive=322410633,id={109
-10
62 -21 19 102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={},
command={},
domai
n={}, ]. This operation will timeout if no session state has been
received
within 60 seconds.
[CLUSTERLOG] pool-1-thread-1 - INFO - 02/12/2008 10:02:43,112 -
ThroughputInterceptor - ThroughputInterceptor Report[
        Tx Msg:1 messages
        Sent:0,00 MB (total)
        Sent:0,00 MB (application)
        Time:0,01 seconds
        Tx Speed:0,04 MB/sec (total)
        TxSpeed:0,04 MB/sec (application)
        Error Msg:0
        Rx Msg:6 messages
        Rx Speed:0,00 MB/sec (since 1st msg)
        Received:0,01 MB]

[CLUSTERLOG] main - INFO - 02/12/2008 10:02:43,719 - DeltaManager -
Manager
[]; session state send at 12.02.08 10:02 received in 624 ms.
[CLUSTERLOG] Thread-31 - INFO - 02/12/2008 10:04:55,941 -
TcpFailureDetector
- Verification complete. Member
disappeared[org.apache.catalina.tribes.membership
.MemberImpl[tcp://223.99.114.20:14022,223.99.114.20,14022,
alive=322543666,id={109 -10 62 -21 19 102 69 -42 -92 -123 -59 -15 37 16
50
-5 }, payload={}, comman
d={66 65 66 89 45 65 76 69 88 ...(9)}, domain={}, ]]
[CLUSTERLOG] Thread-30 - INFO - 02/12/2008 10:04:55,941 -
TcpFailureDetector
- Verification complete. Member
disappeared[org.apache.catalina.tribes.membership
.MemberImpl[tcp://223.99.114.20:14022,223.99.114.20,14022,
alive=322543666,id={109 -10 62 -21 19 102 69 -42 -92 -123 -59 -15 37 16
50
-5 }, payload={}, comman
d={66 65 66 89 45 65 76 69 88 ...(9)}, domain={}, ]]
[CLUSTERLOG] Thread-31 - INFO - 02/12/2008 10:04:55,941 -
SimpleTcpCluster -
Received member
disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp:
//223.99.114.20:14022,223.99.114.20,14022, alive=322543666,id={109 -10 62
-21 19 102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={}, command={66
65
66 89 4
5 65 76 69 88 ...(9)}, domain={}, ]
[CLUSTERLOG] Thread-30 - INFO - 02/12/2008 10:04:55,941 -
SimpleTcpCluster -
Received member
disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp:
//223.99.114.20:14022,223.99.114.20,14022, alive=322543666,id={109 -10 62
-21 19 102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={}, command={66
65
66 89 4
5 65 76 69 88 ...(9)}, domain={}, ]
[CLUSTERLOG] Thread-36 - INFO - 02/12/2008 10:05:19,299 -
SimpleTcpCluster -
Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://2
23.99.114.20:14022,223.99.114.20,14022, alive=2028,id={19 -115 -1 -101 8
91
66 -91 -112 105 -121 109 -82 11 56 50 }, payload={}, command={},
domain={},
]
[CLUSTERLOG] pool-2-thread-4 - INFO - 02/12/2008 10:25:11,742 -
ThroughputInterceptor - ThroughputInterceptor Report[
        Tx Msg:892 messages
        Sent:5,82 MB (total)
        Sent:5,82 MB (application)
        Time:1,33 seconds
        Tx Speed:4,39 MB/sec (total)
        TxSpeed:4,39 MB/sec (application)
        Error Msg:0
        Rx Msg:1000 messages
        Rx Speed:0,00 MB/sec (since 1st msg)
        Received:4,47 MB]

...
[CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
INFO -
02/12/2008 15:01:16,005 - DeltaManager - Manager [] expiring sessions
upon
shutdo
wn
[CLUSTERLOG] pool-2-thread-1 - WARN - 02/12/2008 15:01:20,014 -
ClusterListener - Context manager doesn't exist:
[CLUSTERLOG] pool-2-thread-1 - WARN - 02/12/2008 15:01:20,015 -
ClusterListener - Context manager doesn't exist:
[CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
INFO -
02/12/2008 15:01:20,219 - DeltaManager - Register manager  to cluster
element Hos
t with name localhost
[CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
INFO -
02/12/2008 15:01:20,220 - DeltaManager - Starting clustering manager at
[CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
WARN -
02/12/2008 15:01:20,220 - DeltaManager - Manager [], requesting session
state fro
m
org.apache.catalina.tribes.membership.MemberImpl[tcp://223.99.114.20:14022,223.99.114.20,14022,
alive=17762449,id={19 -115 -1 -101 8 91 66 -91 -112 105 -121
 109 -82 11 56 50 }, payload={}, command={}, domain={}, ]. This operation
will timeout if no session state has been received within 60 seconds.
[CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
INFO -
02/12/2008 15:01:24,093 - DeltaManager - Manager []; session state send
at
12.02.
08 15:01 received in 3.872 ms.
[CLUSTERLOG] pool-1-thread-1 - INFO - 02/12/2008 15:22:41,143 -
ThroughputInterceptor - ThroughputInterceptor Report[
        Tx Msg:21000 messages
        Sent:150,30 MB (total)
        Sent:150,30 MB (application)
        Time:70,58 seconds
        Tx Speed:2,13 MB/sec (total)
        TxSpeed:2,13 MB/sec (application)
        Error Msg:0
        Rx Msg:20534 messages
        Rx Speed:0,01 MB/sec (since 1st msg)
        Received:156,92 MB]

after this entry we're NOT receiving any messages (Rx Msg):

[CLUSTERLOG] pool-1-thread-2 - INFO - 02/12/2008 15:40:02,612 -
ThroughputInterceptor - ThroughputInterceptor Report[
        Tx Msg:22000 messages
        Sent:154,75 MB (total)
        Sent:154,75 MB (application)
        Time:72,92 seconds
        Tx Speed:2,12 MB/sec (total)
        TxSpeed:2,12 MB/sec (application)
        Error Msg:0
        Rx Msg:20534 messages
        Rx Speed:0,01 MB/sec (since 1st msg)
        Received:156,92 MB]

[CLUSTERLOG] pool-1-thread-2 - INFO - 02/12/2008 15:54:15,184 -
ThroughputInterceptor - ThroughputInterceptor Report[
        Tx Msg:23022 messages
        Sent:156,30 MB (total)
        Sent:156,30 MB (application)
        Time:74,46 seconds
        Tx Speed:2,10 MB/sec (total)
        TxSpeed:2,10 MB/sec (application)
        Error Msg:0
        Rx Msg:20534 messages
        Rx Speed:0,01 MB/sec (since 1st msg)
        Received:156,92 MB]
...



Many thanks for your help!!!












---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]






---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Reply via email to