Ouch ! I talk too early !
We still suffer same problems after upgrade to 1.6.0_20.
In JMX StreamingService, I see several wired incoming/outgoing transfer:
In Host A, 192.168.2.87
StreamingService Status:
Done with transfer to /192.168.2.88
StreamingService StreamSources:
[/192.168.2.88]
StreamingService StreamDestinations:
[/192.168.2.88]
StreamingService getIncomingFiles=192.168.2.88
[
UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Index.db
0/5718,
UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Filter.db
0/325,
UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Data.db
0/29831,
UserState: /var/lib/cassandra/data/UserState/csArena-tmp-13-Index.db
0/47623,
... omit several 0 received pending files.....
UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-19-Data.db
0/355041,
UserState: /var/lib/cassandra/data/UserState/mahjong-tmp-12-Data.db
27711/2173906,
UserState: /var/lib/cassandra/data/UserState/darkChess-tmp-12-Data.db
27711/18821998,
UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-6-Data.db
27711/743037,
UserState: /var/lib/cassandra/data/UserState/big2-tmp-12-Index.db
27711/189214,
UserState: /var/lib/cassandra/data/UserState/facebookPoker99-tmp-6-Data.db
27711/1892375,
UserState: /var/lib/cassandra/data/UserState/facebookPoker99-tmp-6-Index.db
27711/143216,
UserState: /var/lib/cassandra/data/UserState/csArena-tmp-6-Data.db
27711/201188,
UserState: /var/lib/cassandra/data/UserState/darkChess-tmp-12-Index.db
27711/354923,
UserState: /var/lib/cassandra/data/UserState/big2-tmp-12-Data.db
27711/1260768,
UserState: /var/lib/cassandra/data/UserState/mahjong-tmp-12-Index.db
27711/332649,
UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-6-Index.db
27711/39739
]
lots of files stalled after receiving 27711 bytes. this strange number is
the length of first file to income, see Host B
Host B, 192.168.2.88
StreamingService Status:
Receiving stream
StreamingService StreamSources:
StreamSources: [/192.168.2.87]
StreamingService StreamDestinations:
[/192.168.2.87]
StreamingService getOutgoingFiles=192.168.2.87
[
/var/lib/cassandra/data/UserState/stream/csArena-6-Index.db 27711/27711,
/var/lib/cassandra/data/UserState/stream/csArena-6-Filter.db 0/1165,
/var/lib/cassandra/data/UserState/stream/csArena-6-Data.db 0/201188,
... omit pending outgoing files ....
]
It seems that outgoing files does not terminate properly. and cause the
receiver goes into infinite loop to cause busy thread. From thread dump, it
looks like fc.transferFrom() in IncomingStreamReader never return:
while (bytesRead < pendingFile.getExpectedBytes()) {
bytesRead += fc.transferFrom(socketChannel, bytesRead,
FileStreamTask.CHUNK_SIZE);
pendingFile.update(bytesRead);
}
On Tue, Apr 20, 2010 at 05:48, Rob Coli <[email protected]> wrote:
> On 4/17/10 6:47 PM, Ingram Chen wrote:
>
>> after upgrading jdk from 1.6.0_16 to 1.6.0_20, the problem solved.
>>
>
> FYI, this sounds like it might be :
>
> https://issues.apache.org/jira/browse/CASSANDRA-896
>
> http://bugs.sun.com/view_bug.do;jsessionid=60c39aa55d3666c0c84dd70eb826?bug_id=6805775
>
> Where garbage collection issues in JVM/JDKs before 7.b70 leads to GC
> storming which hoses performance.
>
> =Rob
>