I check system.log both, but there is no exception logged. On Tue, Apr 20, 2010 at 10:40, Jonathan Ellis <[email protected]> wrote:
> I don't see csArena-tmp-6-Index.db in the incoming files list. If > it's not there, that means that it did break out of that while loop. > > Did you check both logs for exceptions? > > On Mon, Apr 19, 2010 at 9:36 PM, Ingram Chen <[email protected]> wrote: > > Ouch ! I talk too early ! > > > > We still suffer same problems after upgrade to 1.6.0_20. > > > > In JMX StreamingService, I see several wired incoming/outgoing transfer: > > > > In Host A, 192.168.2.87 > > > > StreamingService Status: > > Done with transfer to /192.168.2.88 > > > > StreamingService StreamSources: > > [/192.168.2.88] > > > > StreamingService StreamDestinations: > > [/192.168.2.88] > > > > StreamingService getIncomingFiles=192.168.2.88 > > [ > > UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Index.db > > 0/5718, > > UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Filter.db > > 0/325, > > UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Data.db > > 0/29831, > > UserState: /var/lib/cassandra/data/UserState/csArena-tmp-13-Index.db > > 0/47623, > > > > ... omit several 0 received pending files..... > > > > UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-19-Data.db > > 0/355041, > > > > UserState: /var/lib/cassandra/data/UserState/mahjong-tmp-12-Data.db > > 27711/2173906, > > UserState: /var/lib/cassandra/data/UserState/darkChess-tmp-12-Data.db > > 27711/18821998, > > UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-6-Data.db > > 27711/743037, > > UserState: /var/lib/cassandra/data/UserState/big2-tmp-12-Index.db > > 27711/189214, > > UserState: > /var/lib/cassandra/data/UserState/facebookPoker99-tmp-6-Data.db > > 27711/1892375, > > UserState: > /var/lib/cassandra/data/UserState/facebookPoker99-tmp-6-Index.db > > 27711/143216, > > UserState: /var/lib/cassandra/data/UserState/csArena-tmp-6-Data.db > > 27711/201188, > > UserState: /var/lib/cassandra/data/UserState/darkChess-tmp-12-Index.db > > 27711/354923, > > UserState: /var/lib/cassandra/data/UserState/big2-tmp-12-Data.db > > 27711/1260768, > > UserState: /var/lib/cassandra/data/UserState/mahjong-tmp-12-Index.db > > 27711/332649, > > UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-6-Index.db > > 27711/39739 > > ] > > > > lots of files stalled after receiving 27711 bytes. this strange number is > > the length of first file to income, see Host B > > > > Host B, 192.168.2.88 > > > > StreamingService Status: > > Receiving stream > > > > StreamingService StreamSources: > > StreamSources: [/192.168.2.87] > > > > StreamingService StreamDestinations: > > [/192.168.2.87] > > > > StreamingService getOutgoingFiles=192.168.2.87 > > [ > > /var/lib/cassandra/data/UserState/stream/csArena-6-Index.db 27711/27711, > > /var/lib/cassandra/data/UserState/stream/csArena-6-Filter.db 0/1165, > > /var/lib/cassandra/data/UserState/stream/csArena-6-Data.db 0/201188, > > > > ... omit pending outgoing files .... > > ] > > > > It seems that outgoing files does not terminate properly. and cause the > > receiver goes into infinite loop to cause busy thread. From thread dump, > it > > looks like fc.transferFrom() in IncomingStreamReader never return: > > > > while (bytesRead < pendingFile.getExpectedBytes()) { > > bytesRead += fc.transferFrom(socketChannel, bytesRead, > > FileStreamTask.CHUNK_SIZE); > > pendingFile.update(bytesRead); > > } > > > > > > On Tue, Apr 20, 2010 at 05:48, Rob Coli <[email protected]> wrote: > >> > >> On 4/17/10 6:47 PM, Ingram Chen wrote: > >>> > >>> after upgrading jdk from 1.6.0_16 to 1.6.0_20, the problem solved. > >> > >> FYI, this sounds like it might be : > >> > >> https://issues.apache.org/jira/browse/CASSANDRA-896 > >> > >> > http://bugs.sun.com/view_bug.do;jsessionid=60c39aa55d3666c0c84dd70eb826?bug_id=6805775 > >> > >> Where garbage collection issues in JVM/JDKs before 7.b70 leads to GC > >> storming which hoses performance. > >> > >> =Rob > > > > > > > > > > > -- Ingram Chen online share order: http://dinbendon.net blog: http://www.javaworld.com.tw/roller/page/ingramchen
