Michael Kjellman created CASSANDRA-5229:
-------------------------------------------
Summary: after a IOException is thrown during streaming, streaming
tasks hang in netstats
Key: CASSANDRA-5229
URL: https://issues.apache.org/jira/browse/CASSANDRA-5229
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.2.1
Environment: Ubuntu 12.04
Reporter: Michael Kjellman
Priority: Critical
After an IOExcpetion, streaming tasks marked as "successful" in the logs are
hung in netstats
With TRACE debugging on streaming on the receiving node everything about the
sstable in the log (not very much)
{code}
INFO [AntiEntropyStage:1] 2013-02-07 11:23:44,717 StreamOut.java (line 151)
Stream context metadata
[/data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-5-Data.db
sections=3068 progress=0/2785204713 - 0%,
/data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-25-Data.db
sections=2696 progress=0/758409465 - 0%,
/data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-60-Data.db
sections=3099 progress=0/238876436 - 0%,
/data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-63-Data.db
sections=1166 progress=0/2125323 - 0%,
/data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-38-Data.db
sections=2507 progress=0/515992757 - 0%,
/data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-26-Data.db
sections=3153 progress=0/994857654 - 0%,
/data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-57-Data.db
sections=3116 progress=0/129398170 - 0%,
/data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-58-Data.db
sections=217 progress=0/72286 - 0%,
/data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-59-Data.db
sections=3146 progress=0/3357709019 - 0%], 27 sstables.
INFO [AntiEntropyStage:1] 2013-02-07 11:23:52,964 StreamOut.java (line 151)
Stream context metadata
[/data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-5-Data.db
sections=2930 progress=0/2799914560 - 0%,
/data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-25-Data.db
sections=2590 progress=0/761266059 - 0%,
/data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-60-Data.db
sections=2956 progress=0/241362497 - 0%,
/data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-63-Data.db
sections=1153 progress=0/2125323 - 0%,
/data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-38-Data.db
sections=2422 progress=0/522126371 - 0%,
/data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-26-Data.db
sections=3004 progress=0/998401202 - 0%,
/data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-57-Data.db
sections=2974 progress=0/129722346 - 0%,
/data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-58-Data.db
sections=220 progress=0/72286 - 0%,
/data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-59-Data.db
sections=2998 progress=0/3375554099 - 0%], 27 sstables.
{code}
node that is streaming out thinks that the streaming session was successful
{code}
INFO [MiscStage:1] 2013-02-07 11:23:38,022 StreamOut.java (line 151) Stream
context metadata
[/data/cassandra/evidence/fingerprints/evidence-fingerprints-ia-472-Data.db
sections=1727 progress=0/210208515 - 0%,
/var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-919-Data.db
sections=1746 progress=0/119438030 - 0%,
/data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-920-Data.db
sections=1681 progress=0/54498226 - 0%,
/var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-922-Data.db
sections=16 progress=0/13490 - 0%,
/var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-918-Data.db
sections=632 progress=0/70019542 - 0%,
/var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-921-Data.db
sections=1644 progress=0/39870238 - 0%,
/data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-497-Data.db
sections=1569 progress=0/208331077 - 0%,
/var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-923-Data.db
sections=1572 progress=0/30870478 - 0%,
/var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-925-Data.db
sections=167 progress=0/1845123 - 0%,
/var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-703-Data.db
sections=1574 progress=0/287386471 - 0%,
/var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-913-Data.db
sections=811 progress=0/103776521 - 0%,
/var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-915-Data.db
sections=1539 progress=0/141864261 - 0%], 12 sstables.
INFO [MiscStage:1] 2013-02-07 11:23:49,938 StreamOut.java (line 151) Stream
context metadata
[/var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-919-Data.db
sections=3153 progress=0/994857654 - 0%,
/var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-918-Data.db
sections=2507 progress=0/515992757 - 0%,
/var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-923-Data.db
sections=3153 progress=0/131969347 - 0%,
/var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-703-Data.db
sections=3068 progress=0/2784807967 - 0%,
/var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-913-Data.db
sections=2696 progress=0/758409465 - 0%,
/data/cassandra/evidence/fingerprints/evidence-fingerprints-ia-472-Data.db
sections=3150 progress=0/1792868996 - 0%,
/data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-920-Data.db
sections=3123 progress=0/240094510 - 0%,
/var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-929-Data.db
sections=18 progress=0/29468 - 0%,
/var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-922-Data.db
sections=217 progress=0/13490 - 0%,
/var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-921-Data.db
sections=3124 progress=0/130320291 - 0%,
/data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-497-Data.db
sections=3055 progress=0/1749539598 - 0%,
/var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-925-Data.db
sections=1608 progress=0/13357410 - 0%,
/var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-915-Data.db
sections=3096 progress=0/1241397203 - 0%], 13 sstables.
INFO [Streaming to /10.8.25.132:2] 2013-02-07 11:24:18,780
StreamReplyVerbHandler.java (line 44) Successfully sent
/var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-919-Data.db
to /10.8.25.132
{code}
node that is being streamed to has nothing in the logs about this particular
sstable
{code}
Streaming from: /10.8.30.13
evidence:
/var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-919-Data.db
sections=3153 progress=218225400/994857654 - 21%
_______
Streaming from: /10.138.12.10
evidence:
/data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-26-Data.db
sections=3153 progress=218225400/994857654 - 21%
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
