Hi Tommy, Thank you for the reply, but I don't think this is the reason, because we don't have any static column.
-- Regards, Adeline From: Tommy Stendahl [mailto:tommy.stend...@ericsson.com] Sent: Thursday, November 17, 2016 5:16 PM To: user@cassandra.apache.org Subject: Re: lots of DigestMismatchException in cassandra3 Hi, Could it be CASSANDRA-12090<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CASSANDRA-2D12090&d=CwMF-g&c=4ZIZThykDLcoWk-GVjSLm9hvvvzvGv0FLoWSRuCSs5Q&r=552xSDXEzKpvsyZM5wpE0TGEUDzVsX35L-K72hRjpLc&m=-f-6NfSYZVngB5-JpViJQf8p87WqCGDVfLrX_wrbNIY&s=lJNXNXd0alPlegiAnSD-h3srxIrZH5Qcv4er75LnFB0&e=>? Regards, /Tommy On 2016-11-17 09:47, adeline....@thomsonreuters.com<mailto:adeline....@thomsonreuters.com> wrote: Hi, We are using Cassandra 3.7 and we have a single datacenter of 4 nodes, RF =3, and the consistency level is quorum. I found lots of DigestMismatchException in debug.log, as below: DEBUG [ReadRepairStage:11] 2016-11-17 08:24:57,066 ReadCallback.java:235 - Digest mismatch: org.apache.cassandra.service.DigestMismatchException: Mismatch for key DecoratedKey(-7187709033646443438, 0008000001586f95380000000405f72c7a0000040000000300) (c8d67cfae7be8470ca1786eaa0d8968e vs 3aa974d3eca302682c6f31ca250ea641) at org.apache.cassandra.service.DigestResolver.resolve(DigestResolver.java:85) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:226) ~[apache-cassandra-3.7.jar:3.7] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_25] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_25] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_25] INFO [HintsDispatcher:5] 2016-11-17 08:25:06,958 HintsStore.java:126 - Deleted hint file bcd84017-6068-4a76-be89-96c0130599f3-1479371095876-1.hints INFO [HintsDispatcher:5] 2016-11-17 08:25:06,962 HintsDispatchExecutor.java:263 - Finished hinted handoff of file bcd84017-6068-4a76-be89-96c0130599f3-1479371095876-1.hints to endpoint bcd84017-6068-4a76-be89-96c0130599f3 DEBUG [GossipStage:1] 2016-11-17 08:25:32,551 FailureDetector.java:456 - Ignoring interval time of 2003165297 for /10.52.153.107 DEBUG [CompactionExecutor:9] 2016-11-17 08:25:33,034 CompactionTask.java:223 - Compacted (3b3d7600-ac9f-11e6-8b55-03fd515998e4) 4 sstables to [/data/c878jrn/data/pc_object_data_beta/vsc_data-53935ff0a74811e69ab46393a19c464a/mb-2256-big,] to level=0. 85.011MiB to 53.580MiB (~63% of original) in 75,524ms. Throughput = 726.471KiB/s. 0 total partitions merged to 15,552. Partition merge counts were {1:11623, 2:3543, 3:385, 4:1, } ERROR [MessagingService-Incoming-/10.52.153.109] 2016-11-17 08:25:42,126 CassandraDaemon.java:217 - Exception in thread Thread[MessagingService-Incoming-/10.52.153.109,5,main] java.io.IOError: java.io.IOException: Corrupt empty row found in unfiltered partition at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:224) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:212) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:681) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:642) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:331) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:350) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:287) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.net.MessageIn.read(MessageIn.java:114) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:190) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92) ~[apache-cassandra-3.7.jar:3.7] Caused by: java.io.IOException: Corrupt empty row found in unfiltered partition at org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:388) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:219) ~[apache-cassandra-3.7.jar:3.7] ... 11 common frames omitted Is this normal? -- Regards, Adeline
