[
https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010738#comment-14010738
]
T Jake Luciani commented on CASSANDRA-7245:
-------------------------------------------
Good news is I can reproduce now with the following dtest:
{code}
import time
from dtest import Tester, debug
from assertions import assert_unavailable
from tools import (since)
#Run with MAX_HEAP_SIZE=150M HEAP_NEWSIZE=100M
class TestNettyTransport(Tester):
@since('2.1')
def reused_bytebuffers_test(self):
debug("Creating a ring")
cluster = self.cluster
cluster.set_configuration_options(values={'concurrent_writes' : 2},
batch_commitlog=True)
cluster.populate(2).start()
[node1, node2] = cluster.nodelist()
cluster.start()
node1.stress(['write', 'n=100000', '-schema',
'replication(factor=2)','-rate', 'threads=191','-mode','native','cql3',
'prepared','-col', 'n=fixed(21)', 'size=exp(11..42)'])
node1.stress(['read', 'n=100000', '-mode','native','cql3', 'prepared'])
{code}
I've confirmed the issue is the pooled buffer allocator reclaiming the buffers
too soon. Working on patching the Mutations to keep references to the netty
frame.
> Out-of-Order keys with stress + CQL3
> ------------------------------------
>
> Key: CASSANDRA-7245
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7245
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Pavel Yaskevich
> Assignee: T Jake Luciani
> Fix For: 2.1 rc1
>
>
> We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718
> and found following problem almost in every SSTable generated (~200 GB of
> data and 821 SSTables).
> We set up they keys to be 10 bytes in size (default) and population between 1
> and 600000000.
> Once I ran 'sstablekeys' on the generated SSTable files I got following
> exceptions:
> _There is a problem with sorting of normal looking keys:_
> 30303039443538353645
> 30303039443745364242
> java.io.IOException: Key out of order! DecoratedKey(-217680888487824985,
> *30303039443745364242*) > DecoratedKey(-1767746583617597213,
> *30303039443437454333*)
> 00000a30303033343933
> 37344400001388343933
> java.io.IOException: Key out of order! DecoratedKey(5440473860101999581,
> *37344400001388343933*) > DecoratedKey(-7565486415339257200,
> *30303033344639443137*)
> 30303033354244363031
> 30303033354133423742
> java.io.IOException: Key out of order! DecoratedKey(2687072396429900180,
> *30303033354133423742*) > DecoratedKey(-7838239767410066684,
> *30303033354145344534*)
> 30303034313442354137
> 30343136353633340000
> java.io.IOException: Key out of order! DecoratedKey(1516003874415400462,
> *30343136353633340000*) > DecoratedKey(-9106177395653818217,
> *30303034313333444238*)
> 30303035373044373435
> 30303035373044334631
> java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540,
> *30303035373044334631*) > DecoratedKey(-4296696226469000945,
> *30303035373132364138*)
> _And completely different ones:_
> 30303041333745373543
> 7cd045c59a90d7587d8d
> java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196,
> *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690,
> *30303041333943303232*)
> 30303033323144444144
> 30303033323346343932
> java.io.IOException: Key out of order! DecoratedKey(7071845511166615635,
> *30303033323346343932*) > DecoratedKey(5233296131921119414,
> *53d83e00000012287e03*)
> 30303034314531374431
> 3806734b256c27e41ec2
> java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193,
> *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663,
> *30303034314136413343*)
> _And sometimes there is no problem at all:_
> 30303033353144463637
> 0000002a31b3b31a1c2f
> 5d616dd38211ebb5d6ec
> 44423645000013880000
> 00001388138844463744
> 30303033353143394343
> It's worth to mention that we have got 22 timeout exceptions but number of
> out-of-order keys is much larger than that.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
