Hi Yuji, can you reproduce the behaviour with a single node?
The reason I ask is because I probably have the same issue with my automated tests (which run truncate between every test), which run on my local laptop. Maybe around 5 tests randomly fail out of my 1800. I can see that the failed tests sometimes show data from other tests, which I think must be because of a failed truncate. This behaviour is seems to be CQL only, or at least has gotten worse with CQL. I did not experience this with Thrift. regards, Christian On Mon, Aug 8, 2016 at 7:34 AM, Yuji Ito <y...@imagine-orb.com> wrote: > Hi all, > > I have a question about clearing table and commit log replay. > After some tables were truncated consecutively, I got some stale values. > This problem doesn't occur when I clear keyspaces with DROP (and CREATE). > > I'm testing the following test with node failure. > Some stale values appear at checking phase. > > Test iteration: > 1. initialize tables as below > 2. request a lot of read/write concurrently > 3. check all records > 4. repeat from the beginning > > I use C* 2.2.6. There are 3 nodes (replication_factor: 3). > Each node kills cassandra process at random intervals and restarts it > immediately. > > My initialization: > 1. clear tables with TRUNCATE > 2. INSERT initial records > 3. check if all values are correct > > If any phase fails (because of node failure), the initialization starts > all over again. > So, tables are sometimes truncated consecutively. > Though the check in the initialization is OK, stale data appears when I > execute "SELECT * FROM mykeyspace.mytable;" after a lot of requests are > completed. > > The problem is likely to occur when the ReplayPosition's value in > "truncated_at" is initialized as below after an empty table is truncated. > > Column Family ID: truncated_at > XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX: 0xffffffffffffffff000000000000 > 0156597cd4c7 > (this value was acquired just after phase 1 in my initialization) > > I guess some unexpected replays occur. > Does anyone know the behavior? > > Thanks, > Yuji >
