Hi Yuji, ok, perhaps you are seeing a different issue than I do.
Have you tried with durable_writes=False? If the issue is caused by the commitlog, then it should work if you disable durable_writes. Cheers, Christian On Tue, Aug 9, 2016 at 3:04 PM, Yuji Ito <y...@imagine-orb.com> wrote: > Thanks Christian > > can you reproduce the behaviour with a single node? > > I tried my test with a single node. But I can't. > > This behaviour is seems to be CQL only, or at least has gotten worse with >> CQL. I did not experience this with Thrift. > > I truncate tables with CQL. I've never tried with Thrift. > > I think that my problem can happen when truncating even succeeds. > That's because I check all records after truncating. > > I checked the source code. > ReplayPosition.segment and position become -1 and 0 (ReplayPosition.NONE) > in dscardSSTables() at truncating a table when there is no SSTable. > I guess that ReplayPosition.segment shouldn't be -1 at truncating a table > in this case. > replayMutation() can request unexpected replay mutations because of this > segment's value. > > Is there anyone familiar with truncate and replay? > > Regards, > Yuji > > > On Mon, Aug 8, 2016 at 6:36 PM, horschi <hors...@gmail.com> wrote: > >> Hi Yuji, >> >> can you reproduce the behaviour with a single node? >> >> The reason I ask is because I probably have the same issue with my >> automated tests (which run truncate between every test), which run on my >> local laptop. >> >> Maybe around 5 tests randomly fail out of my 1800. I can see that the >> failed tests sometimes show data from other tests, which I think must be >> because of a failed truncate. This behaviour is seems to be CQL only, or at >> least has gotten worse with CQL. I did not experience this with Thrift. >> >> regards, >> Christian >> >> >> >> On Mon, Aug 8, 2016 at 7:34 AM, Yuji Ito <y...@imagine-orb.com> wrote: >> >>> Hi all, >>> >>> I have a question about clearing table and commit log replay. >>> After some tables were truncated consecutively, I got some stale values. >>> This problem doesn't occur when I clear keyspaces with DROP (and CREATE). >>> >>> I'm testing the following test with node failure. >>> Some stale values appear at checking phase. >>> >>> Test iteration: >>> 1. initialize tables as below >>> 2. request a lot of read/write concurrently >>> 3. check all records >>> 4. repeat from the beginning >>> >>> I use C* 2.2.6. There are 3 nodes (replication_factor: 3). >>> Each node kills cassandra process at random intervals and restarts it >>> immediately. >>> >>> My initialization: >>> 1. clear tables with TRUNCATE >>> 2. INSERT initial records >>> 3. check if all values are correct >>> >>> If any phase fails (because of node failure), the initialization starts >>> all over again. >>> So, tables are sometimes truncated consecutively. >>> Though the check in the initialization is OK, stale data appears when I >>> execute "SELECT * FROM mykeyspace.mytable;" after a lot of requests are >>> completed. >>> >>> The problem is likely to occur when the ReplayPosition's value in >>> "truncated_at" is initialized as below after an empty table is truncated. >>> >>> Column Family ID: truncated_at >>> XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX: 0xffffffffffffffff000000000000 >>> 0156597cd4c7 >>> (this value was acquired just after phase 1 in my initialization) >>> >>> I guess some unexpected replays occur. >>> Does anyone know the behavior? >>> >>> Thanks, >>> Yuji >>> >> >> >
