> On Sep 17, 2018, at 7:29 AM, Oleksandr Shulgin <oleksandr.shul...@zalando.de>
> wrote:
>
> On Mon, Sep 17, 2018 at 4:04 PM Jeff Jirsa <jji...@gmail.com> wrote:
>>> Again, given that the tables are not updated anymore from the application
>>> and we have repaired them successfully multiple times already, how can it
>>> be that any inconsistency would be found by read-repair or normal repair?
>>>
>>> We have seen this on a number of nodes, including SSTables written at the
>>> time there was guaranteed no repair running.
>> Not obvious to me where the sstable is coming from - you’d have to look in
>> the logs. If it’s read repair, it’ll be created during a memtable flush. If
>> it’s nodetool repair, it’ll be streamed in. It could also be compaction
>> (especially tombstone compaction), in which case it’ll be in the compaction
>> logs and it’ll have an sstable ancestor in the metadata.
>
> Jeff,
>
> Thanks for your reply! Indeed it could be coming from single-SSTable
> compaction, this I didn't think about. By any chance looking into
> compaction_history table could be useful to trace it down?
>
Maybe. Also check your normal system / debug logs (depending on your version),
which will usually tell you inputs and outputs
Marcus’ idea of row lifting seems more likely, since you’re using STCS - it’s
an optimization to “lift” expensive reads into a single sstable for future
reads (if a read touches more than - I think - 4? sstables, we copy it back
into the memtable so it’s flushed into a single sstable), so if you have STCS
and you’re still doing reads, it could definitely be that.
- Jeff
