Re: Max number of windows when using TWCS

2019-02-11 Thread Osman YOZGATLIOĞLU
Hello, By the way, about https://issues.apache.org/jira/browse/CASSANDRA-13418, I'm not sure how to apply this solution. Do you have a guide about it? Regards, Osman On 12.02.2019 01:42, Nitan Kainth wrote: That’s right Jeff. That’s why I am thinking why not compaction gets rid of old exit

Re: Max number of windows when using TWCS

2019-02-11 Thread Nitan Kainth
That’s right Jeff. That’s why I am thinking why not compaction gets rid of old exited sstables? Regards, Nitan Cell: 510 449 9629 > On Feb 11, 2019, at 3:53 PM, Jeff Jirsa wrote: > > It's probably not safe. You shouldn't touch the underlying sstables unless > you're very sure you know what y

Re: Max number of windows when using TWCS

2019-02-11 Thread Jeff Jirsa
It's probably not safe. You shouldn't touch the underlying sstables unless you're very sure you know what you're doing. On Mon, Feb 11, 2019 at 1:05 PM Akash Gangil wrote: > I have in the past tried to delete SSTables manually, but have noticed > bits and pieces of that data still remain, even

Re: Max number of windows when using TWCS

2019-02-11 Thread Akash Gangil
I have in the past tried to delete SSTables manually, but have noticed bits and pieces of that data still remain, even though the sstables of that window is deleted. So always wondered if playing directly with the underlying filesystem is a safe bet? On Mon, Feb 11, 2019 at 1:01 PM Jonathan Hadda

Re: Max number of windows when using TWCS

2019-02-11 Thread Jonathan Haddad
Deleting SSTables manually can be useful if you don't know your TTL up front. For example, you have an ETL process that moves your raw Cassandra data into S3 as parquet files, and you want to be sure that process is completed before you delete the data. You could also start out without setting a

Re: Max number of windows when using TWCS

2019-02-11 Thread Nitan Kainth
Jeff, It means we have to delete sstables manually? Regards, Nitan Cell: 510 449 9629 > On Feb 11, 2019, at 2:40 PM, Jeff Jirsa wrote: > > There's a bit of headache around overlapping sstables being strictly safe to > delete. https://issues.apache.org/jira/browse/CASSANDRA-13418 was added t

Re: Max number of windows when using TWCS

2019-02-11 Thread DuyHai Doan
thanks for the pointer Jeff On Mon, Feb 11, 2019 at 9:40 PM Jeff Jirsa wrote: > There's a bit of headache around overlapping sstables being strictly safe > to delete. https://issues.apache.org/jira/browse/CASSANDRA-13418 was > added to allow the "I know it's not technically safe, but just delet

Re: Max number of windows when using TWCS

2019-02-11 Thread Jeff Jirsa
There's a bit of headache around overlapping sstables being strictly safe to delete. https://issues.apache.org/jira/browse/CASSANDRA-13418 was added to allow the "I know it's not technically safe, but just delete it anyway" use case. For a lot of people who started using TWCS before 13418, "stop c

Re: Max number of windows when using TWCS

2019-02-11 Thread Nitan Kainth
Hi, In regards to comment “Purging data is also straightforward, just dropping SSTables (by a script) where create date is older than a threshold, we don't even need to rely on TTL” Doesn’t the old sstables drop by itself? One ttl and gc grace seconds past whole sstable will have only tombstone

Re: Max number of windows when using TWCS

2019-02-11 Thread DuyHai Doan
No worry for overlapping, the use-case is about events/timeseries and there is almost no delay so it should be fine. On the note-side, since we have the guarantee to have 1 SSTable/day of ingestion, this is very easy to "emulate" incremental backup. You just need to find the generated SSTable with

Re: Max number of windows when using TWCS

2019-02-11 Thread Jeff Jirsa
Wild ass guess based on a large use case I knew about at the time If you go above that, I expect it’d largely be fine as long as you were sure they weren’t overlapping so reads only ever touched a small subset of the windows (ideally 1). If you have one day windows and every read touches all of