Re: Remove folders of deleted tables

2023-12-07 Thread Bowen Song via user
There's no requirement for the partition key to contain the date/time for a TWCS table. The important thing is data need to be written to the table in chronological order (i.e. do not use the "USING TIMESTAMP" in the CQL queries) and the same TTL is used for all partitions. TWCS was introduced

Re: Remove folders of deleted tables

2023-12-07 Thread Sébastien Rebecchi
Thanks Bowen, I also thought about using TTL and TWCS, but in my past experience with Cassandra I have had a lot of issues with data models using TTL and creating many tombstones. I was probably not using the right compaction at that time, but this experiences has a great impact on me and I would s

Re: Remove folders of deleted tables

2023-12-06 Thread Bowen Song via user
There are many different ways to avoid or minimise the chance of schema disagreements, the easiest way is to always send DDL queries to the same node in the cluster. This is very easy to implement and avoids schema disagreements at the cost of creating a single point of failure for DDL queries.

Re: Remove folders of deleted tables

2023-12-06 Thread Sébastien Rebecchi
Hello Jeff, Bowen Thanks for your answer. Now I understand that there is a bug in Cassandra that can not handle concurrent schema modifications, I was not aware of that severity, I thought that temporary schema mismatches were eventually resolved smartly, by a kind of "merge" mechanism. For my use

Re: Remove folders of deleted tables

2023-12-05 Thread Bowen Song via user
The same table name with two different CF IDs is not just "temporary schema disagreements", it's much worse than that. This breaks the eventual consistency guarantee, and leads to silent data corruption. It's silently happening in the background, and you don't realise it until you suddenly do,

Re: Remove folders of deleted tables

2023-12-05 Thread Jeff Jirsa
The last time you mentioned this: On Tue, Dec 5, 2023 at 11:57 AM Sébastien Rebecchi wrote: > Hi Bowen, > > Thanks for your answer. > > I was thinking of extreme use cases, but as far as I am concerned I can > deal with creation and deletion of 2 tables every 6 hours for a keyspace. > So it lets

Re: Remove folders of deleted tables

2023-12-05 Thread Sébastien Rebecchi
Hi Bowen, Thanks for your answer. I was thinking of extreme use cases, but as far as I am concerned I can deal with creation and deletion of 2 tables every 6 hours for a keyspace. So it lets around 8 folders of deleted tables per day - sometimes more cause I can see sometimes 2 folders created fo

Re: Remove folders of deleted tables

2023-12-05 Thread Jon Haddad
I can't think of a reason to keep empty directories around, seems like a reasonable change, but I don't think you're butting up against a thing that most people would run into, as snapshots are enabled by default (auto_snapshot: true) and almost nobody changes it. The use case you described i

Re: Remove folders of deleted tables

2023-12-05 Thread Bowen Song via user
Please rethink your use case. Create and delete tables concurrently often lead to schema disagreement. Even doing so on a single node sequentially will lead to a large number of tombstones in the system tables. On 04/12/2023 19:55, Sébastien Rebecchi wrote: Thank you Dipan. Do you know if the

Re: Remove folders of deleted tables

2023-12-04 Thread Sébastien Rebecchi
Thank you Dipan. Do you know if there is a good reason for Cassandra to let tables folder even when there is no snapshot? I'm thinking of use cases where there is the need to create and delete small tables at a high rate. You could quickly end with more than 65K (limit of ext4) subdirectories in

Re: Remove folders of deleted tables

2023-12-04 Thread Dipan Shah
Hello Sebastien, There are no inbuilt tools that will automatically remove folders of deleted tables. Thanks, Dipan Shah From: Sébastien Rebecchi Sent: 04 December 2023 13:54 To: user@cassandra.apache.org Subject: Remove folders of deleted tables Hello, Wh