Re: Challenge with initial data load with TWCS

2019-10-01 Thread DuyHai Doan
Thanks Alex for confirming Le 30 sept. 2019 09:17, "Oleksandr Shulgin" a écrit : > On Sun, Sep 29, 2019 at 9:42 AM DuyHai Doan wrote: > >> Thanks Jeff for sharing the ideas. I have some question though: >> >> - CQLSSTableWriter and explicitly break between windows --> Even if >> you break betwe

Re: Challenge with initial data load with TWCS

2019-09-30 Thread Oleksandr Shulgin
On Sun, Sep 29, 2019 at 9:42 AM DuyHai Doan wrote: > Thanks Jeff for sharing the ideas. I have some question though: > > - CQLSSTableWriter and explicitly break between windows --> Even if > you break between windows, If we have data worth of 1 years it would > requires us to use CQLSSTableWriter

Re: Challenge with initial data load with TWCS

2019-09-29 Thread DuyHai Doan
Thanks Jeff for sharing the ideas. I have some question though: - CQLSSTableWriter and explicitly break between windows --> Even if you break between windows, If we have data worth of 1 years it would requires us to use CQLSSTableWriter during 1 year (365 days) because the write time taken into ac

Re: Challenge with initial data load with TWCS

2019-09-28 Thread Jeff Jirsa
We used to do either: - CQLSSTableWriter and explicitly break between windows (then nodetool refresh or sstableloader to push them into the system), or - Use the normal write path for a single window at a time, explicitly calling flush between windows. You can’t have current data writing whi

Challenge with initial data load with TWCS

2019-09-28 Thread DuyHai Doan
Hello users TWCS works great for permanent state. It creates SSTables of roughly fixed size if your insertion rate is pretty constant. Now the big deal is about the initial load. Let's say we configure a TWCS with window unit = day and window size = 1, we would have 1 SSTable per day and with TT