These are some great (albeit somewhat hard) questions! > * when you drop compact storage and have some SSTables stored in compact storage and some SSTables stored normally, how do we present data correctly?
There is no difference in SSTable format between normal and compact storage, the difference is only in how Table Metadata is represented. In other words, in code. * does upgradesstables and/or compaction migrate off the compact storage format into the normal format? I think above answers this: the only way to migrate off compact storage is to change table metadata, and accept the differences mentioned above. * how do we know it is safe to remove? Maybe if we're explicit about it and say that, for example, by version X compact storage will not be supported at all, and document all the pitfalls, people will migrate their applications, one at a time. There's been a few suggestions about how to mitigate, including client options in 10857, and introducing "hiding" columns in 15811, and many more that were discussed but not implemented. Hope this won't discourage further discussions, but I suggest we keep this thread focused on pros/cons of a suggested intermediate solution (bring back a minimal subset of CS), and collect ideas about migration, cutoff date and final deprecation in a separate thread or a document. I personally do not have a full answer to "when it is safe to remove", and it seems like the best we can do is to create a clear procedure. On Thu, Oct 22, 2020 at 6:38 PM Ekaterina Dimitrova <e.dimitr...@gmail.com> wrote: > Definitely that is not off the table. > Also, we talked this morning for a plan for additional testing to be > created as part of CASSANDRA-15588 > > On Thu, 22 Oct 2020 at 12:33, David Capwell <dcapw...@gmail.com> wrote: > > > I am in favor of bringing it back, but I do feel we should also plan how > to > > get it removed as well. > > > > Few examples, would love to see fleshed out > > > > * when you drop compact storage and have some SSTables stored in compact > > storage and some SSTables stored normally, how do we present data > > correctly? > > * does upgradesstables and/or compaction migrate off the compact storage > > format into the normal format? > > * how do we know it is safe to remove? > > > > > > > > On Thu, Oct 22, 2020 at 6:55 AM Ekaterina Dimitrova < > e.dimitr...@gmail.com > > > > > wrote: > > > > > Hi Alex, > > > Thanks for bringing this up. > > > I am with you for returning part of the code back and considering this > > as a > > > bug. > > > I truly believe it is too late in the release to document changed > > behavior. > > > I think this contradicts with the project’s promise for no breaking > > > changes. This should have been documented in alpha. > > > Best regards, > > > Ekaterina > > > > > > On Thu, 22 Oct 2020 at 3:20, Oleksandr Petrov < > > oleksandr.pet...@gmail.com> > > > wrote: > > > > > > > Since this is an important subject, I thought it also makes sense to > > > start > > > > a mailing list thread. > > > > > > > > You may know that in 4.0 there was a plan to drop compact storage and > > > > related code. However, there are several behavioural changes related > to > > > > compact storage, and difference in visible behaviour between "normal" > > and > > > > compact tables are larger than most of us have anticipated: we first > > > > thought there’ll be only “appearing column” in dense case, but > there’s > > > > implicit nulls in clusterings thing, and row vs column deletion now, > > TTL, > > > > and more. > > > > > > > > Some of the recent issues on the subject are: CASSANDRA-16048 > > > > <https://issues.apache.org/jira/browse/CASSANDRA-16048>, which > allows > > to > > > > ignore these differences. The other one was an attempt to improve the > > > user > > > > experience of anyone still using compact storage: CASSANDRA-15811 > > > > <https://issues.apache.org/jira/browse/CASSANDRA-15811>. > > > > > > > > Easily reproducible differences are: > > > > > > > > (1) hidden columns show up, which breaks SELECT * queries > > > > (2) DELETE v and UPDATE v WITH TTL would result into row removals in > > > > non-dense compact tables (CASSANDRA-16069 > > > > <https://issues.apache.org/jira/browse/CASSANDRA-16069>) > > > > (3) INSERT allows skipping clusterings, which are filled with nulls > by > > > > default. > > > > > > > > Some of these are tricky to support, as 15811 has shown. Anyone who > > might > > > > want to upgrade to 4.0 while still using compact storage might be > > > affected > > > > by being forced into one of these behaviours. > > > > > > > > Possible solutions are to document these behaviours, or to bring > back a > > > > minimal set of COMPACT STORAGE and keep supporting these in 4.0 > > > > > > > > It looks like it is possible to leave some of the functionality > related > > > to > > > > DENSE flag and allow it to be present in 4.0, but only for these > three > > > (and > > > > potential related, however not directly visible) cases. > > > > > > > > You can find more details on the subject here: > > > > https://issues.apache.org/jira/browse/CASSANDRA-16217 > > > > > > > > Thank you, > > > > > > > > -- Alex > > > > > > > > > > -- alex p
