+1 to this, and my immediate concern is the possibility that the patch I'm working on for CASSANDRA-16226 <https://issues.apache.org/jira/browse/CASSANDRA-16226> will actually require information about whether a table was *created* with COMPACT STORAGE.
On Thu, Oct 22, 2020 at 2:41 PM David Capwell <dcapw...@gmail.com> wrote: > Thanks for the reply. > > Talked with Alex in Slack and I was under the impression deleted rows were > handled differently on read. > > collect ideas about migration, cutoff date and final deprecation in a > > separate thread or a document. > > > +1 > > I am good bringing a subset back to ease upgrades to 4.0 > > On Thu, Oct 22, 2020 at 11:42 AM Oleksandr Petrov < > oleksandr.pet...@gmail.com> wrote: > > > These are some great (albeit somewhat hard) questions! > > > > > * when you drop compact storage and have some SSTables stored in > > compact storage and some SSTables stored normally, how do we present data > > correctly? > > > > There is no difference in SSTable format between normal and compact > > storage, the difference is only in how Table Metadata is represented. In > > other words, in code. > > > > * does upgradesstables and/or compaction migrate off the compact > > storage format into the normal format? > > > > I think above answers this: the only way to migrate off compact storage > is > > to change table metadata, and accept the differences mentioned above. > > > > * how do we know it is safe to remove? > > > > Maybe if we're explicit about it and say that, for example, by version X > > compact storage will not be supported at all, and document all the > > pitfalls, people will migrate their applications, one at a time. There's > > been a few suggestions about how to mitigate, including client options in > > 10857, and introducing "hiding" columns in 15811, and many more that were > > discussed but not implemented. > > > > Hope this won't discourage further discussions, but I suggest we keep > this > > thread focused on pros/cons of a suggested intermediate solution (bring > > back a minimal subset of CS), and collect ideas about migration, cutoff > > date and final deprecation in a separate thread or a document. I > personally > > do not have a full answer to "when it is safe to remove", and it seems > like > > the best we can do is to create a clear procedure. > > > > On Thu, Oct 22, 2020 at 6:38 PM Ekaterina Dimitrova < > e.dimitr...@gmail.com > > > > > wrote: > > > > > Definitely that is not off the table. > > > Also, we talked this morning for a plan for additional testing to be > > > created as part of CASSANDRA-15588 > > > > > > On Thu, 22 Oct 2020 at 12:33, David Capwell <dcapw...@gmail.com> > wrote: > > > > > > > I am in favor of bringing it back, but I do feel we should also plan > > how > > > to > > > > get it removed as well. > > > > > > > > Few examples, would love to see fleshed out > > > > > > > > * when you drop compact storage and have some SSTables stored in > > compact > > > > storage and some SSTables stored normally, how do we present data > > > > correctly? > > > > * does upgradesstables and/or compaction migrate off the compact > > storage > > > > format into the normal format? > > > > * how do we know it is safe to remove? > > > > > > > > > > > > > > > > On Thu, Oct 22, 2020 at 6:55 AM Ekaterina Dimitrova < > > > e.dimitr...@gmail.com > > > > > > > > > wrote: > > > > > > > > > Hi Alex, > > > > > Thanks for bringing this up. > > > > > I am with you for returning part of the code back and considering > > this > > > > as a > > > > > bug. > > > > > I truly believe it is too late in the release to document changed > > > > behavior. > > > > > I think this contradicts with the project’s promise for no breaking > > > > > changes. This should have been documented in alpha. > > > > > Best regards, > > > > > Ekaterina > > > > > > > > > > On Thu, 22 Oct 2020 at 3:20, Oleksandr Petrov < > > > > oleksandr.pet...@gmail.com> > > > > > wrote: > > > > > > > > > > > Since this is an important subject, I thought it also makes sense > > to > > > > > start > > > > > > a mailing list thread. > > > > > > > > > > > > You may know that in 4.0 there was a plan to drop compact storage > > and > > > > > > related code. However, there are several behavioural changes > > related > > > to > > > > > > compact storage, and difference in visible behaviour between > > "normal" > > > > and > > > > > > compact tables are larger than most of us have anticipated: we > > first > > > > > > thought there’ll be only “appearing column” in dense case, but > > > there’s > > > > > > implicit nulls in clusterings thing, and row vs column deletion > > now, > > > > TTL, > > > > > > and more. > > > > > > > > > > > > Some of the recent issues on the subject are: CASSANDRA-16048 > > > > > > <https://issues.apache.org/jira/browse/CASSANDRA-16048>, which > > > allows > > > > to > > > > > > ignore these differences. The other one was an attempt to improve > > the > > > > > user > > > > > > experience of anyone still using compact storage: CASSANDRA-15811 > > > > > > <https://issues.apache.org/jira/browse/CASSANDRA-15811>. > > > > > > > > > > > > Easily reproducible differences are: > > > > > > > > > > > > (1) hidden columns show up, which breaks SELECT * queries > > > > > > (2) DELETE v and UPDATE v WITH TTL would result into row removals > > in > > > > > > non-dense compact tables (CASSANDRA-16069 > > > > > > <https://issues.apache.org/jira/browse/CASSANDRA-16069>) > > > > > > (3) INSERT allows skipping clusterings, which are filled with > nulls > > > by > > > > > > default. > > > > > > > > > > > > Some of these are tricky to support, as 15811 has shown. Anyone > who > > > > might > > > > > > want to upgrade to 4.0 while still using compact storage might be > > > > > affected > > > > > > by being forced into one of these behaviours. > > > > > > > > > > > > Possible solutions are to document these behaviours, or to bring > > > back a > > > > > > minimal set of COMPACT STORAGE and keep supporting these in 4.0 > > > > > > > > > > > > It looks like it is possible to leave some of the functionality > > > related > > > > > to > > > > > > DENSE flag and allow it to be present in 4.0, but only for these > > > three > > > > > (and > > > > > > potential related, however not directly visible) cases. > > > > > > > > > > > > You can find more details on the subject here: > > > > > > https://issues.apache.org/jira/browse/CASSANDRA-16217 > > > > > > > > > > > > Thank you, > > > > > > > > > > > > -- Alex > > > > > > > > > > > > > > > > > > > > > > > > -- > > alex p > > >
