First of all, thx for all the ideas. Benedict ElIiott Smith, in code comments I found a notice that data in EncodingStats can be wrong, not sure that its good idea to use it for accurate results. As I understand incorrect data is not a problem for the current use case of it, but not for my one. Currently, I added fields for every AtomicBTreePartition. That fields I update in addAllWithSizeDelta call, but also now I get that I should think about the case of data removing.
I currently don't really care about TTL's, but its the case about I should think, thx. Jeremiah Jordan, thx for notice, but I don't really get what are you mean about replica aggregation optimizationâs. Can you please explain it for me? On 2018-01-14 17:16, Benedict Elliott Smith <bened...@apache.org> wrote: > (Obviously, not to detract from the points that Jon and Jeremiah make, i.e. > that if TTLs or tombstones are involved the metadata we have, or can add, > is going to be worthless in most cases anyway) > > On 14 January 2018 at 16:11, Benedict Elliott Smith <bened...@apache.org> > wrote: > > > We already store the minimum timestamp in the EncodingStats of each > > partition, to support more efficient encoding of atom timestamps. This > > just isn't exposed beyond UnfilteredRowIterator, though it probably could > > be. > > > > Storing the max alongside would still require justification, though its > > cost would actually be fairly nominal (probably only a few bytes; it > > depends on how far apart min/max are). > > > > I'm not sure (IMO) that even a fairly nominal cost could be justified > > unless there were widespread benefit though, which I'm not sure this would > > provide. Maintaining a patched variant of your own that stores this > > probably wouldn't be too hard, though. > > > > In the meantime, exposing and utilising the minimum timestamp from > > EncodingStats is probably a good place to start to explore the viability of > > the approach. > > > > On 14 January 2018 at 15:34, Jeremiah Jordan <jerem...@datastax.com> > > wrote: > > > >> Donât forget about deleted and missing data. The bane of all on replica > >> aggregation optimizationâs. > >> > >> > On Jan 14, 2018, at 12:07 AM, Jeff Jirsa <jji...@gmail.com> wrote: > >> > > >> > > >> > Youâre right itâs not stored in metadata now. Adding this to metadata > >> isnât hard, itâs just hard to do it right where itâs useful to > >> people with > >> other data models (besides yours) so it can make it upstream (if thatâs > >> your goal). In particular the worst possible case is a table with no > >> clustering key and a single non-partition key column. In that case storing > >> these extra two long time stamps may be 2-3x more storage than without, > >> which would be a huge regression, so youâd have to have a way to turn > >> that > >> feature off. > >> > > >> > > >> > Worth mentioning that there are ways to do this without altering > >> Cassandra - consider using static columns that represent the min timestamp > >> and max timestamp. Create them both as ints or longs and write them on all > >> inserts/updates (as part of a batch, if needed). The only thing youâll > >> have > >> to do is find a way for âmin timestampâ to work - you can set the min > >> time > >> stamp column with an explicit âusing timestampâ timestamp = 2^31-NOW, > >> so > >> that future writes wonât overwrite those values. That gives you a first > >> write win behavior for that column, which gives you an effective min > >> timestamp for the partition as a whole. > >> > > >> > -- > >> > Jeff Jirsa > >> > > >> > > >> >> On Jan 13, 2018, at 4:58 AM, Arthur Kushka <arhel...@gmail.com> wrote: > >> >> > >> >> Hi folks, > >> >> > >> >> Currently, I working on custom CQL operator that should return the max > >> >> timestamp for some partition. > >> >> > >> >> I don't think that scanning of partition for that kind of data is a > >> nice > >> >> idea. Instead of it, I thinking about adding a metadata to the > >> partition. I > >> >> want to store minTimestamp and maxTimestamp for every partition as it > >> >> already done in Memtable`s. That timestamps will be updated on each > >> >> mutation operation, that is quite cheap in comparison to full scan. > >> >> > >> >> I quite new to Cassandra codebase and want to get some critics and > >> ideas, > >> >> maybe that kind of data already stored somewhere or you have better > >> ideas. > >> >> Is my assumption right? > >> >> > >> >> Best, > >> >> Artur > >> > > >> > --------------------------------------------------------------------- > >> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > >> > For additional commands, e-mail: dev-h...@cassandra.apache.org > >> > > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > >> For additional commands, e-mail: dev-h...@cassandra.apache.org > >> > >> > > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
