Those are two different cases though. It *sounds like* (again, I may be missing the point) you're trying to overwrite a value with another value. You're either going to serialize a blob and overwrite a single cell, or you're going to overwrite all the cells and include a tombstone.
When you do a read, reading a single tombstone vs a single vs is essentially the same thing, performance wise. In your description you said "~ 20-100 events", and you're overwriting the event each time, so I don't know how you go to 10K tombstones either. Compaction will bring multiple tombstones together for a cell in the same way it compacts multiple values for a single cell. I sounds to make like you're taking some advice about tombstones out of context and trying to apply the advice to a different problem. Again, I might be misunderstanding what you're doing. On Fri, Jan 4, 2019 at 10:49 AM Tomas Bartalos <tomas.barta...@gmail.com> wrote: > Hello Jon, > > I thought having tombstones is much higher overhead than just overwriting > values. The compaction overhead can be l similar, but I think the read > performance is much worse. > > Tombstones accumulate and hang for 10 days (by default) before they are > eligible for compaction. > > Also we have tombstone warning and error thresholds. If cassandra scans > more than 10 000 tombstones, she will abort the query. > > According to this article: > https://opencredo.com/blogs/cassandra-tombstones-common-issues/ > > "The cassandra.yaml comments explain in perfectly: *“When executing a > scan, within or across a partition, we need to keep the tombstones seen in > memory so we can return them to the coordinator, which will use them to > make sure other replicas also know about the deleted rows. With workloads > that generate a lot of tombstones, this can cause performance problems and > even exhaust the server heap. "* > > Regards, > Tomas > > On Fri, 4 Jan 2019, 7:06 pm Jonathan Haddad <j...@jonhaddad.com wrote: > >> If you're overwriting values, it really doesn't matter much if it's a >> tombstone or any other value, they still need to be compacted and have the >> same overhead at read time. >> >> Tombstones are problematic when you try to use Cassandra as a queue (or >> something like a queue) and you need to scan over thousands of tombstones >> in order to get to the real data. You're simply overwriting a row and >> trying to avoid a single tombstone. >> >> Maybe I'm missing something here. Why do you think overwriting a single >> cell with a tombstone is any worse than overwriting a single cell with a >> value? >> >> Jon >> >> >> On Fri, Jan 4, 2019 at 9:57 AM Tomas Bartalos <tomas.barta...@gmail.com> >> wrote: >> >>> Hello, >>> >>> I beleive your approach is the same as using spark with " >>> spark.cassandra.output.ignoreNulls=true" >>> This will not cover the situation when a value have to be overwriten >>> with null. >>> >>> I found one possible solution - change the schema to keep only primary >>> key fields and move all other fields to frozen UDT. >>> create table (year, month, day, id, frozen<Event>, primary key((year, >>> month, day), id) ) >>> In this way anything that is null inside event doesn't create tombstone, >>> since event is serialized to BLOB. >>> The penalty is in need of deserializing the whole Event when selecting >>> only few columns. >>> Can anyone confirm if this is good solution performance wise? >>> >>> Thank you, >>> >>> On Fri, 4 Jan 2019, 2:20 pm DuyHai Doan <doanduy...@gmail.com wrote: >>> >>>> "The problem is I can't know the combination of set/unset values" --> >>>> Just for this requirement, Achilles has a working solution for many years >>>> using INSERT_NOT_NULL_FIELDS strategy: >>>> >>>> https://github.com/doanduyhai/Achilles/wiki/Insert-Strategy >>>> >>>> Or you can use the Update API that by design only perform update on not >>>> null fields: >>>> https://github.com/doanduyhai/Achilles/wiki/Quick-Reference#updating-all-non-null-fields-for-an-entity >>>> >>>> >>>> Behind the scene, for each new combination of INSERT INTO table(x,y,z) >>>> statement, Achilles will check its prepared statement cache and if the >>>> statement does not exist yet, create a new prepared statement and put it >>>> into the cache for later re-use for you >>>> >>>> Disclaiment: I'm the creator of Achilles >>>> >>>> >>>> >>>> On Thu, Dec 27, 2018 at 10:21 PM Tomas Bartalos < >>>> tomas.barta...@gmail.com> wrote: >>>> >>>>> Hello, >>>>> >>>>> The problem is I can't know the combination of set/unset values. From >>>>> my perspective every value should be set. The event from Kafka represents >>>>> the complete state of the happening at certain point in time. In my table >>>>> I >>>>> want to store the latest event so the most recent state of the happening >>>>> (in this table I don't care about the history). Actually I used wrong >>>>> expression since its just the opposite of "incremental update", every >>>>> event >>>>> carries all data (state) for specific point of time. >>>>> >>>>> The event is represented with nested json structure. Top level >>>>> elements of the json are table fields with type like text, boolean, >>>>> timestamp, list and the nested elements are UDT fields. >>>>> >>>>> Simplified example: >>>>> There is a new purchase for the happening, event: >>>>> {total_amount: 50, items : [A, B, C, new_item], purchase_time : >>>>> '2018-12-27 13:30', specials: null, customer : {... }, fare_amount,...} >>>>> I don't know what actually happened for this event, maybe there is a >>>>> new item purchased, maybe some customer info have been changed, maybe the >>>>> specials have been revoked and I have to reset them. I just need to store >>>>> the state as it artived from Kafka, there might already be an event for >>>>> this happening saved before, or maybe this is the first one. >>>>> >>>>> BR, >>>>> Tomas >>>>> >>>>> >>>>> On Thu, 27 Dec 2018, 9:36 pm Eric Stevens <migh...@gmail.com wrote: >>>>> >>>>>> Depending on the use case, creating separate prepared statements for >>>>>> each combination of set / unset values in large INSERT/UPDATE statements >>>>>> may be prohibitive. >>>>>> >>>>>> Instead, you can look into driver level support for UNSET values. >>>>>> Requires Cassandra 2.2 or later IIRC. >>>>>> >>>>>> See: >>>>>> Java Driver: >>>>>> https://docs.datastax.com/en/developer/java-driver/3.0/manual/statements/prepared/#parameters-and-binding >>>>>> Python Driver: >>>>>> https://www.datastax.com/dev/blog/python-driver-2-6-0-rc1-with-cassandra-2-2-features#distinguishing_between_null_and_unset_values >>>>>> Node Driver: >>>>>> https://docs.datastax.com/en/developer/nodejs-driver/3.5/features/datatypes/nulls/#unset >>>>>> >>>>>> On Thu, Dec 27, 2018 at 3:21 PM Durity, Sean R < >>>>>> sean_r_dur...@homedepot.com> wrote: >>>>>> >>>>>>> You say the events are incremental updates. I am interpreting this >>>>>>> to mean only some columns are updated. Others should keep their original >>>>>>> values. >>>>>>> >>>>>>> You are correct that inserting null creates a tombstone. >>>>>>> >>>>>>> Can you only insert the columns that actually have new values? Just >>>>>>> skip the columns with no information. (Make the insert generator a bit >>>>>>> smarter.) >>>>>>> >>>>>>> Create table happening (id text primary key, event text, a text, b >>>>>>> text, c text); >>>>>>> Insert into table happening (id, event, a, b, c) values >>>>>>> ("MainEvent","The most complete info we have right now","Priceless","10 >>>>>>> pm","Grand Ballroom"); >>>>>>> -- b changes >>>>>>> Insert into happening (id, b) values ("MainEvent","9:30 pm"); >>>>>>> >>>>>>> >>>>>>> Sean Durity >>>>>>> >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Tomas Bartalos <tomas.barta...@gmail.com> >>>>>>> Sent: Thursday, December 27, 2018 9:27 AM >>>>>>> To: user@cassandra.apache.org >>>>>>> Subject: [EXTERNAL] Howto avoid tombstones when inserting NULL values >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> I’d start with describing my use case and how I’d like to use >>>>>>> Cassandra to solve my storage needs. >>>>>>> We're processing a stream of events for various happenings. Every >>>>>>> event have a unique happening_id. >>>>>>> One happening may have many events, usually ~ 20-100 events. I’d >>>>>>> like to store only the latest event for the same happening (Event is an >>>>>>> incremental update and it contains all up-to date data about happening). >>>>>>> Technically the events are streamed from Kafka, processed with Spark >>>>>>> an saved to Cassandra. >>>>>>> In Cassandra we use upserts (insert with same primary key). So far >>>>>>> so good, however there comes the tombstone... >>>>>>> >>>>>>> When I’m inserting field with NULL value, Cassandra creates >>>>>>> tombstone for this field. As I understood this is due to space >>>>>>> efficiency, >>>>>>> Cassandra doesn’t have to remember there is a NULL value, she just >>>>>>> deletes >>>>>>> the respective column and a delete creates a ... tombstone. >>>>>>> I was hoping there could be an option to tell Cassandra not to be so >>>>>>> space effective and store “unset" info without generating tombstones. >>>>>>> Something similar to inserting empty strings instead of null values: >>>>>>> >>>>>>> CREATE TABLE happening (id text PRIMARY KEY, event text); insert >>>>>>> into happening (‘1’, ‘event1’); — tombstone is generated insert into >>>>>>> happening (‘1’, null); — tombstone is not generated insert into >>>>>>> happening >>>>>>> (‘1’, '’); >>>>>>> >>>>>>> Possible solutions: >>>>>>> 1. Disable tombstones with gc_grace_seconds = 0 or set to reasonable >>>>>>> low value (1 hour ?) . Not good, since phantom data may re-appear 2. >>>>>>> ignore >>>>>>> NULLs on spark side with “spark.cassandra.output.ignoreNulls=true”. Not >>>>>>> good since this will never overwrite previously inserted event field >>>>>>> with >>>>>>> “empty” one. >>>>>>> 3. On inserts with spark, find all NULL values and replace them with >>>>>>> “empty” equivalent (empty string for text, 0 for integer). Very >>>>>>> inefficient >>>>>>> and problematic to find “empty” equivalent for some data types. >>>>>>> >>>>>>> Until tombstones appeared Cassandra was the right fit for our use >>>>>>> case, however now I’m not sure if we’re heading the right direction. >>>>>>> Could you please give me some advice how to solve this problem ? >>>>>>> >>>>>>> Thank you, >>>>>>> Tomas >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >>>>>>> For additional commands, e-mail: user-h...@cassandra.apache.org >>>>>>> >>>>>>> >>>>>>> ________________________________ >>>>>>> >>>>>>> The information in this Internet Email is confidential and may be >>>>>>> legally privileged. It is intended solely for the addressee. Access to >>>>>>> this >>>>>>> Email by anyone else is unauthorized. If you are not the intended >>>>>>> recipient, any disclosure, copying, distribution or any action taken or >>>>>>> omitted to be taken in reliance on it, is prohibited and may be >>>>>>> unlawful. >>>>>>> When addressed to our clients any opinions or advice contained in this >>>>>>> Email are subject to the terms and conditions expressed in any >>>>>>> applicable >>>>>>> governing The Home Depot terms of business or client engagement letter. >>>>>>> The >>>>>>> Home Depot disclaims all responsibility and liability for the accuracy >>>>>>> and >>>>>>> content of this attachment and for any damages or losses arising from >>>>>>> any >>>>>>> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or >>>>>>> other >>>>>>> items of a destructive nature, which may be contained in this attachment >>>>>>> and shall not be liable for direct, indirect, consequential or special >>>>>>> damages in connection with this e-mail message or its attachment. >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >>>>>>> For additional commands, e-mail: user-h...@cassandra.apache.org >>>>>>> >>>>>> >> >> -- >> Jon Haddad >> http://www.rustyrazorblade.com >> twitter: rustyrazorblade >> > -- Jon Haddad http://www.rustyrazorblade.com twitter: rustyrazorblade
