Hi, Currently we have an aggregation system (without kafka) where events are aggregated into Cassandra tables holding aggregate results. We are considering moving to a KafkaStreams solution with exactly-once processing but in this case it seems that all the aggregation tables (reaching TB) need to be kept also in Kafka as ktables(Rocksdb)+compacted topics(Kafka) and the direction of computation would be: events topic -> KS aggregation -> aggregated topics -> one way sync to Cassandra using connector. This poses two problems: - the doubling on the total storage required for the system (which mainly stores aggregates), from 3 C* replicas to 2-3 K replicas + Rocksdb - the time to reconstruct a Rocksdb instance during rollover update can be half an hour if the rollover is fast
Is there any way in Kafka Streams (even dropping the exactly once) in which we can work just with aggregate tables in Cassandra? For sure there is a way working with Kafka Consumer, pulling a batch of messages, aggregating, and adding aggregate to Cassandra. Not sure if possible with KafkaStreams given the higher level / FP modeling with its own clear advantages but this disadvantage. Please advise, Nicu Marasoiu Geschäftsanschrift/Business address: METRO SYSTEMS GmbH, Metro-Straße 12, 40235 Düsseldorf, Germany Aufsichtsrat/Supervisory Board: Heiko Hutmacher (Vorsitzender/ Chairman) Geschäftsführung/Management Board: Dr. Dirk Toepfer (Vorsitzender/CEO), Wim van Herwijnen Sitz Düsseldorf, Amtsgericht Düsseldorf, HRB 18232/Registered Office Düsseldorf, Commercial Register of the Düsseldorf Local Court, HRB 18232 Betreffend Mails von *@metrosystems.net Die in dieser E-Mail enthaltenen Nachrichten und Anhänge sind ausschließlich für den bezeichneten Adressaten bestimmt. Sie können rechtlich geschützte, vertrauliche Informationen enthalten. Falls Sie nicht der bezeichnete Empfänger oder zum Empfang dieser E-Mail nicht berechtigt sind, ist die Verwendung, Vervielfältigung oder Weitergabe der Nachrichten und Anhänge untersagt. Falls Sie diese E-Mail irrtümlich erhalten haben, informieren Sie bitte unverzüglich den Absender und vernichten Sie die E-Mail. Regarding mails from *@metrosystems.net This e-mail message and any attachment are intended exclusively for the named addressee. They may contain confidential information which may also be protected by professional secrecy. Unless you are the named addressee (or authorised to receive for the addressee) you may not copy or use this message or any attachment or disclose the contents to anyone else. If this e-mail was sent to you by mistake please notify the sender immediately and delete this e-mail.
