Re: RocksDB state store disk space estimation

Thanks Bruno. On Thu, Feb 18, 2021 at 9:19 AM Bruno Cadonna wrote: > Hi Chris, > > your estimation looks correct to me. > > I do not know how big M might be. Maybe the following link can help you > with the estimation: > > https://github.com/facebook/rocksdb/wiki/Rocksdb-BlockBasedTable-Format >

Re: Window Store

Hmmm, thanks Navneeth, I feel like a session store set to an inactivity period of 10 minutes, suppressed until session window closed, combined with a GlobalKTable would be how I'd start to approach this in the DSL, with the below topology. I have no idea if my ASCII art below will survive email fo

Re: Abort transaction semantics

Thanks for the question. I think Gary provided an excellent answer. Additionally, you could check out the code example for EOS, which shows you how to reset the state while abort

Re: Abort transaction semantics

You have to perform seeks (using the consumer) to the lowest unprocessed offset for each partition returned by the poll, before the next poll. From: Peter Cipov Sent: Thursday, February 18, 2021 1:20 PM To: users@kafka.apache.org Subject: Abort transaction semant

Abort transaction semantics

Hello I have a question regarding aborting transactions in kafka client 2.4.1. lets have following code : ... propper transaction producer consumer creation, consumer autocommit = false producer.transactionInit(); while(true) { records = consumer.poll(); logRecordOffsets(records) producer

Re: RocksDB state store disk space estimation

Hi Chris, your estimation looks correct to me. I do not know how big M might be. Maybe the following link can help you with the estimation: https://github.com/facebook/rocksdb/wiki/Rocksdb-BlockBasedTable-Format There are also some additional files that RocksDB keeps in its directory. I gue

RocksDB state store disk space estimation

We're using RocksDB as a persistent Kafka state store for compacted topics and need to be able to estimate the maximum disk space required. We're using the default config. settings provided by Kafka, which include Universal compaction, no compression, and 4k block size. Given these settings and a

[Spark SQL] - Not able to consume Kafka topics

Hello, Issues : * I and my team are trying to consume some kafka topics based on the timestamps using startingOffsetsByTimestamps option, and the code works fine when we run via a Databricks notebook. * There is a need to setup the whole process in a local system (IntelliJ), but the sa