chia_03: Regarding the header size, using a Varint is consistent with Kafka's serialization standards. It avoids the overhead of a large fixed-size field while still achieving the efficient skipping capability we want.
chia_04: That makes sense. Alieh Saeedi via dev <[email protected]> 於 2026年1月15日週四 下午10:59寫道： > Hi Chia-Ping Tsai, > > Thanks for the feedback. > > chia_03: The difficulty with adding a header length is deciding between a > fixed-size field for all records or a configuration allowing users to > define a maximum. Alternatively, we could consider using a varint for the > header length to remain flexible and space-efficient. > > chia_04: > It only makes sense to give the second column family its own RocksDB > config if its access pattern or data characteristics are materially > different. > Here we have the same keys, the > same or very similar read/write patterns (e.g., same get, put, range > queries), > and roughly comparable value sizes (CF2 slightly larger per entry). > Then from RocksDB’s perspective the two CFs behave very similarly: > both are generic key–value blobs, written and read with the same > pattern. Most of the important RocksDB options (compaction style, > write buffer sizes, block cache, bloom filters, etc.) would be tuned > the same way for both. > Do you see huge difference between these two? > > Thanks, > Alieh > > On Thu, Jan 15, 2026 at 3:03 AM Chia-Ping Tsai <[email protected]> > wrote: > > > hi > > > > chia_03: should we provide a more effective way to load the value without > > scanning the header bytes? (e.g., by storing the total size of headers) > > > > chia_04: Do we need to allow separate Rocksdb configuration for the new > > column family > > > > Best, > > Chia-Ping > > > > On 2026/01/09 22:14:18 Alieh Saeedi via dev wrote: > > > Hi all, > > > > > > I’d like to start a discussion on KIP-1271, which proposes allowing > Kafka > > > Streams state stores to preserve record headers. > > > This would let header-based metadata like schema IDs, tracing info, and > > > feature flags be stored and restored alongside values. > > > The KIP introduces header-aware store types and a small config to cap > the > > > size of headers written into state. > > > Details are in the KIP: > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1271%3A+Allow+to+Store+Record+Headers+in+State+Stores > > > . > > > I’d appreciate your feedback and questions on the proposal. > > > > > > Thanks, > > > Alieh > > > > > >
