Hi Chia-Ping Tsai, Thanks for the feedback.
chia_03: The difficulty with adding a header length is deciding between a fixed-size field for all records or a configuration allowing users to define a maximum. Alternatively, we could consider using a varint for the header length to remain flexible and space-efficient. chia_04: It only makes sense to give the second column family its own RocksDB config if its access pattern or data characteristics are materially different. Here we have the same keys, the same or very similar read/write patterns (e.g., same get, put, range queries), and roughly comparable value sizes (CF2 slightly larger per entry). Then from RocksDB’s perspective the two CFs behave very similarly: both are generic key–value blobs, written and read with the same pattern. Most of the important RocksDB options (compaction style, write buffer sizes, block cache, bloom filters, etc.) would be tuned the same way for both. Do you see huge difference between these two? Thanks, Alieh On Thu, Jan 15, 2026 at 3:03 AM Chia-Ping Tsai <[email protected]> wrote: > hi > > chia_03: should we provide a more effective way to load the value without > scanning the header bytes? (e.g., by storing the total size of headers) > > chia_04: Do we need to allow separate Rocksdb configuration for the new > column family > > Best, > Chia-Ping > > On 2026/01/09 22:14:18 Alieh Saeedi via dev wrote: > > Hi all, > > > > I’d like to start a discussion on KIP-1271, which proposes allowing Kafka > > Streams state stores to preserve record headers. > > This would let header-based metadata like schema IDs, tracing info, and > > feature flags be stored and restored alongside values. > > The KIP introduces header-aware store types and a small config to cap the > > size of headers written into state. > > Details are in the KIP: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1271%3A+Allow+to+Store+Record+Headers+in+State+Stores > > . > > I’d appreciate your feedback and questions on the proposal. > > > > Thanks, > > Alieh > > >
