Hello there! I was wondering if anyone (perhaps an early developer or power-user of Kafka Streams) knows why the Streams developers made the default setting for RocksDB compaction "Universal" compaction rather than "Level" compaction?
My understanding (in which I am extremely UNconfident) is as follows— Supposedly Universal compaction leads to lower write amplification after compaction finishes. In a run of Universal compaction, all data is compacted; as per the RocksDB documentation it is possible for temporary write amplification of up to 2x during this process. There have also been reports of "write stalls" during this process [1]. In Level compaction, only certain levels (tiers of SST files) are compacted at once, meaning that the compaction process is shorter and less intensive, but that write amplification after compaction finishes is higher than with universal compaction. Can anyone confirm/deny/correct this? [1] https://github.com/solana-labs/solana/issues/14586 (not Streams-related, but it is RocksDB) Thanks in advance, Colt McNealy *Founder, LittleHorse.dev*
