On Fri, Nov 15, 2019 at 11:04 AM Jun Rao <j...@confluent.io> wrote: > Hi, Satish and Harsha, > > The following is a more detailed high level feedback for the KIP. Overall, > the KIP seems useful. The challenge is how to design it such that it’s > general enough to support different ways of implementing this feature and > support existing features. > > 40. Local segment metadata storage: The KIP makes the assumption that the > metadata for the archived log segments are cached locally in every broker > and provides a specific implementation for the local storage in the > framework. We probably should discuss this more. For example, some tier > storage providers may not want to cache the metadata locally and just rely > upon a remote key/value store if such a store is already present. If a > local store is used, there could be different ways of implementing it > (e.g., based on customized local files, an embedded local store like > RocksDB, etc). An alternative of designing this is to just provide an > interface for retrieving the tier segment metadata and leave the details of > how to get the metadata outside of the framework. >
[Ying] Early this year, when we just started design tiered storage, we did plan to make RemoteLogManager a Kafka plugin. So that, there can be totally different implementations of tiered storage. However, one feedback we received from the community is that developing RemoteLogManager implementations are too hard for most potential users. People actually prefer to one standard implementation that can satisfy most of the requirements. We accepted that feedback, and decided to trade some of the flexibility for simplicity in the 1st version. It's still possible to allow users provide different implementations in the future. We had discussions with different companies (e.g. Slack, AirBnb) that are interested in tiered storage. Our conclusion is that the current design (a standard RemoteLogManager that caches remote metadata locally + HDFS and S3 RemoteStorageMangers) is good enough for all of the companies we have talked with. We don't have much knowledge about the use-cases out of Internet industry. Do any consumers of Confluent need to manage the metadata in different ways? > 43. There are quite a few existing features not covered by the KIP. It > would be useful to discuss each of those. > 43.1 I won’t say that compacted topics are rarely used and always small. > For example, KStreams uses compacted topics for storing the states and > sometimes the size of the topic could be large. While it might be ok to not > support compacted topics initially, it would be useful to have a high level > idea on how this might be supported down the road so that we don’t have to > make incompatible API changes in the future. > 43.2 We need to discuss how EOS is supported. In particular, how is the > producer state integrated with the remote storage. > 43.3 Now that KIP-392 (allow consumers to fetch from closest replica) is > implemented, we need to discuss how reading from a follower replica is > supported with tier storage. > 43.4 We need to discuss how JBOD is supported with tier storage. > > [Ying] The support of compacted topics and EOS are definitely possible. We will discuss the possible design in the KIP. But for the 1st version, we prefer to focus on a relatively small scope, and develop a simple and just enough solution for most users. Most features will be gradually added in the future releases. For compacted topic, we can save a new version of remote segment files after each compact. The old remote version will be deleted after the new version is available on remote storage. For EOS, the snapshots can also be shipped to remote storage. KIP-392 will be supported in the 1st version of tiered storage. We will add the design details in the KIP. JBOD of in remote storage is provided by the remote storage system (e.g. HDFS, S3). This should be totally transparent for Kafka. Tiered storage will make Kafka local storage much smaller, and make JBOD of local storage less needed. We should be able to support JBOD in local storage in the future. This shouldn't require any changes in RSM, because only Kafka and RemoteLogManager talk with local storage. So, there shouldn't be any compatibility issue when we support local storage JBOD in the next version. > Thanks, > > Jun > > >
