Hi Sumant, We are tracking HDFS and S3 implementations as part of this project to see the proposed APIs are addressing file/object stores. As Jun mentioned in the earlier mail thread, no RSM implementation will be part of Apache Kafka. This is in line with Kafka Connectors development. Devs/Users can build their own implementations and host them to share with others.
We have HDFS/S3 RSMs which can be treated as reference implementations. These can be referenced to build your own implementations if needed. Thanks, Satish. On Tue, 13 Jul 2021 at 02:25, Sumant Tambe <suta...@gmail.com> wrote: > > Hi Israel, > > Linkedin is interested in evaluating KIP-405 for HDFS and S3 in the short > term and Azure Blob Storage in the long run. You may already know that > Linkedin > is migrating to Azure > <https://engineering.linkedin.com/blog/2019/building-next-infra>. We think > that Blobs will provide us with the optimal cost/availability/operability > trade-offs for Kafka in Azure. > What's the context behind your interest in KIP-405 and Azure Blobs? Do you > have any data/experience of using Azure blobs at scale? > > Satish, any word on the KIP-405 and the RSM implementations? > > Regards, > Sumant > Kafka Dev, Linkedin > > On Mon, 15 Mar 2021 at 09:08, Satish Duggana <satish.dugg...@gmail.com> > wrote: > > > Hi Israel, > > Thanks for your interest in tiered storage. As mentioned by Jun earlier, we > > decided not to have any implementations in Apache Kafka repo like Kafka > > connectors. We plan to have RSM implementations for HDFS, S3, GCP, and > > Azure storages in a separate repo. We will let you know once they are ready > > for review. > > > > Best, > > Satish. > > > > On Sat, 13 Mar 2021 at 01:27, Israel Ekpo <israele...@gmail.com> wrote: > > > > > Thanks @Jun for the prompt response. > > > > > > That's ok and I think it is a great strategy just like the Connect > > > ecosystem. > > > > > > However, I am still in search for repos and samples that demonstrate > > > implementation for the KIP. > > > > > > I will keep searching but was just wondering if there were sample > > > implementations for S3 or HDFS I could take a look at. > > > > > > Thanks. > > > > > > On Fri, Mar 12, 2021 at 2:19 PM Jun Rao <j...@confluent.io.invalid> > > wrote: > > > > > > > Hi, Israel, > > > > > > > > Thanks for your interest. As part of KIP-405, we have made the decision > > > not > > > > to host any plugins for external remote storage directly in Apache > > Kafka. > > > > Those plugins could be hosted outside of Apache Kafka. > > > > > > > > Jun > > > > > > > > On Thu, Mar 11, 2021 at 5:15 PM Israel Ekpo <israele...@gmail.com> > > > wrote: > > > > > > > > > Thanks Satish, Sriharsha, Suresh and Ying for authoring this KIP and > > > > thanks > > > > > to everyone that participated in the review and discussion to take it > > > to > > > > > where it is today. > > > > > > > > > > I would like to contribute by working on integrating Azure Storage > > > (Blob > > > > > and ADLS) with Tiered Storage for this KIP > > > > > > > > > > I have created this issue to track this work > > > > > https://issues.apache.org/jira/browse/KAFKA-12458 > > > > > > > > > > Are there any sample implementations for HDFS/S3 that I can reference > > > to > > > > > get started? > > > > > > > > > > When you have a moment, please share. > > > > > > > > > > Thanks. > > > > > > > > > > > > > >
