Hi Bowen, Thanks for your comments. I'll add catalog details as you suggested.
One more question: since we decide to not implement source part of the connector at the moment. What can users do with a Pulsar catalog? Create a table backed by Pulsar and check existing pulsar tables to see their schemas? Drop tables maybe? Best, Yijie On Thu, Oct 17, 2019 at 1:04 AM Bowen Li <bowenl...@gmail.com> wrote: > Hi Yijie, > > Per the discussion, maybe you can move pulsar source to 'future work' > section in the FLIP for now? > > Besides, the FLIP seems to be quite rough at the moment, and I'd recommend > to add more details . > > A few questions mainly regarding the proposed pulsar catalog. > > - Can you provide some background of pulsar schema registry and how it > works? > - The proposed design of pulsar catalog is very vague now, can you > share some details of how a pulsar catalog would work internally? E.g. > - which APIs does it support exactly? E.g. I see from your > prototype that table creation is supported but not alteration. > - is it going to connect to a pulsar schema registry via a http > client or a pulsar client, etc > - will it be able to handle multiple versions of pulsar, or just > one? How is compatibility handles between different Flink-Pulsar > versions? > - will it support only reading from pulsar schema registry , or > both read/write? Will it work end-to-end in Flink SQL for users to > create > and manipulate a pulsar table such as "CREATE TABLE t WITH > PROPERTIES(type=pulsar)" and "DROP TABLE t"? > - Is a pulsar topic always gonna be a non-partitioned table? How is > a partitioned topic mapped to a Flink table? > - How to map Flink's catalog/database namespace to pulsar's > multi-tenant namespaces? I'm not very familiar with how multi tenancy works > in pulsar, and some background context/use cases may help here too. E.g. > - can a pulsar client/consumer/producer be multiple-tenant at the > same time? > - how does authentication work in pulsar's multi-tenancy and the > catalog? asking since I didn't see the proposed pulsar catalog has > username/password configs > - the FLIP seems propose mapping a pulsar cluster and > 'tenant/namespace' respectively to Flink's 'catalog' and 'database'. I > wonder whether it totally makes sense, or should we actually map > "tenant" > to "catalog", and "namespace" to "database"? > > Cheers, > Bowen > > On Fri, Sep 20, 2019 at 1:16 AM Yijie Shen <henry.yijies...@gmail.com> > wrote: > >> Hi everyone, >> >> Per discussion in the previous thread >> < >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Contribute-Pulsar-Flink-connector-back-to-Flink-tc32538.html >> >, >> I have created FLIP-72 to kick off a more detailed discussion on the Flink >> Pulsar connector: >> >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-72%3A+Introduce+Pulsar+Connector >> >> In short, the connector has the following features: >> >> - >> >> Pulsar as a streaming source with exactly-once guarantee. >> - >> >> Sink streaming results to Pulsar with at-least-once semantics. >> - >> >> Build upon Flink new Table API Type system (FLIP-37 >> < >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System >> > >> ), and can automatically (de)serialize messages with the help of Pulsar >> schema. >> - >> >> Integrate with Flink new Catalog API (FLIP-30 >> < >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs >> >), >> which enables the use of Pulsar topics as tables in Table API as well >> as >> SQL client. >> >> >> >> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#heading=h.28v5v23yeq1u >> >> >> Would love to here your thoughts on this. >> >> Best, >> Yijie >> >
