Hi Yijie,
Per the discussion, maybe you can move pulsar source to 'future work'
section in the FLIP for now?
Besides, the FLIP seems to be quite rough at the moment, and I'd recommend
to add more details .
A few questions mainly regarding the proposed pulsar catalog.
- Can you provide some background of pulsar schema registry and how it
works?
- The proposed design of pulsar catalog is very vague now, can you share
some details of how a pulsar catalog would work internally? E.g.
- which APIs does it support exactly? E.g. I see from your prototype
that table creation is supported but not alteration.
- is it going to connect to a pulsar schema registry via a http
client or a pulsar client, etc
- will it be able to handle multiple versions of pulsar, or just one?
How is compatibility handles between different Flink-Pulsar versions?
- will it support only reading from pulsar schema registry , or both
read/write? Will it work end-to-end in Flink SQL for users to create and
manipulate a pulsar table such as "CREATE TABLE t WITH
PROPERTIES(type=pulsar)" and "DROP TABLE t"?
- Is a pulsar topic always gonna be a non-partitioned table? How is a
partitioned topic mapped to a Flink table?
- How to map Flink's catalog/database namespace to pulsar's multi-tenant
namespaces? I'm not very familiar with how multi tenancy works in pulsar,
and some background context/use cases may help here too. E.g.
- can a pulsar client/consumer/producer be multiple-tenant at the
same time?
- how does authentication work in pulsar's multi-tenancy and the
catalog? asking since I didn't see the proposed pulsar catalog has
username/password configs
- the FLIP seems propose mapping a pulsar cluster and
'tenant/namespace' respectively to Flink's 'catalog' and 'database'. I
wonder whether it totally makes sense, or should we actually map "tenant"
to "catalog", and "namespace" to "database"?
Cheers,
Bowen
On Fri, Sep 20, 2019 at 1:16 AM Yijie Shen <henry.yijies...@gmail.com>
wrote:
> Hi everyone,
>
> Per discussion in the previous thread
> <
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Contribute-Pulsar-Flink-connector-back-to-Flink-tc32538.html
> >,
> I have created FLIP-72 to kick off a more detailed discussion on the Flink
> Pulsar connector:
>
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-72%3A+Introduce+Pulsar+Connector
>
> In short, the connector has the following features:
>
> -
>
> Pulsar as a streaming source with exactly-once guarantee.
> -
>
> Sink streaming results to Pulsar with at-least-once semantics.
> -
>
> Build upon Flink new Table API Type system (FLIP-37
> <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> >
> ), and can automatically (de)serialize messages with the help of Pulsar
> schema.
> -
>
> Integrate with Flink new Catalog API (FLIP-30
> <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> >),
> which enables the use of Pulsar topics as tables in Table API as well as
> SQL client.
>
>
>
> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#heading=h.28v5v23yeq1u
>
>
> Would love to here your thoughts on this.
>
> Best,
> Yijie
>
