Hi everyone,

I've updated the Catalog PR and make all settings small case. And tests are
added as well.
Hi Bowen, could you please take a look.
https://github.com/apache/flink/pull/10455

For the sink part of the connector, I've made a separate PR
https://github.com/apache/flink/pull/10875. Could someone help review this?


Best,
Yijie


On Thu, Jan 9, 2020 at 8:44 AM Bowen Li <bowenl...@gmail.com> wrote:

> Hi Yijie,
>
> There's just one more concern on the yaml configs. Otherwise, I think we
> should be good to go.
>
> Can you update your PR and ensure all tests pass? I can help review and
> merge in the next couple weeks.
>
> Thanks,
> Bowen
>
>
> On Mon, Dec 23, 2019 at 7:03 PM Yijie Shen <henry.yijies...@gmail.com>
> wrote:
>
> > Hi Bowen,
> >
> > I've done updated the design doc, PTAL.
> > Btw the PR for catalog is https://github.com/apache/flink/pull/10455,
> > could
> > you please take a look?
> >
> > Best,
> > Yijie
> >
> > On Mon, Dec 9, 2019 at 8:44 AM Bowen Li <bowenl...@gmail.com> wrote:
> >
> > > Hi Yijie,
> > >
> > > I took a look at the design doc. LGTM overall, left a few questions.
> > >
> > > On Tue, Dec 3, 2019 at 10:39 PM Becket Qin <becket....@gmail.com>
> wrote:
> > >
> > > > Yes, you are absolutely right. Cannot believe I posted in the wrong
> > > > thread...
> > > >
> > > > On Wed, Dec 4, 2019 at 1:46 PM Jark Wu <imj...@gmail.com> wrote:
> > > >
> > > >> Thanks Becket the the updating,
> > > >>
> > > >> But shouldn't this message be posted in FLIP-27 discussion
> thread[1]?
> > > >>
> > > >>
> > > >> Best,
> > > >> Jark
> > > >>
> > > >> [1]:
> > > >>
> > > >>
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-27-Refactor-Source-Interface-td24952.html
> > > >>
> > > >> On Wed, 4 Dec 2019 at 12:12, Becket Qin <becket....@gmail.com>
> wrote:
> > > >>
> > > >> > Hi all,
> > > >> >
> > > >> > Sorry for the long belated update. I have updated FLIP-27 wiki
> page
> > > with
> > > >> > the latest proposals. Some noticeable changes include:
> > > >> > 1. A new generic communication mechanism between SplitEnumerator
> and
> > > >> > SourceReader.
> > > >> > 2. Some detail API method signature changes.
> > > >> >
> > > >> > We left a few things out of this FLIP and will address them in
> > > separate
> > > >> > FLIPs. Including:
> > > >> > 1. Per split event time.
> > > >> > 2. Event time alignment.
> > > >> > 3. Fine grained failover for SplitEnumerator failure.
> > > >> >
> > > >> > Please let us know if you have any question.
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > Jiangjie (Becket) Qin
> > > >> >
> > > >> > On Tue, Nov 19, 2019 at 10:28 AM Yijie Shen <
> > > henry.yijies...@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > Hi everyone,
> > > >> > >
> > > >> > > I've put the catalog part design in separate doc with more
> details
> > > for
> > > >> > > easier communication.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://docs.google.com/document/d/1LMnABtXn-wQedsmWv8hopvx-B-jbdr8-jHbIiDhdsoE/edit?usp=sharing
> > > >> > >
> > > >> > > I would love to hear your thoughts on this.
> > > >> > >
> > > >> > > Best,
> > > >> > > Yijie
> > > >> > >
> > > >> > > On Mon, Oct 21, 2019 at 11:15 AM Yijie Shen <
> > > >> henry.yijies...@gmail.com>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Hi everyone,
> > > >> > > >
> > > >> > > > Glad to receive your valuable feedbacks.
> > > >> > > >
> > > >> > > > I'd first separate the Pulsar catalog as another doc and show
> > more
> > > >> > design
> > > >> > > > and implementation details there.
> > > >> > > >
> > > >> > > > For the current FLIP-72, I would separate it into the sink
> part
> > > for
> > > >> > > > current work and keep the source part as future works until we
> > > reach
> > > >> > > > FLIP-27 finals.
> > > >> > > >
> > > >> > > > I also reply to some of the comments in the design doc. I will
> > > >> rewrite
> > > >> > > the
> > > >> > > > catalog part in regarding to Bowen's advice in both email and
> > > >> comments.
> > > >> > > >
> > > >> > > > Thanks for the help again.
> > > >> > > >
> > > >> > > > Best,
> > > >> > > > Yijie
> > > >> > > >
> > > >> > > > On Fri, Oct 18, 2019 at 12:40 AM Rong Rong <
> walter...@gmail.com
> > >
> > > >> > wrote:
> > > >> > > >
> > > >> > > >> Hi Yijie,
> > > >> > > >>
> > > >> > > >> I also agree with Jark on separating the Catalog part into
> > > another
> > > >> > FLIP.
> > > >> > > >>
> > > >> > > >> With FLIP-27[1] also in the air, it is also probably great to
> > > split
> > > >> > and
> > > >> > > >> unblock the sink implementation contribution.
> > > >> > > >> I would suggest either putting in a detail implementation
> plan
> > > >> section
> > > >> > > in
> > > >> > > >> the doc, or (maybe too much separation?) splitting them into
> > > >> different
> > > >> > > >> FLIPs. What do you guys think?
> > > >> > > >>
> > > >> > > >> --
> > > >> > > >> Rong
> > > >> > > >>
> > > >> > > >> [1]
> > > >> > > >>
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
> > > >> > > >>
> > > >> > > >> On Wed, Oct 16, 2019 at 9:00 PM Jark Wu <imj...@gmail.com>
> > > wrote:
> > > >> > > >>
> > > >> > > >> > Hi Yijie,
> > > >> > > >> >
> > > >> > > >> > Thanks for the design document. I agree with Bowen that the
> > > >> catalog
> > > >> > > part
> > > >> > > >> > needs more details.
> > > >> > > >> > And I would suggest to separate Pulsar Catalog as another
> > FLIP.
> > > >> IMO,
> > > >> > > it
> > > >> > > >> has
> > > >> > > >> > little to do with source/sink.
> > > >> > > >> > Having a separate FLIP can unblock the contribution for
> sink
> > > (or
> > > >> > > source)
> > > >> > > >> > and keep the discussion more focus.
> > > >> > > >> > I also left some comments in the documentation.
> > > >> > > >> >
> > > >> > > >> > Thanks,
> > > >> > > >> > Jark
> > > >> > > >> >
> > > >> > > >> > On Thu, 17 Oct 2019 at 11:24, Yijie Shen <
> > > >> henry.yijies...@gmail.com
> > > >> > >
> > > >> > > >> > wrote:
> > > >> > > >> >
> > > >> > > >> > > Hi Bowen,
> > > >> > > >> > >
> > > >> > > >> > > Thanks for your comments. I'll add catalog details as you
> > > >> > suggested.
> > > >> > > >> > >
> > > >> > > >> > > One more question: since we decide to not implement
> source
> > > >> part of
> > > >> > > the
> > > >> > > >> > > connector at the moment.
> > > >> > > >> > > What can users do with a Pulsar catalog?
> > > >> > > >> > > Create a table backed by Pulsar and check existing pulsar
> > > >> tables
> > > >> > to
> > > >> > > >> see
> > > >> > > >> > > their schemas? Drop tables maybe?
> > > >> > > >> > >
> > > >> > > >> > > Best,
> > > >> > > >> > > Yijie
> > > >> > > >> > >
> > > >> > > >> > > On Thu, Oct 17, 2019 at 1:04 AM Bowen Li <
> > > bowenl...@gmail.com>
> > > >> > > wrote:
> > > >> > > >> > >
> > > >> > > >> > > > Hi Yijie,
> > > >> > > >> > > >
> > > >> > > >> > > > Per the discussion, maybe you can move pulsar source to
> > > >> 'future
> > > >> > > >> work'
> > > >> > > >> > > > section in the FLIP for now?
> > > >> > > >> > > >
> > > >> > > >> > > > Besides, the FLIP seems to be quite rough at the
> moment,
> > > and
> > > >> I'd
> > > >> > > >> > > recommend
> > > >> > > >> > > > to add more details .
> > > >> > > >> > > >
> > > >> > > >> > > > A few questions mainly regarding the proposed pulsar
> > > catalog.
> > > >> > > >> > > >
> > > >> > > >> > > >    - Can you provide some background of pulsar schema
> > > >> registry
> > > >> > and
> > > >> > > >> how
> > > >> > > >> > it
> > > >> > > >> > > >    works?
> > > >> > > >> > > >    - The proposed design of pulsar catalog is very
> vague
> > > now,
> > > >> > can
> > > >> > > >> you
> > > >> > > >> > > >    share some details of how a pulsar catalog would
> work
> > > >> > > internally?
> > > >> > > >> > E.g.
> > > >> > > >> > > >       - which APIs does it support exactly? E.g. I see
> > from
> > > >> your
> > > >> > > >> > > >       prototype that table creation is supported but
> not
> > > >> > > alteration.
> > > >> > > >> > > >       - is it going to connect to a pulsar schema
> > registry
> > > >> via a
> > > >> > > >> http
> > > >> > > >> > > >       client or a pulsar client, etc
> > > >> > > >> > > >       - will it be able to handle multiple versions of
> > > >> pulsar,
> > > >> > or
> > > >> > > >> just
> > > >> > > >> > > >       one? How is compatibility handles between
> different
> > > >> > > >> Flink-Pulsar
> > > >> > > >> > > versions?
> > > >> > > >> > > >       - will it support only reading from pulsar schema
> > > >> > registry ,
> > > >> > > >> or
> > > >> > > >> > > >       both read/write? Will it work end-to-end in Flink
> > SQL
> > > >> for
> > > >> > > >> users
> > > >> > > >> > to
> > > >> > > >> > > create
> > > >> > > >> > > >       and manipulate a pulsar table such as "CREATE
> > TABLE t
> > > >> WITH
> > > >> > > >> > > >       PROPERTIES(type=pulsar)" and "DROP TABLE t"?
> > > >> > > >> > > >       - Is a pulsar topic always gonna be a
> > non-partitioned
> > > >> > table?
> > > >> > > >> How
> > > >> > > >> > is
> > > >> > > >> > > >       a partitioned topic mapped to a Flink table?
> > > >> > > >> > > >    - How to map Flink's catalog/database namespace to
> > > >> pulsar's
> > > >> > > >> > > >    multi-tenant namespaces? I'm not very familiar with
> > how
> > > >> multi
> > > >> > > >> > tenancy
> > > >> > > >> > > works
> > > >> > > >> > > >    in pulsar, and some background context/use cases may
> > > help
> > > >> > here
> > > >> > > >> too.
> > > >> > > >> > > E.g.
> > > >> > > >> > > >       - can a pulsar client/consumer/producer be
> > > >> multiple-tenant
> > > >> > > at
> > > >> > > >> the
> > > >> > > >> > > >       same time?
> > > >> > > >> > > >       - how does authentication work in pulsar's
> > > >> multi-tenancy
> > > >> > and
> > > >> > > >> the
> > > >> > > >> > > >       catalog? asking since I didn't see the proposed
> > > pulsar
> > > >> > > catalog
> > > >> > > >> > has
> > > >> > > >> > > >       username/password configs
> > > >> > > >> > > >       - the FLIP seems propose mapping a pulsar cluster
> > and
> > > >> > > >> > > >       'tenant/namespace' respectively to Flink's
> > 'catalog'
> > > >> and
> > > >> > > >> > > 'database'. I
> > > >> > > >> > > >       wonder whether it totally makes sense, or should
> we
> > > >> > actually
> > > >> > > >> map
> > > >> > > >> > > "tenant"
> > > >> > > >> > > >       to "catalog", and "namespace" to "database"?
> > > >> > > >> > > >
> > > >> > > >> > > > Cheers,
> > > >> > > >> > > > Bowen
> > > >> > > >> > > >
> > > >> > > >> > > > On Fri, Sep 20, 2019 at 1:16 AM Yijie Shen <
> > > >> > > >> henry.yijies...@gmail.com>
> > > >> > > >> > > > wrote:
> > > >> > > >> > > >
> > > >> > > >> > > >> Hi everyone,
> > > >> > > >> > > >>
> > > >> > > >> > > >> Per discussion in the previous thread
> > > >> > > >> > > >> <
> > > >> > > >> > > >>
> > > >> > > >> > >
> > > >> > > >> >
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Contribute-Pulsar-Flink-connector-back-to-Flink-tc32538.html
> > > >> > > >> > > >> >,
> > > >> > > >> > > >> I have created FLIP-72 to kick off a more detailed
> > > >> discussion
> > > >> > on
> > > >> > > >> the
> > > >> > > >> > > Flink
> > > >> > > >> > > >> Pulsar connector:
> > > >> > > >> > > >>
> > > >> > > >> > > >>
> > > >> > > >> > > >>
> > > >> > > >> > >
> > > >> > > >> >
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-72%3A+Introduce+Pulsar+Connector
> > > >> > > >> > > >>
> > > >> > > >> > > >> In short, the connector has the following features:
> > > >> > > >> > > >>
> > > >> > > >> > > >>    -
> > > >> > > >> > > >>
> > > >> > > >> > > >>    Pulsar as a streaming source with exactly-once
> > > guarantee.
> > > >> > > >> > > >>    -
> > > >> > > >> > > >>
> > > >> > > >> > > >>    Sink streaming results to Pulsar with at-least-once
> > > >> > semantics.
> > > >> > > >> > > >>    -
> > > >> > > >> > > >>
> > > >> > > >> > > >>    Build upon Flink new Table API Type system (FLIP-37
> > > >> > > >> > > >>    <
> > > >> > > >> > > >>
> > > >> > > >> > >
> > > >> > > >> >
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > > >> > > >> > > >> >
> > > >> > > >> > > >>    ), and can automatically (de)serialize messages
> with
> > > the
> > > >> > help
> > > >> > > of
> > > >> > > >> > > Pulsar
> > > >> > > >> > > >>    schema.
> > > >> > > >> > > >>    -
> > > >> > > >> > > >>
> > > >> > > >> > > >>    Integrate with Flink new Catalog API (FLIP-30
> > > >> > > >> > > >>    <
> > > >> > > >> > > >>
> > > >> > > >> > >
> > > >> > > >> >
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > > >> > > >> > > >> >),
> > > >> > > >> > > >>    which enables the use of Pulsar topics as tables in
> > > Table
> > > >> > API
> > > >> > > as
> > > >> > > >> > well
> > > >> > > >> > > >> as
> > > >> > > >> > > >>    SQL client.
> > > >> > > >> > > >>
> > > >> > > >> > > >>
> > > >> > > >> > > >>
> > > >> > > >> > > >>
> > > >> > > >> > >
> > > >> > > >> >
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#heading=h.28v5v23yeq1u
> > > >> > > >> > > >>
> > > >> > > >> > > >>
> > > >> > > >> > > >> Would love to here your thoughts on this.
> > > >> > > >> > > >>
> > > >> > > >> > > >> Best,
> > > >> > > >> > > >> Yijie
> > > >> > > >> > > >>
> > > >> > > >> > > >
> > > >> > > >> > >
> > > >> > > >> >
> > > >> > > >>
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>

Reply via email to