+1 for starting with pulsar sink and catalog.
Sijie
On Tue, Sep 24, 2019 at 3:36 PM Jark Wu wrote:
> I agree that we can start from Pulsar sink and Pulsar Catalog.
> When we finish that, maybe FLIP-27 is ready then.
>
> Best,
> Jark
>
> On Tue, 24 Sep 2019 at 17:06, Becket Qin wrote:
>
>> Hi T
I agree that we can start from Pulsar sink and Pulsar Catalog.
When we finish that, maybe FLIP-27 is ready then.
Best,
Jark
On Tue, 24 Sep 2019 at 17:06, Becket Qin wrote:
> Hi Till,
>
> My understanding is that we are going to hold back a little bit on the
> Pulsar source connector until FLIP-
Hi Till,
My understanding is that we are going to hold back a little bit on the
Pulsar source connector until FLIP-27 is ready. If we check in the old
Pulsar source connector right away, we will have to maintain that old
connector for quite a while (a year or more), which is something we don't
wan
What is the conclusion of this discussion? Are we gonna merge the old
source/sink or do we believe that FLIP-27 will be completed in time so that
we can merge the new source right away?
Cheers,
Till
On Tue, Sep 24, 2019 at 3:43 AM 郭士榕 wrote:
>
>
> unscribe
>
>
>
>
>
>
> At 2019-09-24 08:53:25,
Thanks, Stephan.
Sounds good to me. We can still try our best to get new Pulsar connector in
Flink 1.10. In case we do not have time to do that, we will prominently
link the Pulsar connector from the Flink connector docs.
Thanks,
Jiangjie (Becket) Qin
On Mon, Sep 23, 2019 at 4:11 PM Stephan Ewe
Okay, I see your point, Becket.
Then let us prominently link the Pulsar connector from the Flink connector
docs then, so that users can find it easily.
As soon as FLIP 27 is done, we reach out the Pulsar folks to contribute a
new connector.
On Mon, Sep 23, 2019 at 3:11 AM Becket Qin wrote:
> H
Hi Stephan,
I have no doubt about the value of adding Pulsar connector to Flink repo.
My concern is about how exactly we are going to do it.
As mentioned before, I believe that we can handle connectors more
> pragmatically and less strict than the core of Flink, if it helps unlocking
> users.
I
My assumption is as Sijie's, that once the connector is either part of
Flink, or part of the streamnative repo. No double maintenance.
I feel this discussion is very much caught in problems that are all
solvable if we want to solve them.
Maybe we can think what our goal for users and the communiti
Thanks Becket.
I think it is better for the Flink community to judge the benefits of doing
this. I was trying to provide some views from outsiders.
Thanks,
Sijie
On Fri, Sep 20, 2019 at 10:25 AM Becket Qin wrote:
> Hi Sijie,
>
> Yes, we will have to support existing old connectors and new conn
Hi Sijie,
Yes, we will have to support existing old connectors and new connectors in
parallel for a while. We have to take that maintenance overhead because
existing connectors have been used by the users for a long time. I guess It
may take at least a year for us to fully remove the old connector
Thanks everyone here. Sorry for jumping into the discussion here.
I am not very familiar about the deprecation process in Flink. If I
misunderstood the process, please fix me.
As far as I understand, FLIP-27 is introducing a new unified API for
connectors. After it introduces the new API
and befo
Thanks for the explanation, Stephan. I have a few questions / thoughts.
So that means we will remove the old connector without a major version
bump, is that correct?
I am not 100% sure if mixing 1.10 connectors with 1.11 connectors will
always work because we saw some dependency class collisions
My take would be the following:
- If we merge the connector now and replace it with a FLIP-27 version
before the 1.10 release, then we need no deprecation process
- If we don't manage to replace it with a FLIP-27 version before the 1.10
release, than it is good that we have the other version,
Hi,
You should have the necessary permissions.
Let me know if something doesn't work.
Cheers, Fabian
Am Do., 19. Sept. 2019 um 10:56 Uhr schrieb Yijie Shen <
henry.yijies...@gmail.com>:
> Hi Fabian,
>
> It's yijieshen
>
> Thanks for your help!
>
> On Thu, Sep 19, 2019 at 4:52 PM Fabian Hueske
Hi Stephan,
Thanks for the clarification. I completely agree with you and Thomas on the
process of adding connectors to Flink repo. However, I am wondering what is
the deprecation process? Given the main concern here was that we may have
to maintain two Pulsar connector code bases until the old on
Hi Fabian,
It's yijieshen
Thanks for your help!
On Thu, Sep 19, 2019 at 4:52 PM Fabian Hueske wrote:
>
> Hi Yijie,
>
> I can give you permission if you tell me your Confluence user name.
>
> Thanks, Fabian
>
> Am Do., 19. Sept. 2019 um 10:49 Uhr schrieb Yijie Shen <
> henry.yijies...@gmail.co
Hi Yijie,
I can give you permission if you tell me your Confluence user name.
Thanks, Fabian
Am Do., 19. Sept. 2019 um 10:49 Uhr schrieb Yijie Shen <
henry.yijies...@gmail.com>:
> > > > Could you please follow the FLIP process to start a new FLIP
> [DISCUSSION]
> > > > thread in the mailing lis
> > > Could you please follow the FLIP process to start a new FLIP [DISCUSSION]
> > > thread in the mailing list?
It seems that I don't have permission to create pages in FLIP.
> > > > Thanks for sharing the pulsar FLIP.
> > > > Would you mind enabling comments/suggestions on the google doc link
+1 to what Stephan (and Thomas) said.
Am Do., 19. Sept. 2019 um 09:54 Uhr schrieb Stephan Ewen :
> Some quick thoughts on the connector contribution process. I basically
> reiterate here what Thomas mentioned in another thread about the Kinesis
> connector.
>
> For connectors, we should favor a l
Some quick thoughts on the connector contribution process. I basically
reiterate here what Thomas mentioned in another thread about the Kinesis
connector.
For connectors, we should favor a low-overhead contribution process, and
accept user code and changes more readily than in the core system.
Tha
Hi Yijie,
Could you please follow the FLIP process to start a new FLIP [DISCUSSION]
thread in the mailing list?
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process
I see two FLIP-69 discussion in the mailing list now. So there is a FLIP
Hi Yijie,
Thanks for sharing the pulsar FLIP.
Would you mind enabling comments/suggestions on the google doc link? This
way the contributors from the community can comment on the doc.
Best,
Rong
On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen
wrote:
> Hello everyone,
>
> I've drafted a FLIP that de
Hello everyone,
I've drafted a FLIP that describes the current design of the Pulsar connector:
https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#
Please take a look and let me know what you think.
Thanks,
Yijie
On Sat, Sep 14, 2019 at 12:08 AM Rong Rong wrot
Hi All,
Sorry for joining the discussion late and thanks Yijie & Sijie for driving
the discussion.
I also think the Pulsar connector would be a very valuable addition to
Flink. I can also help out a bit on the review side :-)
Regarding the timeline, I also share concerns with Becket on the
relati
Technically speaking, removing the old connector code is a backwards
incompatible change which requires a major version bump, i.e. Flink 2.x.
Given that we don't have a clear plan on when to have the next major
version release, it seems unclear how long the old connector code will be
there if we ch
Agreed, if we check in the old code, we should make it clear that it will
be removed as soon as the FLIP-27 based version of the connector is there.
We should not commit to maintaining the old version, that would be indeed
too much overhead.
On Thu, Sep 12, 2019 at 3:30 AM Becket Qin wrote:
> Hi
Hi Stephan,
Thanks for the volunteering to help.
Yes, the overhead would just be review capacity. In fact, I am not worrying
too much about the review capacity. That is just a one time cost. My
concern is mainly about the long term burden. Assume we have new source
interface ready in 1.10 with ne
Hi all!
Nice to see this lively discussion about the Pulsar connector.
Some thoughts on the open questions:
## Contribute to Flink or maintain as a community package
Looks like the discussion is more going towards contribution. I think that
is good, especially if we think that we want to build a
Hi everyone!
Thanks for your attention and the promotion of this work.
We will prepare a FLIP as soon as possible for more specific discussions.
For FLIP-27, it seems that we have not reached a consensus. Therefore,
I will explain all the functionalities of the existing connector in
the FLIP (in
Hi Sijie,
If we agree that the goal is to have Pulsar connector in 1.10, how about we
do the following:
0. Start a FLIP to add Pulsar connector to Flink main repo as it is a new
public interface to Flink main repo.
1. Start to review the Pulsar sink right away as there is no change to the
sink in
Thank you Bowen and Becket.
What's the take from Flink community? Shall we wait for FLIP-27 or shall we
proceed to next steps? And what the next steps are? :-)
Thanks,
Sijie
On Thu, Sep 5, 2019 at 2:43 PM Bowen Li wrote:
> Hi,
>
> I think having a Pulsar connector in Flink can be a good mutual
Hi,
I think having a Pulsar connector in Flink can be a good mutual benefit to
both communities.
Another perspective is that Pulsar connector is the 1st streaming connector
that integrates with Flink's metadata management system and Catalog APIs.
It'll be cool to see how the integration turns out
Hi Till,
You are right. It all depends on when the new source interface is going to
be ready. Personally I think it would be there in about a month or so. But
I could be too optimistic. It would also be good to hear what do Aljoscha
and Stephan think as they are also involved in FLIP-27.
In gener
Hi everyone,
I'm wondering what the problem would be if we committed the Pulsar
connector before the new source interface is ready. If I understood it
correctly, then we need to support the old source interface anyway for the
existing connectors. By checking it in early I could see the benefit tha
Hi Sijie and Yijie,
Thanks for sharing your thoughts.
Just want to have some update on FLIP-27. Although the FLIP wiki and
discussion thread has been quiet for some time, a few committer /
contributors in Flink community were actually prototyping the entire thing.
We have made some good progress
Thanks for all the feedback and suggestions!
As Sijie said, the goal of the connector has always been to provide
users with the latest features of both systems as soon as possible. We
propose to contribute the connector to Flink and hope to get more
suggestions and feedback from Flink experts to e
Thanks everyone for the comments and feedback.
It seems to me that the main question here is about - "how can the Flink
community maintain the connector?".
Here are two thoughts from myself.
1) I think how and where to host this integration is kind of less important
here. I believe there can be
Hi Yijie,
Thanks for the interest in contributing the Pulsar connector.
In general, I think having Pulsar connector with strong support is a
valuable addition to Flink. So I am happy the shepherd this effort.
Meanwhile, I would also like to provide some context and recent efforts on
the Flink con
I'm quite worried that we may end up repeating history.
There were already 2 attempts at contributing a pulsar connector, both
of which failed because no committer was getting involved, despite the
contributor opening a dedicated discussion thread about the contribution
beforehand and getting
Hi everyone,
thanks a lot for starting this discussion Yijie. I think the Pulsar
connector would be a very valuable addition since Pulsar becomes more and
more popular and it would further expand Flink's interoperability. Also
from a project perspective it makes sense for me to place the connector
Hi Yun,
Since I was the main driver behind FLINK-9641 and FLINK-9168, let me try to
add more context on this.
FLINK-9641 and FLINK-9168 was created for bringing Pulsar as source and
sink for Flink. The integration was done with Flink 1.6.0. We sent out pull
requests about a year ago and we ended
Hi Yijie
I can see that Pulsar becomes more and more popular recently and very glad to
see more people willing to contribute to Flink ecosystem.
Before any further discussion, would you please give some explanation of the
relationship between this thread to current existing JIRAs of pulsar sour
42 matches
Mail list logo
