It's nice. Will present flink source connector be pushed to bahir-flink? I can add netty-source to bahir-flink.
Maven repository have no bahir-flink's. https://mvnrepository.com/artifact/org.apache.bahir -----邮件原件----- 发件人: Greg Hogan [mailto:c...@greghogan.com] 发送时间: 2016年9月27日 20:58 收件人: dev@flink.apache.org 主题: Re: 答复: [DISCUSS] add netty tcp/restful pushed source support Apache Bahir's website only suggests support for additional frameworks, but there is a Flink repository at https://github.com/apache/bahir-flink On Tue, Sep 27, 2016 at 8:38 AM, shijinkui <shijin...@huawei.com> wrote: > Hey, Stephan Ewen > > 1. bahir's target is spark. The contributer are rxin, srowen, tdas, > mateiz and so on. > If we want bahir used by flink, we can suggest bahir provide > streaming connecter interface, such as store(), start(), stop(), > restart(), receiving(Any)... > Then same streaming connector can be implemented by spark and > flink. But I think this is impossible, as bahir depend spark-streaming > and spark sql. > 2. About connector fault tolerance. Bahir's mqtt and akka connector > are themselves' storage. But netty have no persist data feature. > I think we can append data to a ringbuffer. When SourceContext > collect() throw error, then write message to inform client to stop > send message. When flink SourceContext is normal, then write the > ringbuffer data to flink, inform client to go on. > Because pushing mode is hard to control the flow throughput, > the upstream client can. > This netty connector's purpose is end-to-end streaming, > minimum time delay. > 3. Later on, We can provide http protocol ability, put, post, get, > even websocket or jersey restful based on netty. > > -----邮件原件----- > 发件人: Stephan Ewen [mailto:se...@apache.org] > 发送时间: 2016年9月27日 16:54 > 收件人: dev@flink.apache.org > 主题: Re: [DISCUSS] add netty tcp/restful pushed source support > > I think that could be an interesting source. Two quick questions to > move forward > > - To keep the Flink code base from becoming too big (hard to > maintain and > test) we started working with Apache Bahir as a project dedicated to > streaming connectors. Would that be a good target for the connector? > > - What are your thoughts on fault tolerance for that connector? > > On Mon, Sep 26, 2016 at 3:01 PM, shijinkui <shijin...@huawei.com> wrote: > > > Hi, all > > > > 1. In order to support end-to-end pushed source, I create FLINK-4630< > > https://issues.apache.org/jira/browse/FLINK-4630>. I want to know > > whether is this idea worth? > > > > ------------------- > > When source stream get start, listen a provided tcp port, receive > > stream data from user data source. > > This netty tcp source is keepping alive and end-to-end, that is from > > business system to flink worker directly. > > > > user app push -----> netty server source of Flink > > > > describe the source in detail below: > > > > 1. source run as a netty tcp server > > 2. user provide a tcp port, if the port is in used, increace the port > > number between 1024 to 65535. Source can parallel. > > 3. callback the provided url to report the real port to listen > > 4. user push streaming data to netty server, then collect the data to > > flink > > > > > > Thanks > > > > Jinkui Shi > > > > >
