Hello Shammon, Please have a look for below and share views.
Rgds, Kamal From: Kamal Mittal via user <user@flink.apache.org> Sent: 02 August 2023 08:02 AM To: Shammon FY <zjur...@gmail.com>; user@flink.apache.org Subject: RE: Flink netty connector for TCP source Thanks Shammon. Purpose of opening server socket in Split Enumerator was that it has only one instance per source and so the server socket too (port binding can happen only once). And then accepted Socket connections (serversocket.accept()) will act as splits which will be further processed by readers. Let me know please if there is issue you see in above understanding or some other way you can suggest. Issue is that server socket can only bind a port once over a machine node and that’s why we thought to open it in split enumerator which is called only once and per source. Rgds, Kamal From: Shammon FY <zjur...@gmail.com<mailto:zjur...@gmail.com>> Sent: 02 August 2023 07:48 AM To: Kamal Mittal <kamal.mit...@ericsson.com<mailto:kamal.mit...@ericsson.com>>; user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Flink netty connector for TCP source Hi Kamal, It confuses me a little that what's the purpose of opening a server socket in SplitEnumerator? Currently there will be only one SplitEnumerator instance in JobManager for each one source not each source subtask. If there's only one source in your job, no matter how much parallelism this source has, there will be only one SplitEnumerator instance in JM which can connect and send events to the source subtasks. Additionally, if you want to read data from a netty source, you can implement your NettySplitEnumerator to get splits from netty source and assign them to a NettySourceReader which will read data from netty source according to the splits. Best, Shammon FY On Tue, Aug 1, 2023 at 12:02 PM Kamal Mittal via user <user@flink.apache.org<mailto:user@flink.apache.org>> wrote: Hello, I created a custom server socket source in which opened server socket in split enumerator, source parallelism is =2 and it gives error while running - “Address is already in use” i.e. it tried to open two server sockets as per parallelism which is not correct as parallelism is applicable for source operator and not for split enumerator? Please correct me if above understanding is not correct. Rgds, Kamal From: Hang Ruan <ruanhang1...@gmail.com<mailto:ruanhang1...@gmail.com>> Sent: 01 August 2023 08:55 AM To: Kamal Mittal <kamal.mit...@ericsson.com<mailto:kamal.mit...@ericsson.com>> Cc: liu ron <ron9....@gmail.com<mailto:ron9....@gmail.com>>; user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Flink netty connector for TCP source Hi, Kamal. The SplitEnumerator is contained in the SourceCoordinator. They are only used in JM. Best, Hang Kamal Mittal via user <user@flink.apache.org<mailto:user@flink.apache.org>> 于2023年8月1日周二 10:43写道： Thanks. I looked at the link for custom data sources, one query here that how to make sure for Split enumerator to execute on Job Manager rather than at Task manager? From: liu ron <ron9....@gmail.com<mailto:ron9....@gmail.com>> Sent: 31 July 2023 10:06 PM To: user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Flink netty connector for TCP source Hi, Kamal Currently, we don't provide the connector like TCP source in Flink main repo. If you need this connector, you can try to implement it refer to the FLIP-27 source docs [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/sources/ Best, Ron Shammon FY <zjur...@gmail.com<mailto:zjur...@gmail.com>> 于2023年7月27日周四 11:23写道： I cannot find any information about netty source in flink website and it is not in the connector list[1], so I'm think that it is not supported by flink community [1] https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/ Best, Shammon FY On Thu, Jul 27, 2023 at 10:53 AM Kamal Mittal <kamal.mit...@ericsson.com<mailto:kamal.mit...@ericsson.com>> wrote: Hello Shammon, Yes socket text stream I am aware of but was thinking if something like as ‘https://github.com/apache/bahir-flink/tree/master/flink-connector-netty<https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-ffc853ff875d9ed3&q=1&e=30c3fa9e-7c08-4486-8017-0d78c3c714ac&u=https%3A%2F%2Fgithub.com%2Fapache%2Fbahir-flink%2Ftree%2Fmaster%2Fflink-connector-netty>’ is also supported by Flink? Rgds, Kamal From: Shammon FY <zjur...@gmail.com<mailto:zjur...@gmail.com>> Sent: 27 July 2023 08:15 AM To: Kamal Mittal <kamal.mit...@ericsson.com<mailto:kamal.mit...@ericsson.com>> Cc: user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Flink netty connector for TCP source Hi Kamal, There's socket text stream in `DataStream` and you can refer to [1] for more details. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/overview/#example-program Best, Shammon FY On Wed, Jul 26, 2023 at 4:26 PM Kamal Mittal via user <user@flink.apache.org<mailto:user@flink.apache.org>> wrote: Hello, Does flink provides netty connector for custom TCP source? Any documentation details please share? Rgds, Kamal
