Yes, you are right ! On Thu, Aug 3, 2023 at 1:35 PM Kamal Mittal <kamal.mit...@ericsson.com> wrote:
> Hello Shammon, > > > > As it is said one split enumerator for one source means multiple sub-tasks > of that source (if parallelism >1) will use same split enumerator instance > right? > > > > Rgds, > > Kamal > > > > *From:* Shammon FY <zjur...@gmail.com> > *Sent:* 03 August 2023 10:54 AM > *To:* Kamal Mittal <kamal.mit...@ericsson.com> > *Cc:* user@flink.apache.org > *Subject:* Re: Flink netty connector for TCP source > > > > Hi Kamal, > > > > I think it depends on your practical application. In general, the built-in > source in Flink such as kafka or hive will proactively fetch splits from > source, instead of starting a service with a source service pushed over. > > > > Returning to the issue of port conflicts, you may need to check if the > specified port in the split enumerator is being used by other services, or > if random ports can be used. Generally speaking, there will only be one > split enumerator for one source, but when the job master fails to restart, > the split enumerator of the previous job may not be completely shutdown. If > they use the same port, it may also cause conflicts. > > > > Best, > > Shammon FY > > > > On Thu, Aug 3, 2023 at 11:57 AM Kamal Mittal <kamal.mit...@ericsson.com> > wrote: > > Hello Shammon, > > > > Please have a look for below and share views. > > > > Rgds, > > Kamal > > > > *From:* Kamal Mittal via user <user@flink.apache.org> > *Sent:* 02 August 2023 08:02 AM > *To:* Shammon FY <zjur...@gmail.com>; user@flink.apache.org > *Subject:* RE: Flink netty connector for TCP source > > > > Thanks Shammon. > > > > Purpose of opening server socket in Split Enumerator was that it has only > one instance per source and so the server socket too (port binding can > happen only once). And then accepted Socket connections > (serversocket.accept()) will act as splits which will be further processed > by readers. > > > > Let me know please if there is issue you see in above understanding or > some other way you can suggest. Issue is that server socket can only bind a > port once over a machine node and that’s why we thought to open it in split > enumerator which is called only once and per source. > > > > Rgds, > > Kamal > > > > *From:* Shammon FY <zjur...@gmail.com> > *Sent:* 02 August 2023 07:48 AM > *To:* Kamal Mittal <kamal.mit...@ericsson.com>; user@flink.apache.org > *Subject:* Re: Flink netty connector for TCP source > > > > Hi Kamal, > > > > It confuses me a little that what's the purpose of opening a server socket > in SplitEnumerator? Currently there will be only one SplitEnumerator > instance in JobManager for each one source not each source subtask. If > there's only one source in your job, no matter how much parallelism this > source has, there will be only one SplitEnumerator instance in JM which can > connect and send events to the source subtasks. > > > > Additionally, if you want to read data from a netty source, you can > implement your NettySplitEnumerator to get splits from netty source and > assign them to a NettySourceReader which will read data from netty source > according to the splits. > > > > Best, > > Shammon FY > > > > > > On Tue, Aug 1, 2023 at 12:02 PM Kamal Mittal via user < > user@flink.apache.org> wrote: > > Hello, > > > > I created a custom server socket source in which opened server socket in > split enumerator, source parallelism is =2 and it gives error while running > - “Address is already in use” i.e. it tried to open two server sockets as > per parallelism which is not correct as parallelism is applicable for > source operator and not for split enumerator? > > > > Please correct me if above understanding is not correct. > > > > Rgds, > > Kamal > > > > *From:* Hang Ruan <ruanhang1...@gmail.com> > *Sent:* 01 August 2023 08:55 AM > *To:* Kamal Mittal <kamal.mit...@ericsson.com> > *Cc:* liu ron <ron9....@gmail.com>; user@flink.apache.org > *Subject:* Re: Flink netty connector for TCP source > > > > Hi, Kamal. > > > > The SplitEnumerator is contained in the SourceCoordinator. They are only > used in JM. > > > > Best, > > Hang > > > > Kamal Mittal via user <user@flink.apache.org> 于2023年8月1日周二 10:43写道： > > Thanks. > > > > I looked at the link for custom data sources, one query here that how to > make sure for Split enumerator to execute on Job Manager rather than at > Task manager? > > > > *From:* liu ron <ron9....@gmail.com> > *Sent:* 31 July 2023 10:06 PM > *To:* user@flink.apache.org > *Subject:* Re: Flink netty connector for TCP source > > > > Hi, Kamal > > Currently, we don't provide the connector like TCP source in Flink main > repo. If you need this connector, you can try to implement it refer to the > FLIP-27 source docs > > > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/sources/ > > > > Best, > > Ron > > > > Shammon FY <zjur...@gmail.com> 于2023年7月27日周四 11:23写道： > > I cannot find any information about netty source in flink website and it > is not in the connector list[1], so I'm think that it is not supported by > flink community > > > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/ > > > > Best, > > Shammon FY > > > > On Thu, Jul 27, 2023 at 10:53 AM Kamal Mittal <kamal.mit...@ericsson.com> > wrote: > > Hello Shammon, > > > > Yes socket text stream I am aware of but was thinking if something like as > ‘https://github.com/apache/bahir-flink/tree/master/flink-connector-netty > <https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-ffc853ff875d9ed3&q=1&e=30c3fa9e-7c08-4486-8017-0d78c3c714ac&u=https%3A%2F%2Fgithub.com%2Fapache%2Fbahir-flink%2Ftree%2Fmaster%2Fflink-connector-netty>’ > is also supported by Flink? > > > > Rgds, > > Kamal > > > > *From:* Shammon FY <zjur...@gmail.com> > *Sent:* 27 July 2023 08:15 AM > *To:* Kamal Mittal <kamal.mit...@ericsson.com> > *Cc:* user@flink.apache.org > *Subject:* Re: Flink netty connector for TCP source > > > > Hi Kamal, > > > > There's socket text stream in `DataStream` and you can refer to [1] for > more details. > > > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/overview/#example-program > > > > Best, > > Shammon FY > > > > On Wed, Jul 26, 2023 at 4:26 PM Kamal Mittal via user < > user@flink.apache.org> wrote: > > Hello, > > > > Does flink provides netty connector for custom TCP source? > > > > Any documentation details please share? > > > > Rgds, > > Kamal > >
