I had a similar situation with my Elasticsearch source where you don't know before executing the query (via scroll API for example) how many splits you will find. How should you handle those situation with new Source API?
On Mon, Jul 17, 2023 at 10:09 AM Martijn Visser <martijnvis...@apache.org> wrote: > Hi Kamal, > > It would require you to find a way to create a TCP connection on task > managers where you would only read the assigned part of the TCP connection. > Looking at the protocol itself, that most likely would be an issue. A TCP > connection would also be problematic in case of replays and checkpoint > integration, since you can't roll back to the previous messages of the TCP > connection. > > Best regards, > > Martijn > > On Sat, Jul 15, 2023 at 7:20 AM Kamal Mittal via user < > user@flink.apache.org> wrote: > >> Hello Community, >> >> >> >> Please share views for below mail. >> >> >> >> Rgds, >> >> Kamal >> >> >> >> *From:* Kamal Mittal via user <user@flink.apache.org> >> *Sent:* 14 July 2023 12:55 PM >> *To:* user@flink.apache.org >> *Subject:* TCP Socket stream scalability >> >> >> >> Hello, >> >> >> >> TCP Socket stream can be scaled across task managers similarly to file >> enumerator and source reader below? >> >> >> >> Job is submitted with TCP socket source function and a socket will bind >> on a port once and by a task manager. Is it possible to open socket at job >> manager and then scale / divide the work among task managers similarly to >> below? >> >> >> >> >> >> Rgds, >> >> Kamal >> >
