This is a conclusion I also came across implementing an application receiving events via TCP/HTTP.
This also brings the benefit, as already said, that you can use the modern ecosystem for implementing such things and don't need to so it yourself. I summarized my conclusions developing my application in that paper: http://dx.doi.org/10.1145/3579142.3594292 Best regards, Martin Am 2. Juli 2023 09:13:14 MESZ schrieb "Jan Lukavský" <je...@seznam.cz>: >Hi, > >in plain TCP socket, there is no 'state' you can return to when restoring from >checkpoint. All you can do is reopening the socket. You would have to ensure >fault-tolerance by some (custom) higher-level protocol you implement, >including persistent storage, replication, etc. You will soon see you are >actually implementing a commit-log like Apache Kafka, so it might be better to >use Kafka directly. If your clients need to send data over plain TCP socket, >you can receive it over TCP (with a custom TCP server application), store to >Kafka and use Kafka as a data source for your Flink application. > >Best, > > Jan > >On 7/1/23 05:20, Kamal Mittal wrote: >> >> Hello, >> >> Thanks for your suggestion but please confirm below. >> >> Is it the case that TCP socket source job can’t be restored from last >> checkpoint? >> >> Rgds, >> >> Kamal >> >> *From:*Jan Lukavský <je...@seznam.cz> >> *Sent:* 29 June 2023 06:18 PM >> *To:* user@flink.apache.org >> *Subject:* Re: Flink TCP custom source - secured server socket >> >> > ... a state backward in (processing) time ... >> (of course not processing, I meant to say event time) >> >> On 6/29/23 14:45, Jan Lukavský wrote: >> >> Hi Kamal, >> >> you probably have several options: >> >> a) bundle your private key and certificate into your Flink >> application's jar (not recommended, your service's private key >> will have to be not exactly "private") >> b) create a service which will provide certificate for your >> service during runtime (e.g. ACME based or similar) >> >> I have a different note, though. Flink (or for that matters any >> streaming engine, I'm more focused on Apache Beam) heavily relies >> on the ability of sources to restore a state backward in >> (processing) time. That is definitely not the case of a plain TCP >> socket. It is likely you will experience data-loss issues with >> this solution (regardless of security). This might be okay for >> you, I just felt it would be good to stress this out. >> >> Best, >> >> Jan >> >> On 6/29/23 12:53, Kamal Mittal via user wrote: >> >> Hello Community, >> >> I have created TCP stream custom source and reading data from >> TCP stream source. >> >> But this TCP connection needs to be secured i.e. SSL based, >> query is how to configure/provide certificates via Flink for >> Client-Server secured TCP connection? >> >> Rgds, >> >> Kamal >>
