Hi Dhavan, We have looked at using pyflink for data stream enrichment and found the performance lacking compared to the java counterpart. One option for you might be to use statefun for the enrichment stages. We've also changed our model for enrichment, we're pushing the enrichment data into the pipeline instead of pulling it, but this won't work in a lot of situations.
Hope that gives you some ideas. On Mon, 2 May 2022 at 22:54, Dhavan Vaidya <dhavan.vai...@kofluence.com> wrote: > Hello! > > I want to make HTTP(S) calls to enrich data streams. The HTTP services are > running on our VPC, so the delay is limited, but sometimes these services > end up calling third party APIs, and latencies become high. > > From documentation ( > https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/python/overview/) > it seems PyFlink does not support "asyncio operator" like Java does ( > https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/operators/asyncio/). > Am I missing something? How should this be approached? > > Thanks! > > -- > Dhavan > -- This email and any attachments are proprietary and confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily reflect or represent those of SwitchDin Pty Ltd. If you have received this email in error, please let us know immediately by reply email and delete it from your system. You may not use, disseminate, distribute or copy this message nor disclose its contents to anyone. SwitchDin Pty Ltd (ABN 29 154893857) PO Box 1165, Newcastle NSW 2300 Australia
