Hi Dian, all, I will come back to the other points asap. However, I’m still confused about this performance. Is this what I can expect in PyFlink in terms of performance? ~ 1000ms latency for single events? I also had a very simple setup where I send 1000 events to Kafka per second and response times/latencies was around 15 seconds for single events. I understand there is some Python/JVM overhead but since Flink is so performant, I would expect much better numbers. In the current situation, PyFlink would just be unusable if you care about latency. Is this something that you expect to be improved in the future?
I will verify how this works out for Beam in a remote environment. Thanks again! Wouter On Thu, 8 Jul 2021 at 08:28, Dian Fu <dian0511...@gmail.com> wrote: > Hi Wouter, > > 1) Regarding the performance difference between Beam and PyFlink, I guess > it’s because you are using an in-memory runner when running it locally in > Beam. In that case, the code path is totally differently compared to > running in a remote cluster. > 2) Regarding to `flink run`, I’m surprising that it’s running locally. > Could you submit a java job with similar commands to see how it runs? > 3) Regarding to `flink run-application`, could you share the exception > stack? > > Regards, > Dian > > 2021年7月6日 下午4:58，Wouter Zorgdrager <zorgdrag...@gmail.com> 写道： > > uses > > >
