Re: Beam slowness compared to flink-native

Hi Ifat, can you try adding 'use_deprecated_read' experiment to the PipelineOptions? IIRC the default expansion for KafkaIO uses splittable DoFn now, which could be the cause for the performance difference you see.  You can add the option on command line using "--experiments=use_deprecated_re

Re: Beam slowness compared to flink-native

Hi Jan, I tried it, but didn’t see any significant effect on performance. Thanks for the suggestion. Ifat From: Jan Lukavský Reply-To: "user@beam.apache.org" Date: Thursday, 12 May 2022 at 10:22 To: "user@beam.apache.org" Subject: Re: Beam slowness compared to flink-native Hi Ifat, can you

Looking at logs from Java Fn harness

Hi, Is there a way to see the logs for code running in Java Fn Harness? When I do "docker logs -f ...", I can see something like this: SDK Fn Harness started Harness ID 1-1 Logging location url: "host.docker.internal:58738" After reading cwiki [1], I also set --sdkHarnessLogLevelOverrides, but i

Re: [Question] - Flink Runner Savepoint Enabling

Hi Ivan, Did you mean to take an automatic savepoint or checkpoint? A savepoint is usually manual and initiated during (for example) Flink version upgrade or Flink job update. [1] You can set Flink checkpoint interval with Beam's Flink runner. Thanks, Deepak [1] https://nightlies.apache.org/fli