Re: sudden problems in running a test pipieline

Hi, Is this the full stack trace? How do you start this pipeline? Med Vänlig Hälsning / Best Regards *Matyas Manninger* *Data Engineer Consultant* +46 76 050 6326 m...@lunar.app, lunar.app CVR: DK36982837 On Wed, 7 Dec 2022 at 19:40, Sofia’s World wrote: > Hello all >

Re: sudden problems in running a test pipieline

Hi Thanks for getting back to me. No it's not and I think it's pytest related ..something clashing with my data flow Options (which parses initial args) have overcome issue by renaming my options Params but will need to go to the bottom of this... If anyone has experience same pls advise I m using

Beam saves filepaths in Flink's state

Hi everyone, I noticed that the Flink state contains KafkaIO's consumer config properties. When restarting the Beam pipeline (Java SDK) from state, the Flink Runner translation layer will deserialize the KafkaUnboudedReader (via UnboundedSourceWrapper) from Flink's state. This happens *before* th

Re: Beam saves filepaths in Flink's state

This doesn't sound ideal to me. For contrast, Dataflow doesn't save any of these things (coders, transforms, configs) in state, which makes it easier for Dataflow to update pipelines. On Thu, Dec 8, 2022 at 7:48 AM Cristian Constantinescu wrote: > Hi everyone, > > I noticed that the Flink state

SDK Harness Memory Usage

Hi Beam Team, Can somebody help me understand what are the factors behind SDK Harness memory usage? My first guess is that the SDK Harness memory usage depends on: 1. User code (i.e. DoFns) 2. Bundle size Basically, the maximum memory usage an SDK Harness needs is however much memory it takes fo