Python/Beam Windowing + Triggering Recommendations

2020-06-11 Thread Pradip Thachile
Hi, I'm writing a pipeline in Python/Beam 2.19 and wanted to get the opinions of the folks on list here around the best way to implement the following logic, specifically around the window + trigger combinations to use. HourlySource := a PCollection that receives an event an hour from another

Re: KafkaIO Read Latency

2020-06-11 Thread Alexey Romanenko
Thank you for details. I expect that StartBundle and FinishBundle methods are called in your DoFn (KafkaMessageExtractor). Since it’s called before your Window transform, I don’t think it should affect it. Did you count how many records are processed by one bundle? What is actually happening

Re: BEAM-2217 NotImplementedError - DataflowRunner parsing Protos from PubSub (Python SDK)

2020-06-11 Thread Lien Michiels
Hi Udi, Brian, Thanks so much 🙌 I'm now able to deploy the pipeline. Lien On Thu, 11 Jun 2020 at 00:02, Udi Meiri wrote: > I don't believe you need to register a coder unless you create your own > coder class. > > Since the coder has a proto_message_type field, it could return that in > to_