Re: Java prefix external pipeline with go sdk

Hello, Thank you very much for the reply. I will try running the second example with dataflow runner. Here is the full output from the first example: 2022/08/02 12:50:19 Executing pipeline with the direct runner. 2022/08/02 12:50:19 Pipeline: 2022/08/02 12:50:19 Nodes: {1: []uint8/bytes GLO} {2:

[Question]: Side input got stuck

Hi There, I am implementing a beam pipeline,which is reading events from pub-sub and using sliding windowing and then combinegloblally. The thing is that I am not able to test my pipeline on local machine. below is the code sample. I am not able to feed event_out to event_fi my pipeline gets froze

Re: [Question]: Side input got stuck

I had similar issues with the local runner, I deployed the code to Dataflow and it worked. Try deploying it and see if that helps. On 2022. Aug 2., Tue at 16:31, P Singh wrote: > Hi There, > > I am implementing a beam pipeline,which is reading events from pub-sub and > using sliding windowing an

Re: [Question]: Side input got stuck

Hi Mátyás, Ohh okay I can try that but something throws this error too if there's no data in the pub-sub topic. window.WindowFn.AssignContext(source_window.max_timestamp([-1] AttributeError: 'NoneType' object has no attribute 'max_timestamp' On Tue, 2 Aug 2022 at 23:03, Mátyás Manninger via

Re: [Question]: Side input got stuck

Hi There, Actually it's not a runner problem.. If you look at the code, the combiner doesn't emit the records until the window ends. In my use-case I can't keep a short window to get meaningful records. Here is my question: How to implement a combiner to successfully emit records many times per w

Re: Java prefix external pipeline with go sdk

Okay so I have gone back and confirmed that the translation failed error also originates from the direct runner ( https://github.com/apache/beam/blob/8da6363b6fef8da5e73976be2b1277a776c05239/sdks/go/pkg/beam/runners/direct/direct.go#L337) so you should try to execute on your runner of choice. I'm g

Re: [Question]: Side input got stuck

Look into triggers. You can trigger windows based on number of elements and other conditions. Here are the basics: https://beam.apache.org/documentation/programming-guide/#triggers BR Med Vänlig Hälsning / Best Regards *Matyas Manninger* *Data Engineer Consultant* +46 76 05