[Proposal]- Go SDK add a function to convert from m.Time to time.Time

I’ve noticed in the mtime package (go/pkg/beam/core/graph/mtime/time.go), there is a function FromTime which takes a time.Time and returns a mtime.Time. In my own project I’ve needed to convert to time.Time and to do so I needed to first convert to milliseconds and the use those milliseconds to

Re: [Proposal]- Go SDK add a function to convert from m.Time to time.Time

Thanks guys, I was looking at the 2.38 version and that’s was why I didn’t see Danny’s change. Do you know if there are small changes that would be a good first issue to tackle? Thanks, Kane On 6 Jun 2022, at 16:12, Danny McCormick mailto:dannymccorm...@google.com>> wrote: Hey Kane, I also

[Questions] KafkaIO SplittableDoFn offset managment

Hi, I am trying to use KafkaIO with the new SplittableDoFn, but I am having trouble understanding how offset management is supposed to work (using beam version 2.36.0). - In the comments of ReadFromKafkaDoFn it is stated that the initial range for KafkaSourceDescriptor startOffset can take t

Re: [EXTERNAL] Re: [Questions] KafkaIO SplittableDoFn offset managment

Hi John,Thanks for your Jean Wisser Data Engineer [https://www.flowtraders.com/sites/flowtraders-corp/files/images/flow-traders-logo-mv.png] Flow Traders B.V. T: +31 20 799 6497 F: +31 20 799 6780 Jacob Bontiusplaats 9 1018 LL Amsterdam Nederland www.flowtraders.com

Re: [EXTERNAL] Re: [Questions] KafkaIO SplittableDoFn offset managment

Hi John, Thanks for your answer. I have tried using the latest version 2.39.0. While I see the consumer group in KafkaCommitOffset is now correct and I see offsets being committed to kafka, each time I restart the pipeline it still start again from the first offset and not the latest committe

How to configure external service for Kafka IO to run the flink job in k8s

Hi Folks, I am trying to set up the Beam environment to run our Python pipeline which reads data from Kafka. According to some previous thread , it seems like the Java SDK harness doesn't work with PROCESS environment_type, and thu