Re: Registering Protobuf schema

2020-08-18 Thread Brian Hulette
Hi Robert, Sorry for the late reply on this. I think you should be able to do this by registering it in your pipeline's SchemaRegistry manually, like so: Pipeline p; p.getSchemaRegistry().registerSchemaProvider(Fx.class, ProtoMessageSchema.class); Of course this isn't quite as nice as just ad

Re: Staged PIP package mysteriously ungzipped, non-installable inside the worker

2020-08-18 Thread Eugene Kirpichov
Thanks all! Sent https://github.com/apache/beam/pull/12619 to cherrypick into 2.24. On Mon, Aug 17, 2020 at 3:37 PM Robert Bradshaw wrote: > I checked Java, it looks like the way things are structured we do not > have that bug there. > > On Mon, Aug 17, 2020 at 3:31 PM Robert Bradshaw > wrote:

Removing Duplicates from Sliding Window

2020-08-18 Thread Mohil Khare
Hello All, Firstly I am using beam java sdk 2.23.0. I have a use case where I continuously read streaming data from Kafka and dump output to Elasticsearch after doing a bunch of PTransforms. One such transform depends on the number of requests we have seen so far in the last one hour (Last one h