Re: Checkpointing on Google Cloud Dataflow Runner

I looked into snapshots and they do seem useful for providing a means to save state and resume, however they aren't as seamless as I was hoping for with the automatic checkpointing that is supported by other runners. It looked like snapshots would be user initiated and would pause the pipeline whil

Re: Checkpointing on Google Cloud Dataflow Runner

Snapshots are expected to happen nearly instantaneously. While processing is paused while the snapshot is in progress, the pause should usually be very brief. It's true that Dataflow does not support automated snapshots - you would have to create them yourself using a cron. Checkpoints on Flink ar

[Question] Beam SQL failed with NPE

Hi There, I am using beam 2.35.0 to build a simple sql based pipeline but got below error. I am trying to debug into the code and could not found this class: org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.BuiltInMetadata.ExplainVisibility.Handler. Not sure if any lib missin

Re: [Question] Beam SQL failed with NPE

Hi Zheng, Could you share a minimal example that reproduces the issue? Also, have you tried using Beam >2.35.0? I'm curious if this happens in the 2.41.0 release as well. Brian On Tue, Aug 30, 2022 at 11:10 AM Zheng Ni wrote: > Hi There, > > I am using beam 2.35.0 to build a simple sql based pi

Re: [Question] Beam SQL failed with NPE

Hi, Use apache-beam==2.37.0 On Wed, 31 Aug 2022 at 00:02, Brian Hulette via user wrote: > Hi Zheng, > Could you share a minimal example that reproduces the issue? > Also, have you tried using Beam >2.35.0? I'm curious if this happens in > the 2.41.0 release as well. > > Brian > > On Tue, Aug 30

Re: [question] Good Course to learn beam

+dev for additional visibility/input On Mon, Aug 29, 2022 at 11:10 AM Leandro Nahabedian via user < user@beam.apache.org> wrote: > Hi community! > > I'm looking for a good course to learn apache beam and I saw this one >

Re: [question] Good Course to learn beam

Hi, The packt ebook is a lot better, guide yourself practically as you read the book. Plus, it is from beam committer. https://www.packtpub.com/product/building-big-data-pipelines-with-apache-beam/9781800564930 Thanks, Adeel On Tue, 30 Aug 2022 at 22:06, Evan Galpin wrote: > > +dev for additio

Re: [Question] Beam SQL failed with NPE

Hi Brain & Singh, Thanks for the reply. Actually, beam version from 2.35.0 to 2.41.0 uses the same calcite version, which is 1.28.0. So i do not think there is any difference between them. but I will give it a try. Basides, i found beam try to load class "org.apache.beam.vendor.calcite.v1_28_0.o

Re: [Question] Beam SQL failed with NPE

Seems "org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.BuiltInMetadata.ExplainVisibility.Handler" is not a valid class name passed to classloader. Java Doc: Any class name provided as a String parameter to me

Re: [question] Good Course to learn beam

Hi, Yeah I agree this book covers pretty much what you need as beginner, though I would suggest building stuff along reading. On Wed, 31 Aug 2022 at 03:08, Adeel wrote: > Hi, > > The packt ebook is a lot better, guide yourself practically as you > read the book. Plus, it is from beam committer.