Sweet. I have not seen that video. Cool. I'm curious about how well AWS's managed services (like the Kinesis Data Analytics managed Flink runner) handle the updates. I'd guess it is best effort from the saved state (if enabled). If this is all delegated by Beam to Flink, then this is more of a question for AWS.
On Wed, Jun 10, 2020 at 2:18 PM Austin Bennett <whatwouldausti...@gmail.com> wrote: > Hi Dan, > > AWS EMR generally runs Flink and/or Spark as supported Beam Runners. For > EMR, you might want to check compatibility for versions of Beam/Flink can > run, and the status of beam pipelines using either of those runners. > > On running Beam in AWS, had you seen: > https://www.youtube.com/watch?v=eCgZRJqdt_I > > > > Cheers, > Austin > > On Wed, Jun 10, 2020 at 2:02 PM Dan Hill <quietgol...@gmail.com> wrote: > >> No. I just sent AWS Support a message. >> >> On Wed, Jun 10, 2020 at 1:00 PM Luke Cwik <lc...@google.com> wrote: >> >>> The runner needs to support it and I'm not aware of an EMR runner for >>> Apache Beam let alone one that supports pipeline update. Have you tried >>> reaching out to AWS? >>> >>> On Wed, Jun 10, 2020 at 11:14 AM Dan Hill <quietgol...@gmail.com> wrote: >>> >>>> Hi! I found great docs about Apache Beam on Dataflow (which makes >>>> sense). I was not able to find this about AWS EMR. >>>> >>>> https://cloud.google.com/dataflow/docs/guides/updating-a-pipeline >>>> >>>> >>>> https://medium.com/google-cloud/restarting-cloud-dataflow-in-flight-9c688c49adfd >>>> >>>
