HI,
I have pretty same setup. Regarding Terraform and DataFlow on GCP:
- Terraform apply does check if there is a DataFlow job running with
same `job_name`
- if there is not - it does create a new one and waits till its in
"running" state
- if there is one already - it does try to update t
The deployment in the job is made by terraform. I verified and seems that
terraform do it incorrectly under the hood because it stop the current job
and starts and new one. Thanks for the information !
On Mon, 15 Apr 2024 at 6:42 PM Robert Bradshaw via user <
user@beam.apache.org> wrote:
> Are yo
Are you draining[1] your pipeline or simply canceling it and starting a new
one? Draining should close open windows and attempt to flush all in-flight
data before shutting down. For PubSub you may also need to read from
subscriptions rather than topics to ensure messages are processed by either
one
Hi guys. Good morning.
I haven't done some test in apache beam over data flow in order to see if i
can do an hot update or hot swap meanwhile the pipeline is processing a
bunch of messages that fall in a time window of 10 minutes. What I saw is
that when I do a hot update over the pipeline and cur
