+1 million to this.
I think this could be a real game-changer. I would even more forcefully say
update compatibility has pushed our development style has been pushed into
the "never make significant changes" or "every significant change is wildly
more complex than it should be". It forces our firs
On Fri, Oct 27, 2023, 9:09 AM Robert Bradshaw via dev
wrote:
> On Fri, Oct 27, 2023 at 7:50 AM Kellen Dye via dev
> wrote:
> >
> > > Auto is hard, because it would involve
> > > querying the runner before pipeline construction, and we may not even
> > > know what the runner is at this point
> >
On Fri, Oct 27, 2023 at 7:50 AM Kellen Dye via dev wrote:
>
> > Auto is hard, because it would involve
> > querying the runner before pipeline construction, and we may not even
> > know what the runner is at this point
>
> At the point where pipeline construction will start, you should have access
In Spotify's case we deploy streaming jobs via CI and would ideally verify
compatibility as part of the build process before submitting to dataflow.
Perhaps decoupled from the _running_ pipeline if we had a cache of previous
pipeline versions.
Currently the user experience is poor because any merg
You raise a very good point:
https://github.com/apache/beam/blob/master/model/job-management/src/main/proto/org/apache/beam/model/job_management/v1/beam_job_api.proto#L54
The job management API does allow for the pipeline proto to be returned. So
one could get the live job, so the SDK could make
> Auto is hard, because it would involve
> querying the runner before pipeline construction, and we may not even
> know what the runner is at this point
At the point where pipeline construction will start, you should have access
to the pipeline arguments and be able to determine the runner. What s
Alright, then it is clearer. Thank you for your answers!
On Thu, Oct 26, 2023, 20:36 Robert Bradshaw wrote:
> On Thu, Oct 26, 2023 at 3:59 AM Johanna Öjeling
> wrote:
> >
> > Hi,
> >
> > I like this idea of making it easier to push out improvements, and had a
> look at the PR.
> >
> > One quest
On Thu, Oct 26, 2023 at 3:59 AM Johanna Öjeling wrote:
>
> Hi,
>
> I like this idea of making it easier to push out improvements, and had a look
> at the PR.
>
> One question to better understand how it works today:
>
> The upgrades that the runners do, such as those not visible to the user, can
Regarding 3. I suspect Go wasn't changed because the PR is centering around
implementations of the Expansion Service server, not client callers. The Go
SDK doesn't yet have an expansion service.
On Thu, Oct 26, 2023, 3:59 AM Johanna Öjeling via dev
wrote:
> Hi,
>
> I like this idea of making it
Hi,
I like this idea of making it easier to push out improvements, and had a
look at the PR.
One question to better understand how it works today:
1. The upgrades that the runners do, such as those not visible to the
user, can they be initiated at any time or do they only happen in relatio
Dataflow (among other runners) has the ability to "upgrade" running
pipelines with new code (e.g. capturing bug fixes, dependency updates,
and limited topology changes). Unfortunately some improvements (e.g.
new and improved ways of writing to BigQuery, optimized use of side
inputs, a change in alg
11 matches
Mail list logo
