In Spotify's case we deploy streaming jobs via CI and would ideally verify compatibility as part of the build process before submitting to dataflow. Perhaps decoupled from the _running_ pipeline if we had a cache of previous pipeline versions.
Currently the user experience is poor because any merge of a change ends up being potentially distant in time from the job submission and failure due to incompatibility. There's the additional friction of a few layers of infrastructure between the developer and the job failure logs which means we can't trivially deliver the failure reason to the developer. On Fri, Oct 27, 2023 at 11:07 AM Robert Burke <rob...@frantil.com> wrote: > You raise a very good point: > > > https://github.com/apache/beam/blob/master/model/job-management/src/main/proto/org/apache/beam/model/job_management/v1/beam_job_api.proto#L54 > > The job management API does allow for the pipeline proto to be returned. > So one could get the live job, so the SDK could make reasonable decisions > before sending to the runner. > > Dataflow does have a similar API that xan be adapted. > > I am a touch concerned about spreading the update compatibility checks > around between SDKs and Runners though. But in some cases it would be > easier for the SDK, eg to ensure VersionA of a transform is used vs > VersionB, based on the existing transforma used in the job being updated. > > > On Fri, Oct 27, 2023, 7:50 AM Kellen Dye via dev <dev@beam.apache.org> > wrote: > >> > Auto is hard, because it would involve >> > querying the runner before pipeline construction, and we may not even >> > know what the runner is at this point >> >> At the point where pipeline construction will start, you should have >> access to the pipeline arguments and be able to determine the runner. What >> seems to be missing is a place to query the runner pre-construction. If >> that query could return metadata about the currently running version of the >> job, then that could be incorporated into graph construction as necessary. >> >> That same hook could be a place to for example return the >> currently-running job graph for pre-submission compatibility checks. >> >> >>
