These errors are often seen at the end of a pipeline -- they indicate that due to the failure the backend has been torn down and the attempts to report the current status have failed. If you look in the "Stack Traces" tab in the UI [1] or earlier in the Stackdriver logs, you should (hopefully) be able to find the errors that caused the failure.
In general the best place to ask future questions is probably the google-cloud-dataflow Stackoverflow topic. 1: https://cloud.google.com/dataflow/pipelines/dataflow-monitoring-intf#error-reporting On Thu, Aug 3, 2017 at 2:29 PM Randal Moore <[email protected]> wrote: > I have a batch pipeline that runs well with small inputs but fails with a > larger dataset. > Looking at stackdriver I find a fair number of the following: > > Request failed with code 400, will NOT retry: > https://dataflow.googleapis.com/v1b3/projects/cgs-nonprod/locations/us-central1/jobs/2017-08-03_13_06_11-1588537374036956973/workItems:reportStatus > > How do I investigate to learn more about the cause? > Am I reading this correctly that it is the reason the pipeline failed? > Is this perhaps the result of memory pressure? > How would I monitor the running job to determine its memory needs? > Is there a better place to query about what is likely a dataflow-centric > question? > > Thanks in advance! > rdm > >
