+user
On Thu, Aug 20, 2020 at 9:47 AM Luke Cwik wrote:
> Are you using Dataflow runner v2[1]?
>
> If so, then you can use:
> --number_of_worker_harness_threads=X
>
> Do you know where/why the OOM is occurring?
>
> 1:
> https://cloud.google.com/dataflow/docs/guides/deploying-a-pipeline#dataflow-
Do you mean I can put my simple pipeline multiple times for all topics in
one dataflow job ?
Yes
Is there any side effect having multiple independent DAG on one DF job ?
Not really. It is more about pipeline complexity, logging, debugging,
monitoring which become more complex.
And also why the Tu
Hi Lucas,
> Not really. It is more about pipeline complexity, logging, debugging,
> monitoring which become more complex.
Should I use a different consumer group or should I use the same consumer
group ? And also How Autoscaling will decide worker count ?
What do you mean by it's not working pr
