The only logging options that Dataflow exposes today limit what gets logged and not anything about how many rotated logs there are or how big they are.
All Dataflow logging options are available here: https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowWorkerLoggingOptions.java On Wed, Aug 28, 2019 at 7:00 PM Talat Uyarer <[email protected]> wrote: > Hi All, > > This is my first message for this maillist. Please let me know if I am > sending this message to wrong maillist. > > My stream processing job are running on Google Cloud Dataflow engine. For > logging I am using Stackdriver. I added runtime slf4j-jdk14 and slf4j-api > to enable to stackdriver. However my pipeline create lots of logs and my > instances are getting out of space issue. I checked log rotating and limit > count of log files. I could not find any settings for them. How can I set > this settings. > > Thanks > >
