Re: Custom log appender for YARN

Hi Gyula, How about putting the appender class in lib folder, but choosing the appender only for user code through log4j.properties? That way only the user logger would be sent to Kafka. It seems what you want... Thanks, Biao /'bɪ.aʊ/ On Thu, Aug 1, 2019 at 2:25 PM Gyula Fóra wrote: > Hi Bia

Re: Custom log appender for YARN

Hi Biao, Thanks for the input! This is basically where we got ourselves as well. We are trying to avoid putting things in the lib folder so separating the loggers would be great but we don't have a solution for it yet. Thanks Gyula On Thu, 1 Aug 2019, 07:09 Biao Liu Hi Gyula, > > I guess it sh

Re: Custom log appender for YARN

Hi Gyula, I guess it should work if you put log appender jar under $flink/lib folder. There are two different kinds of classloader in your case. One is for Flink framework, the other is for user code. The framework classloader is the parent of user classloader. The parent classloader could not fi

Re: Error while running flink job on local environment

Hi Vinayak, If `StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment(2)` works for your case, you could try as below. `StreamExecutionEnvironment.setDefaultLocalParallelism(2);` `StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();`

Fwd: Flink metrics with parallel operator

Hi Sibendu, Have you checked the document of metrics[1]? It might be helpful. BTW, I think it's better to ask in user mailing list. So forward it there. 1. https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#system-scope -- Forwarded message - From: S

Re: Why is the size of each checkpoint increasing?

Hi Andrew >From Flink doc[1], there is "Flink guarantees removal only for time-based windows and not for other types, *e.g.* global windows (see Window Assigners ). ", Seems the state of the

Improve data type validation in Flink inputformats according to avro schemas

Hi all, Following this JIRA ticket opened last year : https://issues.apache.org/jira/browse/FLINK-9813 I use to run Flink, currently 1.8, in my professional activity (at DCbrain, a French B2B software editor for physical fluid networks operators) and we wish to share some of our experience with c

Custom log appender for YARN

Hi All! We are trying to configure a custom Kafka log appender for our YARN application and we hit the following problem. We included the log appender dependency in the fatjar of the application because in YARN that should be part of the system class path. However when the YARN cluster entrypoin

Re: Support priority of the Flink YARN application in Flink 1.9

Thanks for bringing this up, Boxiu. The problem make sense to me. For me the concern is should we limit the priorities to 1-9 or not. I think it would be good to open a jira issue and have the discussion there. Thank you~ Xintong Song On Wed, Jul 31, 2019 at 12:22 PM tian boxiu wrote: >

Support priority of the Flink YARN application in Flink 1.9

To: user@flink.apache.org Title: Support priority of the Flink YARN application in Flink 1.9 Hello everyone, Many of our batch jobs have changed the execution engine from spark to flink. The flink is deployed on the yarn cluster. In some scenarios, high-priority core tasks need to be submitted on

Re: Checkpoints very slow with high backpressure

Hi, For Flink 1.8 (and 1.9) the only thing that you can do, is to try to limit amount of data buffered between the nodes (check Flink network configuration [1] for number of buffers and or buffer pool sizes). This can reduce maximal throughput (but only if the network transfer is a significant