Applying Custom metrics

Hi I have questions regarding making my own custom metrics. When exactly is the class RichMapFunction’s map(value) method called/invoked, and what “value” will be passed/expected as an argument to this map(value) method? Does the RichMapFunction’s map() method have any relation to the transforma

Re: Maximum query and refresh rate for metrics from REST API

> On 9/17/2020 12:22 AM, Piper Piper wrote: > > Hello, > > > > What is the recommended way to get metrics (such as CPU, Memory and > > user defined meters and gauges) at the highest frequency rate (i.e. > > with the highest/fastest refresh rate) such as every 500

Maximum query and refresh rate for metrics from REST API

Hello, What is the recommended way to get metrics (such as CPU, Memory and user defined meters and gauges) at the highest frequency rate (i.e. with the highest/fastest refresh rate) such as every 500 milliseconds or less? Is there any rate limiting by default on querying the REST API for metrics?

Measure CPU utilization

Hello, What is the best way to measure the CPU utilization of a TaskManager in Flink, as opposed to using Linux's "top" command? Is querying the REST endpoint http://:/taskmanagers//metrics?get=Status.JVM.CPU.Load\ the best option? Roman's reply (copied below) from the archives suggests that it r

Re: How to use Flink IDE

e artifact, making it a fat jar and deploy it. >> >> >> Steps: >> >> 1. Open main class run/debug configurations >> 2. Click on Include dependencies with Provided scope. >> 3. Apply >> >> >> Thanks, >> Narasimha >> >> >> >

How to use Flink IDE

Hi, Till now, I have only been using Flink binaries. How do I setup Flink in my IntelliJ IDE so that while running/debugging my Flink application program I can also step into the Flink source code? Do I first need to import Flink's source repository into my IDE and build it? Thanks, Piper

Re: Ververica Flink training resources

using a problem? > > Best, > David > > On Sun, Aug 23, 2020 at 12:20 AM Piper Piper wrote: > >> Hi Flink community, >> >> I have two questions regarding the Ververica Flink Training resources. >> >> 1. In the official Flink documentation, the hyperli

Ververica Flink training resources

Hi Flink community, I have two questions regarding the Ververica Flink Training resources. 1. In the official Flink documentation, the hyperlinks to the github sites for the exercises in the "Learn Flink" section are not working. If possible, please provide me with the correct links for the exerc

Re: Java implementations of Streaming applications for Flink

maintain a number of testing jobs. They > are rather artificial, but you still might find some useful things here and > there: https://github.com/apache/flink/tree/master/flink-end-to-end-tests > > Best, > Robert > > > On Mon, Feb 24, 2020 at 9:38 AM Piper Piper wrote: > >>

Java implementations of Streaming applications for Flink

Hi all, The examples in the Flink github repo do not seem to include many standard streaming applications compared to the batch examples. Where can I get standard (recommended) Java implementations of “Streaming” applications for Flink, that are clearly: (1) CPU-intensive, like streaming PageRank

Re: Flink+YARN HDFS replication factor

Please disregard my previous email. I found the answer online. I thought writing data to local disk automatically meant the data would be persisted to HDFS. However, Spark writes data (in between shuffles) to local disk only. Thanks On Thu, Jan 30, 2020, 2:00 PM Piper Piper wrote: > Hi T

Re: Flink+YARN HDFS replication factor

g blobs (Jars and job > artifacts) on HDFS if configured so. > > Cheers, > Till > > On Wed, Jan 29, 2020 at 7:07 AM Piper Piper wrote: > >> Hello, >> >> When using Flink+YARN (with HDFS) and having a long running Flink session >> (mode) cluster with a Flink cli

Flink+YARN HDFS replication factor

Hello, When using Flink+YARN (with HDFS) and having a long running Flink session (mode) cluster with a Flink client submitting jobs, the HDFS could have a replication factor greater than 1 (example 3). So, I would like to know when and how any of the data (like event-data or batch-data) or code (

Re: Flink on YARN: Where to install Flink binaries?

des. The > client is the machine from which you start the Flink cluster. The client > machine needs to have access to the Hadoop/Yarn cluster. Hence you should > configure the HADOOP_CONF_DIR to the Hadoop configuration. > > Cheers, > Till > > On Wed, Dec 4, 2019 at 11:04 AM

Flink on YARN: Where to install Flink binaries?

Hello, I have a YARN/Hadoop 2.7.6 cluster, on which I plan to run Flink in Job mode using: Flink 1.9.1 (with Flink application programs written in Java) Prebundled Hadoop 2.7.5 Question 1: Which scala version must I choose for the Flink 1.9.1 binary (2.11 or 2.12)? Secondly, I had read a documen

Re: Dynamically creating new Task Managers in YARN

l fill the slot request, new TaskManagers will be > started. > Did you find some exceptions? > > Best, > Yang > > Piper Piper 于2019年11月23日周六 上午8:52写道： > >> Hello Yang, >> >> Thank you for the explanation! >> >> I want to control the amount of T

Re: Dynamically creating new Task Managers in YARN

n. > > I'm just curious why do you want to control the amounts of taskmanagers. > Because they are > always allocated on demand. > > > Best, > Yang > > Piper Piper 于2019年11月22日周五 上午11:02写道： > >> Thank you, I will check it out. >> >> On Thu, No

Re: Metrics for Task States

Piper Piper wrote: > Thank you, Kelly! > > On Thu, Nov 21, 2019 at 4:06 PM Kelly Smith > wrote: > >> Hi Piper, >> >> >> >> The repro is pretty simple: >> >>- Submit a job with parallelism set higher than YARN has resources to >>

Re: Dynamically creating new Task Managers in YARN

-yarn. > > Best, > Jingsong Lee > > > On Thu, Nov 21, 2019 at 8:17 PM Piper Piper wrote: > >> Hi Jingsong, >> >> Thank you for your reply! >> >> >Is this what you want? Piper. >> >> Yes. This is exactly what I want. >> >> Is ther

Re: Metrics for Task States

> What I’m looking for is a way to detect when I am in this state using > Flink metrics (ideally the count of tasks in each state for better > observability). > > > > Does that make sense? > > > > Thanks, > > Kelly > > > > *From: *Piper Piper >

Re: Metrics for Task States

Hello Kelly, I thought that Flink scheduler only starts a job if all requested containers/TMs are available and allotted to that job. How can I reproduce your issue on Flink with YARN? Thank you, Piper On Thu, Nov 21, 2019, 1:48 PM Kelly Smith wrote: > I’ve been running Flink in production

Re: Dynamically creating new Task Managers in YARN

;>> AFAIK, The single job (job cluster) mode is more popular than Session >>> mode. >>> >>> Because job cluster mode, Flink let YARN manage resources as far as >>> possible. And this mode can keep isolation from other jobs. >>> >>> IMO, we do

Re: Dynamically creating new Task Managers in YARN

wrote: > Hi Piper, > > Can you share more reason and details of your requirements. > > Best, > Vino > > Piper Piper 于2019年11月21日周四 上午5:48写道： > >> Hi, >> >> How can I make Flink's Resource Manager request YARN to spin up new (or >> d

Dynamically creating new Task Managers in YARN

Hi, How can I make Flink's Resource Manager request YARN to spin up new (or destroy/reclaim existing) TaskManagers in YARN containers? Preferably at runtime (i.e. dynamically). Thank you Piper