Thank you very much for your explanation.
I will keep it in mind.

Best,

Francesco

> Il giorno 27 mag 2020, alle ore 15:43, Piotr Nowojski <pi...@ververica.com> 
> ha scritto:
> 
> Hi Francesco,
> 
> As long as you do not set update interval of metric reporter to some very low 
> value, there should be no visible performance degradation.
> 
> Maybe worth keeping in mind is that if you jobs are bounded (they are working 
> on bounded input and they finish/complete at some point of time), the last 
> updated metric value before job completes might not necessarily reflect the 
> end state of the job. This limitation may not apply if you will be using REST 
> API, as Job Manager might be remembering the values you are looking for.
> 
> Piotrek
> 
>> On 27 May 2020, at 11:41, Francesco Ventura 
>> <francesco.vent...@campus.tu-berlin.de 
>> <mailto:francesco.vent...@campus.tu-berlin.de>> wrote:
>> 
>> Hi Piotrek,
>> 
>> Thank you for you replay and for your suggestions. Just another doubt.
>> Does the usage of metrics reporter and custom metrics will affect the 
>> performances of the running jobs in term of execution time? Since I need the 
>> information about the exact netRunTime of each job maybe using the REST APIs 
>> to get the other information will be more reliable?
>> 
>> Thank you. Best,
>> 
>> Francesco
>> 
>>> Il giorno 25 mag 2020, alle ore 19:54, Piotr Nowojski <pi...@ververica.com 
>>> <mailto:pi...@ververica.com>> ha scritto:
>>> 
>>> Hi Francesco,
>>> 
>>> Have you taken a look at the metrics? [1] And IO metrics [2] in particular? 
>>> You can use some of the pre-existing metric reporter [3] or implement a 
>>> custom one. You could export metrics to some 3rd party system, and get 
>>> JSONs from there, or export them to JSON directly via a custom metric 
>>> reporter.
>>> 
>>> Piotrek
>>> 
>>> [1] 
>>> https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html
>>>  
>>> <https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html>
>>> [2] 
>>> https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#io
>>>  
>>> <https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#io>
>>> [3] 
>>> https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#reporter
>>>  
>>> <https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#reporter>
>>> 
>>>> On 23 May 2020, at 11:31, Francesco Ventura 
>>>> <francesco.vent...@campus.tu-berlin.de 
>>>> <mailto:francesco.vent...@campus.tu-berlin.de>> wrote:
>>>> 
>>>> Hi everybody, 
>>>> 
>>>> I would like to collect the statistics and the real output cardinalities 
>>>> about the execution of many jobs as json files. I know that exist a REST 
>>>> interface that can be used but I was looking for something simpler. In 
>>>> practice, I would like to get the information showed in the WebUI at 
>>>> runtime about a job and store it as a file. I am using the 
>>>> env.getExecutionPlan() to get the execution plan of a job with the 
>>>> estimated cardinalities for each operator. However, it includes only 
>>>> estimated cardinalities and it can be used only before calling 
>>>> env.execute(). 
>>>> 
>>>> There is a similar way to extract the real output cardinalities of each 
>>>> pipeline after the execution? 
>>>> Is there a place where the Flink cluster stores the history of the 
>>>> information about executed jobs?
>>>> Developing a REST client to extract such information is the only way 
>>>> possible? 
>>>> 
>>>> I also would like to avoid adding counters to the job source code since I 
>>>> am monitoring the run time execution and I should avoid everything that 
>>>> can interfere.
>>>> 
>>>> Maybe is a trivial problem but I have a quick look around and I can not 
>>>> find the solution.
>>>> 
>>>> Thank you very much,
>>>> 
>>>> Francesco
>>> 
>> 
> 

Reply via email to