Re: Collecting operators real output cardinalities as json files

2020-05-27 Thread Francesco Ventura
Thank you very much for your explanation. I will keep it in mind. Best, Francesco > Il giorno 27 mag 2020, alle ore 15:43, Piotr Nowojski > ha scritto: > > Hi Francesco, > > As long as you do not set update interval of metric reporter to some very low > value, there should be no visible per

Re: Collecting operators real output cardinalities as json files

2020-05-27 Thread Piotr Nowojski
Hi Francesco, As long as you do not set update interval of metric reporter to some very low value, there should be no visible performance degradation. Maybe worth keeping in mind is that if you jobs are bounded (they are working on bounded input and they finish/complete at some point of time),

Re: Collecting operators real output cardinalities as json files

2020-05-27 Thread Francesco Ventura
Hi Piotrek, Thank you for you replay and for your suggestions. Just another doubt. Does the usage of metrics reporter and custom metrics will affect the performances of the running jobs in term of execution time? Since I need the information about the exact netRunTime of each job maybe using the

Re: Collecting operators real output cardinalities as json files

2020-05-25 Thread Piotr Nowojski
Hi Francesco, Have you taken a look at the metrics? [1] And IO metrics [2] in particular? You can use some of the pre-existing metric reporter [3] or implement a custom one. You could export metrics to some 3rd party system, and get JSONs from there, or export them to JSON directly via a custom