Re: [DISCUSS] FLIP-386: Support adding custom metrics in Recovery Spans

2023-11-22 Thread Piotr Nowojski
Hi, I have started the voting thread for this FLIP as well after modifying according to Roman's suggestions. If someone has some other suggestions, please don't hesitate to raise them here regardless of the ongoing vote :) Best, Piotrek śr., 22 lis 2023 o 12:37 Piotr Nowojski napisał(a): > Ok,

Re: [DISCUSS] FLIP-386: Support adding custom metrics in Recovery Spans

2023-11-22 Thread Piotr Nowojski
Ok, good point about dropping the aggregation method from the API. We can always add it in the future if it will ever be needed. Thanks for your input! Best, Piotrek śr., 22 lis 2023 o 01:53 Roman Khachatryan napisał(a): > Thanks for clarifying, I see your points (although reporting metrics as

Re: [DISCUSS] FLIP-386: Support adding custom metrics in Recovery Spans

2023-11-21 Thread Roman Khachatryan
Thanks for clarifying, I see your points (although reporting metrics as spans still seems counter intuitive to me). As for the aggregation, I'm concerned that it might be unnecessarily ambiguous: where the aggregation is performed (JM/TM); across what (tasks/time); and which aggregation should be

Re: [DISCUSS] FLIP-386: Support adding custom metrics in Recovery Spans

2023-11-20 Thread Piotr Nowojski
Hi Roman! > 1. why the existing MetricGroup interface can't be used? It already had > methods to add metrics and spans ... That's because of the need to: a) associate the spans to specifically Job's initialisation b) we need to logically aggregate the span's attributes across subtasks. `MetricGr

Re: [DISCUSS] FLIP-386: Support adding custom metrics in Recovery Spans

2023-11-16 Thread Roman Khachatryan
Thanks for the proposal, Can you please explain: 1. why the existing MetricGroup interface can't be used? It already had methods to add metrics and spans ... 2. IIUC, based on these numbers, we're going to report span(s). Shouldn't the backend report them as spans? 3. How is the implementation s

[DISCUSS] FLIP-386: Support adding custom metrics in Recovery Spans

2023-11-07 Thread Piotr Nowojski
(Fixing topic) wt., 7 lis 2023 o 09:40 Piotr Nowojski napisał(a): > Hi all! > > I would like to start a discussion on a follow up of FLIP-384: Introduce > TraceReporter and use it to create checkpointing and recovery traces [1]: > > *FLIP-386: Support adding custom metrics in Recovery Spans [2]*