Thanks Karthi for creating the FLIP! Re: Martijn,
As I understand it, the motivation for the Prometheus Sink is for users who want to write metrics to a Prometheus remote_write endpoint as an output of their job graph, so it would be good to treat it as a first-class citizen and do it as part of Flink’s data flow. This way, we would benefit from at least once guarantees, scaling, state management. For example, a user might want to read in logs, perform some aggregations and publish it into a metrics store for visualisation. This might be a great use-case for reducing the cardinality of metrics! I think you might be referring to the metrics of the Flink job itself (e.g. CPU / memory metrics). For this use case, I would agree that we should not use this sink but instead use our metric reporters. Regards, Hong > On 16 May 2023, at 03:37, Lijie Wang <wangdachui9...@gmail.com> wrote: > > CAUTION: This email originated from outside of the organization. Do not click > links or open attachments unless you can confirm the sender and know the > content is safe. > > > > Hi Karthi, > > I think you are using a wrong FLIP id, the FLIP-229 has already be used[1]. > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Introduces+Join+Hint+for+Flink+SQL+Batch+Job > > Best, > Lijie > > Martijn Visser <martijnvis...@apache.org> 于2023年5月16日周二 04:44写道： > >> Hi Karthi, >> >> Thanks for the FLIP and opening up the discussion. My main question is: why >> should we create a separate connector and not use and/or improve the >> existing integrations with Prometheus? I would like to understand more so >> that it can be added to the motivation of the FLIP. >> >> Best regards, >> >> Martijn >> >> On Mon, May 15, 2023 at 6:03 PM Karthi Thyagarajan <kar...@karthitect.com> >> wrote: >> >>> Hello all, >>> >>> We would like to start a discussion thread on FLIP-229: Prometheus Sink >>> Connector [1] where we propose to provide a sink connector for Prometheus >>> [2] based on the Async Sink [3]. Looking forward to comments and >> feedback. >>> Thank you. >>> >>> [1] >>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Prometheus+Sink+Connector >>> [2] https://prometheus.io/ >>> [3] >>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink >>> >>
