Hello, Can you please share view that for “file system sources”, how to create custom metrices e.g. no. of corrupt records count?
Using Flink file source API as in below mail and decoding parquet formatted data. Able to count corrupt records but how to give it to Flink? Rgds, Kamal From: Kamal Mittal via user <user@flink.apache.org> Sent: 07 June 2023 05:48 PM To: Martijn Visser <martijnvis...@apache.org> Cc: Kamal Mittal via user <user@flink.apache.org> Subject: RE: Parquet decoding exception - Flink 1.16.x Hello, Metrices link given in below mail doesn’t give any way to create metrices for source function right? I am using below Flink API to read/decode parquet data, query is where exception can be caught for error like “decoding exception” from internal parquet API like “AvroParquetReader” and create metrices for corrupt records? FileSource.FileSourceBuilder<?> source = FileSource.forRecordStreamFormat(streamformat, path); //streamformat is of type - AvroParquetRecordFormat Please suggest. Rgds, Kamal From: Martijn Visser <martijnvis...@apache.org<mailto:martijnvis...@apache.org>> Sent: 07 June 2023 03:39 PM To: Kamal Mittal <kamal.mit...@ericsson.com<mailto:kamal.mit...@ericsson.com>> Cc: Kamal Mittal via user <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Re: Raise alarm for corrupt records Hi Kamal, Documentation on the metrics can be found at https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/ Best regards, Martijn On Wed, Jun 7, 2023 at 10:13 AM Kamal Mittal via user <user@flink.apache.org<mailto:user@flink.apache.org>> wrote: Hello, Thanks for quick reply. I am using parquet encoder/decoder and during decoding if any corrupt record comes then need to raise alarm and maintain metrices visible over Flink Metrices GUI. So any custom metrices can be created in Flink? Please give some reference of any such documentation. Rgds, Kamal From: Martijn Visser <martijnvis...@apache.org<mailto:martijnvis...@apache.org>> Sent: 07 June 2023 12:31 PM To: Kamal Mittal <kamal.mit...@ericsson.com<mailto:kamal.mit...@ericsson.com>> Cc: user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Raise alarm for corrupt records Hi Kamal, No, but it should be straightforward to create metrics or events for these types of situations and integrate them with your own alerting solution. Best regards, Martijn On Wed, Jun 7, 2023 at 8:25 AM Kamal Mittal via user <user@flink.apache.org<mailto:user@flink.apache.org>> wrote: Hello Community, Is there any way Flink provides out of box to raise alarm for corrupt records (e.g. due to decoding failure) in between of running data pipeline and send this alarm to outside of task manager process? Rgds, Kamal
