Thanks for starting the discussion, Wencong! +1 for the proposal. We can't promise that the definition of "record" in all source implementations totally matches with SourceReaderBase. For example CDC connector source [1] extracts records in the RecordEmitter instead, so the predefined "numRecordsIn" metric in SourceReaderBase could not reflect the actual throughput.
[1] https://github.com/ververica/flink-cdc-connectors/blob/release-2.3.0/flink-connector-mysql-cdc/src/main/java/com/ververica/cdc/connectors/mysql/source/reader/MySqlSourceReader.java Best, Qingsheng Ververica (Alibaba) On Tue, Jan 3, 2023 at 11:50 AM Wencong Liu <liuwencle...@163.com> wrote: > Hi devs, > > > I'd like to start a discussion about FLINK-30234: SourceReaderBase should > provide an option to disable numRecordsIn metric registration [1]. > > > As the FLINK-302345 describes, the numRecordsIn metric is pre-registered > for all sources in SourceReaderBase currently. Considering different > implementation of source reader, the definition of "record" might differ > from the one we use in SourceReaderBase, hence numRecordsIn might be > inaccurate. > > > We could introduce an public option in SourceReaderOptions used in > SourceReaderBase: > > > source.reader.metric.num_records_in.override: false > > > By default, the source reader will use the numRecordsIn metric in > SourceReaderBase. If source reader want to report to metric by self, it can > set source.reader.metric.num_records_in.override to true, which disables > the registration of numRecordsIn in SourceReaderBase and let the actual > implementation to report the metric instead. > > > Any thoughts on this? > > > [1] > https://issues.apache.org/jira/browse/FLINK-30234?jql=project%20%3D%20FLINK > > > Best, Wencong Liu
