Hi Wencong, Thanks for kicking off the discussion! I think it would be nice to address this problem.
Is the config supposed to be publicly visible only to source connector developers but not to end users? It might be a bit unusual to have a subclass use a config to disable a public feature in the superclass. One alternative solution might be to deprecate/remove numRecordsInCounter from SourceReaderBase entirely and ask developers to maintain the counter as appropriate. For KafkaSource and FileSource, the counter can be incremented in KafkaSourceRecordEmitter#emitRecord and FileSourceRecordEmitter#emitRecord respectively. We would still need to add a config in the short term to allow a deprecation period. And the config itself will be marked as @deprecated. The downside of this alternative solution is that we need to deprecate the counting feature in SourceReaderBase. The upside is that we won't need to add a public config and the implementation might be easier to maintain in the long term. What do you think? Thanks, Dong On Tue, Jan 3, 2023 at 11:50 AM Wencong Liu <liuwencle...@163.com> wrote: > Hi devs, > > > I'd like to start a discussion about FLINK-30234: SourceReaderBase should > provide an option to disable numRecordsIn metric registration [1]. > > > As the FLINK-302345 describes, the numRecordsIn metric is pre-registered > for all sources in SourceReaderBase currently. Considering different > implementation of source reader, the definition of "record" might differ > from the one we use in SourceReaderBase, hence numRecordsIn might be > inaccurate. > > > We could introduce an public option in SourceReaderOptions used in > SourceReaderBase: > > > source.reader.metric.num_records_in.override: false > > > By default, the source reader will use the numRecordsIn metric in > SourceReaderBase. If source reader want to report to metric by self, it can > set source.reader.metric.num_records_in.override to true, which disables > the registration of numRecordsIn in SourceReaderBase and let the actual > implementation to report the metric instead. > > > Any thoughts on this? > > > [1] > https://issues.apache.org/jira/browse/FLINK-30234?jql=project%20%3D%20FLINK > > > Best, Wencong Liu
