Hi Venkata, Thank you for opening the discussion about this!
After taking a look at the YARN and Hadoop configurations, the reason why it was implemented this way is that, in case of YARN, every YARN-specific property is prefixed with "yarn.", so to get the final, YARN-side property it is enough to remove the "flink." prefix. In case of Hadoop, there are properties that not prefixed with "hadoop.", e.g. "dfs.replication" so to identify and get the Hadoop-side property it is necessary to duplicate the "hadoop" part in the properties. Taking this into consideration I would personally say -0 to this change. IMO the current behavior can be justified as giving slightly different solutions to slightly different problems, which are well documented. Handling both prefixes would complicate the parsing logic until the APIs can be removed, which as it looks at the moment would only be possible in Flink 3.0, which probably will not happen in the foreseeable future, so I do not see the benefit of the added complexity. Regarding the FLIP, in the "YARN configuration override example" part, I think you should present an example that works correctly at the moment: "flink.yarn.application.classpath" -> "yarn.application.classpath". Best, Ferenc On Friday, March 29th, 2024 at 23:45, Venkatakrishnan Sowrirajan <vsowr...@asu.edu> wrote: > > > Hi Flink devs, > > I would like to start a discussion on FLIP-XXX: Make Flink's Hadoop and > YARN configuration probing consistent > https://docs.google.com/document/d/1I2jBFI0eVkofAVCAEeajNQRfOqKGJsRfZd54h79AIYc/edit?usp=sharing. > > This stems from an earlier discussion thread here > https://lists.apache.org/thread/l2fh5shbf59fjgbt1h73pmmsqj038ppv. > > > This FLIP is proposing to make the configuration probing behavior between > Hadoop and YARN configuration to be consistent. > > Regards > Venkata krishnan
