Hi dev, I'm really tired of the discussion which does not move forward because the argument is not backed by strict ASF policy. We debate based on the interpretation of ASF policy by individuals, which I think makes zero sense.
I really thought about this a lot how to resolve this, and now I'm open to have a hack on the migration logic to eliminate the main concern on the debate, which I can think like following: 1) Checks the config string via pattern, like the string contains ".optimizer.pruneFiltersCanPruneStreamingSubplan" at the end and the string is longer than "spark.sql.optimizer.pruneFiltersCanPruneStreamingSubplan" (or strictly 12 chars longer). 2) Encode the incorrect config name with any hashing algorithm, and use this to compare with. We do not document about the origin string, but probably leave the offending ticket to at least figure out when we forgot about what was the origin string by ourselves when we ever debug this. 3) etcetc (I'm open for better ideas, except just removing the migration logic.) In overall, indirect comparison. This definitely complicates the logic, but the logic is really just 4 lines to begin with, and maybe it will be just 10 lines or so, so it's still something manageable. It might be slightly tricky on testing, but this is a lot easier than debating in there. I'm now regretting not allowing myself to introduce a hack in earlier days - we should have saved 3 weeks. If we are OK to allow a bit of indirect checking on the logic to just remove out the debate, I'm happy to do that. It's obvious that we can just leave this migration logic much longer than what I was proposing, because we eliminate the main concern. I'm open to hear about support and objections. Thanks, Jungtaek Lim (HeartSaVioR)
