HeartSaVioR commented on PR #49983: URL: https://github.com/apache/spark/pull/49983#issuecomment-2664154886
@dongjoon-hyun Let me clarify a bit. 1. I have claimed that the config is not something user (even admin) would understand what it is and try to flip. That said, removing this config does not matter to me at all and I'm +1 to remove in Spark 4.0.0. 2. The only issue is for users who started to run their query in Spark 3.5.4, because the incorrect config is put to offset log and we shouldn't ignore this. I've added the logic to migrate to the incorrect config to new config when we read from offset log (and offset log for further microbatches will follow the new config). This logic should remain for multiple minor releases, so while we can discontinue supporting the incorrect config, we can't remove the incorrect config "key" from the codebase. Please let me know if you are not on the same page in above, or have any question for this. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
