Thanks for the input. > From a quick glance it seems like the incorrect config would just be ignored from the checkpoint, and the new config would just be applied with the default value going forward.
That's not how it works. https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/OffsetSeq.scala#L128 This is the default value of config when the checkpoint does not contain such a config in its offset log. The default value will be flipped. This is to respect backward compatibility as the QO issue is "selectively" applied and the bug would have already impacted the query. That said, we fixed the QO issue in Spark 3.5.4, but for streaming queries which started from Spark 3.5.4, upgrading to Spark 4.0.0+ will "disable" the QO fix, exposing the query to the risk of sudden query failure and zero way to fix the checkpoint. Hope this clarifies your question. Thanks, Jungtaek Lim (HeartSaVioR) On Mon, Mar 10, 2025 at 10:06 PM Adam Binford <adam...@gmail.com> wrote: > As someone who has a lot of streams that have been restarted with 3.5.4, I > would prefer not to have to restart everything with 3.5.5 but it's > definitely doable. But my question is what is the actual behavior if the > migration logic was removed? From a quick glance it seems like the > incorrect config would just be ignored from the checkpoint, and the new > config would just be applied with the default value going forward. Does > something actually hard fail somewhere? Or is the only concern if someone > overrode the default value for the incorrect config, that override wouldn't > be kept going forward automatically? > > Adam > > On Fri, Mar 7, 2025 at 2:08 PM Dongjoon Hyun <dongj...@apache.org> wrote: > >> Thank you for leading the discussion, Jungtaek. >> >> +1 for voting because we couldn't build a unanimous consensus on this >> specific topic. >> >> Thanks, >> Dongjoon. >> >> On 2025/03/07 09:15:39 Jungtaek Lim wrote: >> > I'll need to start VOTE to move this forward. >> > >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >> > > -- > Adam Binford >
