Jungtaek Lim created SPARK-60181:
------------------------------------
Summary: Guard stream-stream join state format and RocksDB merge
operator compatibility
Key: SPARK-60181
URL: https://issues.apache.org/jira/browse/SPARK-60181
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 4.3.0, 4.4.0
Reporter: Jungtaek Lim
Recent stream-stream state format versions require RocksDB state store provider
to be available, but the default state store provider is HDFS so it could be
possible that both configs are not set for the same.
In addition, stream-stream state format version 4 requires RocksDB's merge
operator version to 2, which can happen the same.
In both cases, it is better for the engine to adjust the configs to make the
query run rather than leaving the join operator to be broken.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
