ericm-db commented on code in PR #49277:
URL: https://github.com/apache/spark/pull/49277#discussion_r1914301153
##########
common/utils/src/main/resources/error/error-conditions.json:
##########
@@ -4501,6 +4509,14 @@
],
"sqlState" : "42K06"
},
+ "STATE_STORE_STATE_SCHEMA_FILES_THRESHOLD_EXCEEDED" : {
+ "message" : [
+ "The number of state schema files <numStateSchemaFiles> exceeds the
maximum number of state schema files for this query: <maxStateSchemaFiles>.",
+ "Offending column families: <colFamilyNames>",
+ "Please set 'spark.sql.streaming.stateStore.stateSchemaFilesThreshold'
to a higher number, or revert state schema modifications"
Review Comment:
Yeah it is a short term solution - we plan to enable a full rewrite to limit
the number of schemas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
