[
https://issues.apache.org/jira/browse/SPARK-58945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subhramit Basu Bhowmick updated SPARK-58945:
--------------------------------------------
Description:
Several Spark error paths are broken by mismatches between
{{messageParameters}} keys and the placeholders declared in
[{{error-conditions.json}}|https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/common/utils/src/main/resources/error/error-conditions.json].
In addition, one legacy error class has a message template duplicated from its
neighbour. These either surface {{INTERNAL_ERROR}} instead of the intended
error condition, or report the wrong message text.
h3. Mechanism
Spark's {{StringSubstitutor}} [defaults
to|https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/common/utils/src/main/scala/org/apache/spark/StringSubstitutor.scala#L29]
{{enableUndefinedVariableException = true}}, so an unresolved placeholder
throws {{IllegalArgumentException}}, which {{ErrorClassesJSONReader}} converts
into {{SparkException.internalError}}.
Message rendering is eager: the relevant exception constructors call
{{SparkThrowableHelper.getMessage}} during construction (for example
[{{common/utils/src/main/scala/org/apache/spark/SparkException.scala:288}}|https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/common/utils/src/main/scala/org/apache/spark/SparkException.scala#L288]).
When substitution fails, the intended exception instance is never created, and
the user receives {{INTERNAL_ERROR}} ("Undefined error message parameter for
error class: ...") in place of the actual diagnosis.
h3. Repro
{code:scala}
spark.range(1).write.option("extension", "12").csv(path)
{code}
Expected: {{INVALID_PARAMETER_VALUE.EXTENSION}}
Actual: {{INTERNAL_ERROR}}
!screenshot-1.png!
h3. Parameter key mismatches
Verified against {{error-conditions.json}}, ordered by reachability:
{noformat}
QueryExecutionErrors.scala:3214 INVALID_PARAMETER_VALUE.EXTENSION
expects invalidValue; passes fileExtension + acceptable
reached from CSVOptions.scala:128 via the "extension" write option
QueryCompilationErrors.scala:4677 _LEGACY_ERROR_TEMP_2450
expects clazz; passes invalidClass
reached from HiveSessionStateBuilder.scala:253
QueryExecutionErrors.scala:3242 INVALID_WRITER_COMMIT_MESSAGE
expects detail; passes details
reached from PythonBatchWriterFactory.scala:58,62
H2Dialect.scala:230 TABLE_OR_VIEW_NOT_FOUND
missing searchPath; classifyException receives only pre-rendered
strings from JDBCTableCatalog.scala:104-106, so no Spark-side search
path exists at that point
StateStoreErrors.scala:399 STATE_STORE_COLUMN_FAMILY_SCHEMA_INCOMPATIBLE
expects colFamilyName/oldSchema/newSchema
passes columnFamilyName/oldColumnFamilySchema/newColumnFamilySchema
NOTE: the stateStoreColumnFamilyMismatch factory currently has no
callers, so this one is latent rather than user-facing today.
{noformat}
h3. Duplicated message template
{{_LEGACY_ERROR_TEMP_3069}} and {{_LEGACY_ERROR_TEMP_3070}} have byte-identical
message templates. 3069
([{{FileSourceStrategy.scala:271}}|https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala#L271])
is the reserved-column-name collision and its text is correct. 3070
([{{FileSourceStrategy.scala:291}}|https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala#L291])
is the unrecognized-metadata-field fallback and inherited 3069's text during
the error-class migration in SPARK-46351.
Here the call site is not at fault - its {{Map("field" -> field.toString)}} was
already correct. The template is the bug, and 3070 currently reports a
reserved-column-name collision that never occurred. Before SPARK-46351 this
branch threw {{s"Unrecognized file metadata field: $field"}}.
h3. Why these survived
None of these error paths are exercised by tests. Because message rendering is
eager, any test asserting the intended error condition on an affected path
would fail during exception construction.
Note the site list is a lower bound. The sweep covered
{{errorClass}}/{{messageParameters}} literal pairs in {{sql/}}, {{core/}}, and
{{mllib/}}, and skips non-literal parameter maps, non-literal error class
arguments, and {{.java}} sources.
was:
Several Spark error paths are broken by mismatches between
{{messageParameters}} keys and the placeholders declared in
[{{error-conditions.json}}|https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/common/utils/src/main/resources/error/error-conditions.json].
In addition, one legacy error class has a message template duplicated from its
neighbour. These either surface {{INTERNAL_ERROR}} instead of the intended
error condition, or report the wrong message text.
h3. Mechanism
Spark's {{StringSubstitutor}} [defaults
to|https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/common/utils/src/main/scala/org/apache/spark/StringSubstitutor.scala#L29]
{{enableUndefinedVariableException = true}}, so an unresolved placeholder
throws {{IllegalArgumentException}}, which {{ErrorClassesJSONReader}} converts
into {{SparkException.internalError}}.
Message rendering is eager: the relevant exception constructors call
{{SparkThrowableHelper.getMessage}} during construction (for example
[{{common/utils/src/main/scala/org/apache/spark/SparkException.scala:288}}|https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/common/utils/src/main/scala/org/apache/spark/SparkException.scala#L288]).
When substitution fails, the intended exception instance is never created, and
the user receives {{INTERNAL_ERROR}} ("Undefined error message parameter for
error class: ...") in place of the actual diagnosis.
h3. Repro
{code:scala}
spark.range(1).write.option("extension", "12").csv(path)
{code}
Expected: {{INVALID_PARAMETER_VALUE.EXTENSION}}
Actual: {{INTERNAL_ERROR}}
h3. Parameter key mismatches
Verified against {{error-conditions.json}}, ordered by reachability:
{noformat}
QueryExecutionErrors.scala:3214 INVALID_PARAMETER_VALUE.EXTENSION
expects invalidValue; passes fileExtension + acceptable
reached from CSVOptions.scala:128 via the "extension" write option
QueryCompilationErrors.scala:4677 _LEGACY_ERROR_TEMP_2450
expects clazz; passes invalidClass
reached from HiveSessionStateBuilder.scala:253
QueryExecutionErrors.scala:3242 INVALID_WRITER_COMMIT_MESSAGE
expects detail; passes details
reached from PythonBatchWriterFactory.scala:58,62
H2Dialect.scala:230 TABLE_OR_VIEW_NOT_FOUND
missing searchPath; classifyException receives only pre-rendered
strings from JDBCTableCatalog.scala:104-106, so no Spark-side search
path exists at that point
StateStoreErrors.scala:399 STATE_STORE_COLUMN_FAMILY_SCHEMA_INCOMPATIBLE
expects colFamilyName/oldSchema/newSchema
passes columnFamilyName/oldColumnFamilySchema/newColumnFamilySchema
NOTE: the stateStoreColumnFamilyMismatch factory currently has no
callers, so this one is latent rather than user-facing today.
{noformat}
h3. Duplicated message template
{{_LEGACY_ERROR_TEMP_3069}} and {{_LEGACY_ERROR_TEMP_3070}} have byte-identical
message templates. 3069
([{{FileSourceStrategy.scala:271}}|https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala#L271])
is the reserved-column-name collision and its text is correct. 3070
([{{FileSourceStrategy.scala:291}}|https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala#L291])
is the unrecognized-metadata-field fallback and inherited 3069's text during
the error-class migration in SPARK-46351.
Here the call site is not at fault - its {{Map("field" -> field.toString)}} was
already correct. The template is the bug, and 3070 currently reports a
reserved-column-name collision that never occurred. Before SPARK-46351 this
branch threw {{s"Unrecognized file metadata field: $field"}}.
h3. Why these survived
None of these error paths are exercised by tests. Because message rendering is
eager, any test asserting the intended error condition on an affected path
would fail during exception construction.
Note the site list is a lower bound. The sweep covered
{{errorClass}}/{{messageParameters}} literal pairs in {{sql/}}, {{core/}}, and
{{mllib/}}, and skips non-literal parameter maps, non-literal error class
arguments, and {{.java}} sources.
> Mismatched `messageParameters` keys cause `INTERNAL_ERROR` instead of the
> intended error
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-58945
> URL: https://issues.apache.org/jira/browse/SPARK-58945
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Subhramit Basu Bhowmick
> Priority: Minor
> Labels: pull-request-available
> Attachments: screenshot-1.png
>
>
> Several Spark error paths are broken by mismatches between
> {{messageParameters}} keys and the placeholders declared in
> [{{error-conditions.json}}|https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/common/utils/src/main/resources/error/error-conditions.json].
> In addition, one legacy error class has a message template duplicated from
> its neighbour. These either surface {{INTERNAL_ERROR}} instead of the
> intended error condition, or report the wrong message text.
> h3. Mechanism
> Spark's {{StringSubstitutor}} [defaults
> to|https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/common/utils/src/main/scala/org/apache/spark/StringSubstitutor.scala#L29]
> {{enableUndefinedVariableException = true}}, so an unresolved placeholder
> throws {{IllegalArgumentException}}, which {{ErrorClassesJSONReader}}
> converts into {{SparkException.internalError}}.
> Message rendering is eager: the relevant exception constructors call
> {{SparkThrowableHelper.getMessage}} during construction (for example
> [{{common/utils/src/main/scala/org/apache/spark/SparkException.scala:288}}|https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/common/utils/src/main/scala/org/apache/spark/SparkException.scala#L288]).
> When substitution fails, the intended exception instance is never created,
> and the user receives {{INTERNAL_ERROR}} ("Undefined error message parameter
> for error class: ...") in place of the actual diagnosis.
> h3. Repro
> {code:scala}
> spark.range(1).write.option("extension", "12").csv(path)
> {code}
> Expected: {{INVALID_PARAMETER_VALUE.EXTENSION}}
> Actual: {{INTERNAL_ERROR}}
> !screenshot-1.png!
> h3. Parameter key mismatches
> Verified against {{error-conditions.json}}, ordered by reachability:
> {noformat}
> QueryExecutionErrors.scala:3214 INVALID_PARAMETER_VALUE.EXTENSION
> expects invalidValue; passes fileExtension + acceptable
> reached from CSVOptions.scala:128 via the "extension" write option
> QueryCompilationErrors.scala:4677 _LEGACY_ERROR_TEMP_2450
> expects clazz; passes invalidClass
> reached from HiveSessionStateBuilder.scala:253
> QueryExecutionErrors.scala:3242 INVALID_WRITER_COMMIT_MESSAGE
> expects detail; passes details
> reached from PythonBatchWriterFactory.scala:58,62
> H2Dialect.scala:230 TABLE_OR_VIEW_NOT_FOUND
> missing searchPath; classifyException receives only pre-rendered
> strings from JDBCTableCatalog.scala:104-106, so no Spark-side search
> path exists at that point
> StateStoreErrors.scala:399
> STATE_STORE_COLUMN_FAMILY_SCHEMA_INCOMPATIBLE
> expects colFamilyName/oldSchema/newSchema
> passes columnFamilyName/oldColumnFamilySchema/newColumnFamilySchema
> NOTE: the stateStoreColumnFamilyMismatch factory currently has no
> callers, so this one is latent rather than user-facing today.
> {noformat}
> h3. Duplicated message template
> {{_LEGACY_ERROR_TEMP_3069}} and {{_LEGACY_ERROR_TEMP_3070}} have
> byte-identical message templates. 3069
> ([{{FileSourceStrategy.scala:271}}|https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala#L271])
> is the reserved-column-name collision and its text is correct. 3070
> ([{{FileSourceStrategy.scala:291}}|https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala#L291])
> is the unrecognized-metadata-field fallback and inherited 3069's text during
> the error-class migration in SPARK-46351.
> Here the call site is not at fault - its {{Map("field" -> field.toString)}}
> was already correct. The template is the bug, and 3070 currently reports a
> reserved-column-name collision that never occurred. Before SPARK-46351 this
> branch threw {{s"Unrecognized file metadata field: $field"}}.
> h3. Why these survived
> None of these error paths are exercised by tests. Because message rendering
> is eager, any test asserting the intended error condition on an affected path
> would fail during exception construction.
> Note the site list is a lower bound. The sweep covered
> {{errorClass}}/{{messageParameters}} literal pairs in {{sql/}}, {{core/}},
> and {{mllib/}}, and skips non-literal parameter maps, non-literal error class
> arguments, and {{.java}} sources.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
