[
https://issues.apache.org/jira/browse/SPARK-58945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subhramit Basu Bhowmick updated SPARK-58945:
--------------------------------------------
Attachment: screenshot-1.png
> Mismatched `messageParameters` keys cause `INTERNAL_ERROR` instead of the
> intended error
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-58945
> URL: https://issues.apache.org/jira/browse/SPARK-58945
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Subhramit Basu Bhowmick
> Priority: Minor
> Labels: pull-request-available
> Attachments: screenshot-1.png
>
>
> Several Spark error paths are broken by mismatches between
> {{messageParameters}} keys and the placeholders declared in
> [{{error-conditions.json}}|https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/common/utils/src/main/resources/error/error-conditions.json].
> In addition, one legacy error class has a message template duplicated from
> its neighbour. These either surface {{INTERNAL_ERROR}} instead of the
> intended error condition, or report the wrong message text.
> h3. Mechanism
> Spark's {{StringSubstitutor}} [defaults
> to|https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/common/utils/src/main/scala/org/apache/spark/StringSubstitutor.scala#L29]
> {{enableUndefinedVariableException = true}}, so an unresolved placeholder
> throws {{IllegalArgumentException}}, which {{ErrorClassesJSONReader}}
> converts into {{SparkException.internalError}}.
> Message rendering is eager: the relevant exception constructors call
> {{SparkThrowableHelper.getMessage}} during construction (for example
> [{{common/utils/src/main/scala/org/apache/spark/SparkException.scala:288}}|https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/common/utils/src/main/scala/org/apache/spark/SparkException.scala#L288]).
> When substitution fails, the intended exception instance is never created,
> and the user receives {{INTERNAL_ERROR}} ("Undefined error message parameter
> for error class: ...") in place of the actual diagnosis.
> h3. Repro
> {code:scala}
> spark.range(1).write.option("extension", "12").csv(path)
> {code}
> Expected: {{INVALID_PARAMETER_VALUE.EXTENSION}}
> Actual: {{INTERNAL_ERROR}}
> h3. Parameter key mismatches
> Verified against {{error-conditions.json}}, ordered by reachability:
> {noformat}
> QueryExecutionErrors.scala:3214 INVALID_PARAMETER_VALUE.EXTENSION
> expects invalidValue; passes fileExtension + acceptable
> reached from CSVOptions.scala:128 via the "extension" write option
> QueryCompilationErrors.scala:4677 _LEGACY_ERROR_TEMP_2450
> expects clazz; passes invalidClass
> reached from HiveSessionStateBuilder.scala:253
> QueryExecutionErrors.scala:3242 INVALID_WRITER_COMMIT_MESSAGE
> expects detail; passes details
> reached from PythonBatchWriterFactory.scala:58,62
> H2Dialect.scala:230 TABLE_OR_VIEW_NOT_FOUND
> missing searchPath; classifyException receives only pre-rendered
> strings from JDBCTableCatalog.scala:104-106, so no Spark-side search
> path exists at that point
> StateStoreErrors.scala:399
> STATE_STORE_COLUMN_FAMILY_SCHEMA_INCOMPATIBLE
> expects colFamilyName/oldSchema/newSchema
> passes columnFamilyName/oldColumnFamilySchema/newColumnFamilySchema
> NOTE: the stateStoreColumnFamilyMismatch factory currently has no
> callers, so this one is latent rather than user-facing today.
> {noformat}
> h3. Duplicated message template
> {{_LEGACY_ERROR_TEMP_3069}} and {{_LEGACY_ERROR_TEMP_3070}} have
> byte-identical message templates. 3069
> ([{{FileSourceStrategy.scala:271}}|https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala#L271])
> is the reserved-column-name collision and its text is correct. 3070
> ([{{FileSourceStrategy.scala:291}}|https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala#L291])
> is the unrecognized-metadata-field fallback and inherited 3069's text during
> the error-class migration in SPARK-46351.
> Here the call site is not at fault - its {{Map("field" -> field.toString)}}
> was already correct. The template is the bug, and 3070 currently reports a
> reserved-column-name collision that never occurred. Before SPARK-46351 this
> branch threw {{s"Unrecognized file metadata field: $field"}}.
> h3. Why these survived
> None of these error paths are exercised by tests. Because message rendering
> is eager, any test asserting the intended error condition on an affected path
> would fail during exception construction.
> Note the site list is a lower bound. The sweep covered
> {{errorClass}}/{{messageParameters}} literal pairs in {{sql/}}, {{core/}},
> and {{mllib/}}, and skips non-literal parameter maps, non-literal error class
> arguments, and {{.java}} sources.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
