Ismaël Mejía created SPARK-58673:
------------------------------------
Summary: Upgrade Avro to 1.12.2
Key: SPARK-58673
URL: https://issues.apache.org/jira/browse/SPARK-58673
Project: Spark
Issue Type: Dependency upgrade
Components: Build
Affects Versions: 5.0.0
Reporter: Ismaël Mejía
Upgrade the Avro dependency from 1.12.1 to 1.12.2.
Notable changes required by the upgrade:
- Avro 1.12.2 enforces a ClassSecurityValidator / SERIALIZABLE_PACKAGES
restriction when deserializing schemas. Spark tests that ship reader
schemas to executors need "org.apache.spark" allowlisted via
org.apache.avro.SERIALIZABLE_PACKAGES.
- AVRO-4257 changes Schema.readResolve() to use the non-validating internal
parser (JsonSchemaParser.parseInternal). As a result, a reader schema that
carries invalid but unused default values is no longer re-validated after
crossing the driver/executor boundary, so reads succeed end to end. This is
consistent with the goal of SPARK-39775; the corresponding test is updated
to assert the read returns the expected rows instead of failing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
