[
https://issues.apache.org/jira/browse/SPARK-49960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Herman van Hövell reassigned SPARK-49960:
-----------------------------------------
Assignee: Chris Twiner
> Support end DeveloperAPI provided AgnosticEncoders based on
> ExpressionEncoders to allow migration
> -------------------------------------------------------------------------------------------------
>
> Key: SPARK-49960
> URL: https://issues.apache.org/jira/browse/SPARK-49960
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Chris Twiner
> Assignee: Chris Twiner
> Priority: Major
> Labels: Encoding, pull-request-available
>
> In Spark versions prior to 4.0.0-preview2 ExpressionEncoders were used
> throughout the original api. SPARK-49025 removed re-use of
> ExpressionEncoders for derivation of tuples rendering user provided
> ExpressionEncoders such as Frameless incompatible with joinWith (which only
> uses the AgnosticEncoder hierarchy). There are other likely combinations
> where the encoder is not passed into the function.
> This patch aims to add an extension AgnosticExpressionPathEncoder trait with
> added:
> {code:java}
> def toCatalyst(input: Expression): Expression
> def fromCatalyst(inputPath: Expression): Expression {code}
> functions mirroring the underlying Des/SerializerBuildHelper approach (also
> used by Frameless). In turn the BuildHelper's delegate to these functions
> when the AgnosticEncoder is an AgnosticExpressionPathEncoder.
> Moreover it aims to fix AgnosticEncoder issues identified with moving
> Frameless to AgnosticEncoders instead of ExpressionEncoders allowing users to
> migrate via both the existing Expression codebase on 4.0 and future use of
> AgnosticEncoders (limited in the short term to Classic as serialization of
> derived codecs requires closures)
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
