[
https://issues.apache.org/jira/browse/HUDI-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Kudinkin updated HUDI-3679:
----------------------------------
Description:
Currently we have inconsistent models of aliasing Spark version in the bundle
name ("spark3", "spark3.1.x", "spark3.2.1", etc).
To avoid confusion in the community, and make things consistent following
compatibility pledge is proposed:
+Hudi bundles will stay compatible with _all patch versions_ of Spark w/in
single minor version branch (ie Spark 3.1.x, 3.2.x, etc)+
Such compatibility model will be correspondingly reflected in the bundle names:
corresponding spark minor version branch will be aliased in the bundle name to
signal its compatibility with Spark versions w/in such branch ("spark3.1" for
Spark 3.1.x, etc)
To transition to the new model following steps are proposed:
# Add new artifacts carrying Spark minor versions in the bundle names (for ex,
"spark3.2", "spark3.1", "spark2.4") in 0.11
# Deprecate single and triple digit bundles ("spark3", "spark2", "spark3.2.1")
in favor of double-digit in 0.11
# Discontinue deprecated bundles (spark3, spark2, etc) from 0.12
was:
Given that we’re planning to align with the scheme of pledging to make Hudi
bundles be compatible with _all patch versions_ of Spark’s minor branch, the
proposal is like following: # Keep existing naming for Spark bundles in 0.11
# Add new artifacts carrying Spark minor versions in the bundle names (for ex,
spark3.2, spark3.1, spark2.4) in 0.11
# Deprecate single digit bundles (spark3, spark2, spark) in favor of
double-digit in 0.11
# Remove spark3, spark2, etc in 0.12
> Spark Compatibility Model
> -------------------------
>
> Key: HUDI-3679
> URL: https://issues.apache.org/jira/browse/HUDI-3679
> Project: Apache Hudi
> Issue Type: Epic
> Reporter: Alexey Kudinkin
> Assignee: Alexey Kudinkin
> Priority: Major
>
> Currently we have inconsistent models of aliasing Spark version in the bundle
> name ("spark3", "spark3.1.x", "spark3.2.1", etc).
> To avoid confusion in the community, and make things consistent following
> compatibility pledge is proposed:
> +Hudi bundles will stay compatible with _all patch versions_ of Spark w/in
> single minor version branch (ie Spark 3.1.x, 3.2.x, etc)+
>
> Such compatibility model will be correspondingly reflected in the bundle
> names: corresponding spark minor version branch will be aliased in the bundle
> name to signal its compatibility with Spark versions w/in such branch
> ("spark3.1" for Spark 3.1.x, etc)
> To transition to the new model following steps are proposed:
> # Add new artifacts carrying Spark minor versions in the bundle names (for
> ex, "spark3.2", "spark3.1", "spark2.4") in 0.11
> # Deprecate single and triple digit bundles ("spark3", "spark2",
> "spark3.2.1") in favor of double-digit in 0.11
> # Discontinue deprecated bundles (spark3, spark2, etc) from 0.12
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
