[
https://issues.apache.org/jira/browse/SPARK-58820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uroš Bojanić updated SPARK-58820:
---------------------------------
Description:
*Q1. What are you trying to do? Articulate your objectives using absolutely no
jargon.*
Add a new Spark SQL data type *DECFLOAT* that stores decimal numbers with a
flexible decimal point (floating point decimal numbers). Each value keeps up to
a fixed number of significant decimal digits and its own exponent, so one
column can hold both very large integers and very small fractions without
picking a single fixed scale for the whole column.
The type follows the IEEE 754 decimal floating-point formats:
||*SQL type*||*IEEE format*||*Significant digits*||*Storage width*||
|DECFLOAT(16)|decimal64|16|8 bytes|
|DECFLOAT(34)|decimal128|34|16 bytes|
Bare DECFLOAT means DECFLOAT(34).
Unlike DECIMAL(p,s), there is no column-wide scale. Unlike FLOAT / DOUBLE,
arithmetic is done in base 10, so values such as 0.1 are exact.
Because each value carries at most 16 or 34 significant digits, a result
needing more digits is rounded to the format's precision. Rounding is decimal
(base 10), unlike binary rounding in FLOAT/DOUBLE. The v1 fixed default is IEEE
roundTiesToEven / HALF_EVEN (banker's rounding), which differs from Spark's
existing DECIMAL arithmetic (rounds HALF_UP). This can be configurable in
future iterations.
The type also supports IEEE special values: signed zero, +/-Infinity, and quiet
NaN.
*Q2. What problem is this proposal NOT designed to solve?*
* *Arbitrary-precision / unbounded decimals.* Extending Spark's fixed-point
DECIMAL beyond 38 digits, or adding a PostgreSQL-style unbounded NUMERIC, is
out of scope. Those needs are different from IEEE decimal floating point.
* *Wider-than-IEEE formats in v1.* Formats such as a 70-digit / 256-bit
decimal float (sometimes discussed as DECFLOAT(70) / decimal256) are out of
scope for the first delivery. The type and storage designs should not preclude
them later.
* *Replacing* DECIMAL *or* DOUBLE{*}.{*} Existing fixed-point and binary-float
types remain unchanged. DECFLOAT is additive.
* *Non-IEEE decimal-float dialects as the native type.* Engines such as
Snowflake, Oracle, and Teradata expose a decimal float that is not a strict
IEEE width (e.g. 38 digits, no NaN/Inf). Spark's native type targets IEEE 754
decimal64 and decimal128. Mapping those dialects into Spark is a connector /
cast concern, not a second native type.
* *ORC / CSV / JSON as first-class DECFLOAT storage in v1.* Persistence focus
for v1 is Parquet (and table formats that sit on Parquet). Other formats may
round-trip via string or binary until separately specified.
*Q3. How is it done today, and what are the limits of current practice?*
Spark SQL today offers two numeric families for real numbers:
# DECIMAL(p,s) *(fixed-point).* Precision and scale are fixed for the column.
Spark caps precision at 38. Mixed magnitudes force a tradeoff: many fractional
digits leave few integer digits (and the reverse). Chains of arithmetic expand
precision quickly and often require manual casts.
# FLOAT */* DOUBLE *(binary floating-point).* Wide range, but many common
decimal fractions are not exact. About 15-16 significant decimal digits for
DOUBLE, with binary rounding error.
Workarounds in use today:
||*Workaround*||*Limitation*||
|Map high-precision source decimals to DECIMAL(38,s)|Overflow / out-of-range
errors for large integers; silent rounding of long fractions|
|Cast to DOUBLE|Binary rounding; unsuitable when exact decimal fractions matter|
|Store as STRING|Loses numeric semantics; arithmetic and aggregation must be
rebuilt|
|Emulate with (unscaled DECIMAL, scale INT) structs|Not a first-class type;
poor pushdown, stats, and ecosystem support|
Spark also cannot recognize IEEE decimal floating-point values in Parquet (or
other built-in file sources) as a distinct logical type today, so even when
another system wrote such values, Spark cannot load them as decimals with
per-value exponents.
*Q4. What is new in your approach and why do you think it will be successful?*
The approach is deliberately conventional and layered, following prior Spark
type introductions (TIMESTAMP_NTZ, ANSI intervals, TIME):
# *Standard semantics first.* Align the SQL type with IEEE 754-2019 decimal64
/ decimal128 and with existing SQL DECFLOAT practice (DB2, Firebird, MongoDB).
Prefer IEEE default exception handling (produce Inf/NaN), rather than inventing
a Spark-only numeric model. Same goes for rounding, roundTiesToEven is chosen
because it's the IEEE default and libbid/DB2 behaviour.
# *Concrete physical representation.* Use the IEEE binary integer decimal
(BID) interchange encoding in memory and on disk, with a well-known math
library (Intel Decimal Floating-Point Math Library / libbid) for arithmetic
kernels. This avoids a logical-only typedef over DECIMAL / STRING that would
diverge across implementations.
# *Open storage path.* Propose a
[Parquet|https://docs.google.com/document/d/104397AVUqg_JSlzGBdpa3D98X6Dd-RONIgABn3omBcw/edit?tab=t.0]
logical type for decimal floating point (width-extensible FIXED_LEN_BYTE_ARRAY
annotation; initially decimal64 and decimal128) so Spark, other engines, and
table formats can interoperate without Arrow-footer private conventions.
# *Additive rollout.* Gate the type behind a config during incubation (pattern
used for TIME), then enable by default once coverage matches peer numeric types.
We expect this to succeed because:
* The user-visible type matches what migrators already know from DB2 /
Firebird / MongoDB Decimal128, and closes a clear gap versus warehouses that
already ship a decimal-float type.
* The encoding and math library are industry-proven rather than bespoke.
* Spark already has a playbook for introducing a new atomic type end-to-end
(parser → Catalyst → execution → datasources → Connect / PySpark / JDBC).
*Q5. Who cares? If you are successful, what difference will it make?*
* *Users migrating from PostgreSQL, Oracle, DB2, BigQuery, and similar
systems* who today hit Spark's DECIMAL(38,*) ceiling or lose precision via
DOUBLE / STRING workarounds. Unconstrained or high-precision source numerics
map cleanly to a first-class numeric type.
* *Financial, actuarial, and crypto / fintech workloads* that need exact
decimal fractions and mixed magnitudes in one column (rates, notionals,
micro-quantities and large balances together).
* *Existing Spark users* who need to read or write Parquet datasets produced
by systems that already use IEEE decimal128 / DECFLOAT.
* *The wider storage ecosystem* (Parquet, and eventually Iceberg / Delta
consumers) gains a portable decimal-float logical type rather than
engine-private encodings.
Success means: declare DECFLOAT columns, run SQL arithmetic and aggregations
without leaving the numeric domain, and round-trip values through Parquet with
IEEE semantics preserved (including cohort / quantum where the format retains
it).
*Q6. What are the risks?*
||*Risk*||*Mitigation*||
|Parquet / table-format standardization lagging Spark SQL|Ship Spark SQL type
with a documented on-disk encoding; upstream Parquet logical type in parallel;
avoid relying on undocumented footer keys|
|Surprising type coercion (DECFLOAT outranking DOUBLE / interacting with
DECIMAL)|Document precedence explicitly; add golden SQL tests; match IEEE / SQL
expectations rather than silent demotion to binary float|
|Special-value semantics (NaN, Inf) surprising users coming from engines
without them (e.g. some warehouse DECFLOAT dialects)|Document clearly; default
IEEE non-stop handling; ANSI mode provides fail-fast behavior consistent with
other numeric types.|
|External API choice for Inf/NaN (Java BigDecimal cannot represent them)|Pick
an explicit external type story early (see Appendix B) and test Connect /
PySpark / JDBC thoroughly|
|Scope creep into arbitrary precision or DECFLOAT(70)|Keep v1 strictly at
precisions 16 and 34; record wider formats as follow-ons|
|Implementation overhead and ensuring full behavioural consistency in a custom
JVM implementation versus existing libraries|Implement DECFLOAT via JNI calls
to the existing libbid library (which offers fast, exact math for decimal
floating-point numbers)|
*Q7. How long will it take?*
Rough estimate: *on the order of 9-12 months* for feature-complete parity with
peer numeric types, based on TIME (SPARK-51162), TIMESTAMP_NTZ (SPARK-35662),
and ANSI intervals (SPARK-27790), plus extra time for the math library
integration and Parquet logical type.
Suggested work split (can become JIRA sub-tasks):
# *Base type: ~1 month* DecFloatType, parser / DDL, literals (DECFLOAT '...',
optional DF suffix), casts to/from string and numeric types, etc.
# *Arithmetic and comparison kernels: ~2 months* + - * /, unary minus,
comparisons, ordering (totalOrder for sort), hashing / grouping equality
(canonicalization for =-equal values), codegen / interpreted execution.
# *Functions and aggregates: ~2 months* Core scalars (abs, sign, floor, ceil,
round, sqrt, isnan, …), DECFLOAT-specific helpers (quantize, same_quantum,
total_order), sum / avg / min / max / count, window variants.
# *Persistence: ~3 months* Parquet read/write with logical annotation,
partition values, stats / predicate pushdown, caching / shuffle; coordinate
with Parquet format RFC.
# *Clients: ~1.5 months* Spark Connect proto, JDBC / Thrift / Hive result
mapping, catalog / information_schema.
# *PySpark / Arrow: ~1.5 months* DataFrame API, pandas / Arrow interchange,
Python UDFs.
# *Docs, golden tests, benchmarks: ~1 month (overlaps)*
[OPEN] Confirm estimate after sketching the Parquet dependency and whether
Arrow needs a parallel extension type for transport.
*Q8. What are the mid-term and final "exams" to check for success?*
*Mid-term (~4-5 months):*
* DecFloatType(16|34) usable in SQL: literals, DDL, casts, arithmetic,
comparisons, basic aggregates.
* Correct IEEE behavior for a representative set of finite values, signed
zeros, Inf, and NaN (under both ANSI and non-ANSI modes), and rounding
(especially at the precision boundary, e.g. results exceeding 16/34 digits,
ties).
* Round-trip through at least one built-in file source (Parquet) with a stable
encoding, even if the upstream Parquet annotation is still landing.
* No behavioral change to existing DECIMAL / DOUBLE workloads when the new
type is unused.
*Final exam (~9-12 months):*
* Feature parity with other numeric types for the agreed v1 function set (see
Appendix C sketch).
* Interoperable Parquet read/write against an independent implementation (e.g.
parquet-java ↔ parquet-rs) once the logical type is specified.
* Connect, JDBC, and PySpark can create, query, and collect DECFLOAT columns.
* Documented ANSI / IEEE compliance notes and migration guidance from DECIMAL
/ DOUBLE / string workarounds.
was:
*Q1. What are you trying to do? Articulate your objectives using absolutely no
jargon.*
Add a new Spark SQL data type *DECFLOAT* that stores decimal numbers with a
flexible decimal point (floating point decimal numbers). Each value keeps up to
a fixed number of significant decimal digits and its own exponent, so one
column can hold both very large integers and very small fractions without
picking a single fixed scale for the whole column.
The type follows the IEEE 754 decimal floating-point formats:
||*SQL type*||*IEEE format*||*Significant digits*||*Storage width*||
|DECFLOAT(16)|decimal64|16|8 bytes|
|DECFLOAT(34)|decimal128|34|16 bytes|
Bare DECFLOAT means DECFLOAT(34).
Unlike DECIMAL(p,s), there is no column-wide scale. Unlike FLOAT / DOUBLE,
arithmetic is done in base 10, so values such as 0.1 are exact.
Because each value carries at most 16 or 34 significant digits, a result
needing more digits is rounded to the format's precision. Rounding is decimal
(base 10), unlike binary rounding in FLOAT/DOUBLE. The default is IEEE
roundTiesToEven / HALF_EVEN (banker's rounding), which differs from Spark's
existing DECIMAL arithmetic (rounds HALF_UP).
The type also supports IEEE special values: signed zero, +/-Infinity, and quiet
NaN.
*Q2. What problem is this proposal NOT designed to solve?*
* *Arbitrary-precision / unbounded decimals.* Extending Spark's fixed-point
DECIMAL beyond 38 digits, or adding a PostgreSQL-style unbounded NUMERIC, is
out of scope. Those needs are different from IEEE decimal floating point.
* *Wider-than-IEEE formats in v1.* Formats such as a 70-digit / 256-bit
decimal float (sometimes discussed as DECFLOAT(70) / decimal256) are out of
scope for the first delivery. The type and storage designs should not preclude
them later.
* *Replacing* DECIMAL *or* DOUBLE{*}.{*} Existing fixed-point and binary-float
types remain unchanged. DECFLOAT is additive.
* *Non-IEEE decimal-float dialects as the native type.* Engines such as
Snowflake, Oracle, and Teradata expose a decimal float that is not a strict
IEEE width (e.g. 38 digits, no NaN/Inf). Spark's native type targets IEEE 754
decimal64 and decimal128. Mapping those dialects into Spark is a connector /
cast concern, not a second native type.
* *ORC / CSV / JSON as first-class DECFLOAT storage in v1.* Persistence focus
for v1 is Parquet (and table formats that sit on Parquet). Other formats may
round-trip via string or binary until separately specified.
*Q3. How is it done today, and what are the limits of current practice?*
Spark SQL today offers two numeric families for real numbers:
# DECIMAL(p,s) *(fixed-point).* Precision and scale are fixed for the column.
Spark caps precision at 38. Mixed magnitudes force a tradeoff: many fractional
digits leave few integer digits (and the reverse). Chains of arithmetic expand
precision quickly and often require manual casts.
# FLOAT */* DOUBLE *(binary floating-point).* Wide range, but many common
decimal fractions are not exact. About 15-16 significant decimal digits for
DOUBLE, with binary rounding error.
Workarounds in use today:
||*Workaround*||*Limitation*||
|Map high-precision source decimals to DECIMAL(38,s)|Overflow / out-of-range
errors for large integers; silent rounding of long fractions|
|Cast to DOUBLE|Binary rounding; unsuitable when exact decimal fractions matter|
|Store as STRING|Loses numeric semantics; arithmetic and aggregation must be
rebuilt|
|Emulate with (unscaled DECIMAL, scale INT) structs|Not a first-class type;
poor pushdown, stats, and ecosystem support|
Spark also cannot recognize IEEE decimal floating-point values in Parquet (or
other built-in file sources) as a distinct logical type today, so even when
another system wrote such values, Spark cannot load them as decimals with
per-value exponents.
*Q4. What is new in your approach and why do you think it will be successful?*
The approach is deliberately conventional and layered, following prior Spark
type introductions (TIMESTAMP_NTZ, ANSI intervals, TIME):
# *Standard semantics first.* Align the SQL type with IEEE 754-2019 decimal64
/ decimal128 and with existing SQL DECFLOAT practice (DB2, Firebird, MongoDB).
Prefer IEEE default exception handling (produce Inf/NaN), rather than inventing
a Spark-only numeric model. Same goes for rounding, roundTiesToEven is chosen
because it's the IEEE default and libbid/DB2 behaviour.
# *Concrete physical representation.* Use the IEEE binary integer decimal
(BID) interchange encoding in memory and on disk, with a well-known math
library (Intel Decimal Floating-Point Math Library / libbid) for arithmetic
kernels. This avoids a logical-only typedef over DECIMAL / STRING that would
diverge across implementations.
# *Open storage path.* Propose a
[Parquet|https://docs.google.com/document/d/104397AVUqg_JSlzGBdpa3D98X6Dd-RONIgABn3omBcw/edit?tab=t.0]
logical type for decimal floating point (width-extensible FIXED_LEN_BYTE_ARRAY
annotation; initially decimal64 and decimal128) so Spark, other engines, and
table formats can interoperate without Arrow-footer private conventions.
# *Additive rollout.* Gate the type behind a config during incubation (pattern
used for TIME), then enable by default once coverage matches peer numeric types.
We expect this to succeed because:
* The user-visible type matches what migrators already know from DB2 /
Firebird / MongoDB Decimal128, and closes a clear gap versus warehouses that
already ship a decimal-float type.
* The encoding and math library are industry-proven rather than bespoke.
* Spark already has a playbook for introducing a new atomic type end-to-end
(parser → Catalyst → execution → datasources → Connect / PySpark / JDBC).
*Q5. Who cares? If you are successful, what difference will it make?*
* *Users migrating from PostgreSQL, Oracle, DB2, BigQuery, and similar
systems* who today hit Spark's DECIMAL(38,*) ceiling or lose precision via
DOUBLE / STRING workarounds. Unconstrained or high-precision source numerics
map cleanly to a first-class numeric type.
* *Financial, actuarial, and crypto / fintech workloads* that need exact
decimal fractions and mixed magnitudes in one column (rates, notionals,
micro-quantities and large balances together).
* *Existing Spark users* who need to read or write Parquet datasets produced
by systems that already use IEEE decimal128 / DECFLOAT.
* *The wider storage ecosystem* (Parquet, and eventually Iceberg / Delta
consumers) gains a portable decimal-float logical type rather than
engine-private encodings.
Success means: declare DECFLOAT columns, run SQL arithmetic and aggregations
without leaving the numeric domain, and round-trip values through Parquet with
IEEE semantics preserved (including cohort / quantum where the format retains
it).
*Q6. What are the risks?*
||*Risk*||*Mitigation*||
|Parquet / table-format standardization lagging Spark SQL|Ship Spark SQL type
with a documented on-disk encoding; upstream Parquet logical type in parallel;
avoid relying on undocumented footer keys|
|Surprising type coercion (DECFLOAT outranking DOUBLE / interacting with
DECIMAL)|Document precedence explicitly; add golden SQL tests; match IEEE / SQL
expectations rather than silent demotion to binary float|
|Special-value semantics (NaN, Inf) surprising users coming from engines
without them (e.g. some warehouse DECFLOAT dialects)|Document clearly; default
IEEE non-stop handling; ANSI mode provides fail-fast behavior consistent with
other numeric types.|
|External API choice for Inf/NaN (Java BigDecimal cannot represent them)|Pick
an explicit external type story early (see Appendix B) and test Connect /
PySpark / JDBC thoroughly|
|Scope creep into arbitrary precision or DECFLOAT(70)|Keep v1 strictly at
precisions 16 and 34; record wider formats as follow-ons|
|Implementation overhead and ensuring full behavioural consistency in a custom
JVM implementation versus existing libraries|Implement DECFLOAT via JNI calls
to the existing libbid library (which offers fast, exact math for decimal
floating-point numbers)|
*Q7. How long will it take?*
Rough estimate: *on the order of 9-12 months* for feature-complete parity with
peer numeric types, based on TIME (SPARK-51162), TIMESTAMP_NTZ (SPARK-35662),
and ANSI intervals (SPARK-27790), plus extra time for the math library
integration and Parquet logical type.
Suggested work split (can become JIRA sub-tasks):
# *Base type: ~1 month* DecFloatType, parser / DDL, literals (DECFLOAT '...',
optional DF suffix), casts to/from string and numeric types, session rounding
mode.
# *Arithmetic and comparison kernels: ~2 months* + - * /, unary minus,
comparisons, ordering (totalOrder for sort), hashing / grouping equality
(canonicalization for =-equal values), codegen / interpreted execution.
# *Functions and aggregates: ~2 months* Core scalars (abs, sign, floor, ceil,
round, sqrt, isnan, …), DECFLOAT-specific helpers (quantize, same_quantum,
total_order), sum / avg / min / max / count, window variants.
# *Persistence: ~3 months* Parquet read/write with logical annotation,
partition values, stats / predicate pushdown, caching / shuffle; coordinate
with Parquet format RFC.
# *Clients: ~1.5 months* Spark Connect proto, JDBC / Thrift / Hive result
mapping, catalog / information_schema.
# *PySpark / Arrow: ~1.5 months* DataFrame API, pandas / Arrow interchange,
Python UDFs.
# *Docs, golden tests, benchmarks: ~1 month (overlaps)*
[OPEN] Confirm estimate after sketching the Parquet dependency and whether
Arrow needs a parallel extension type for transport.
*Q8. What are the mid-term and final "exams" to check for success?*
*Mid-term (~4-5 months):*
* DecFloatType(16|34) usable in SQL: literals, DDL, casts, arithmetic,
comparisons, basic aggregates.
* Correct IEEE behavior for a representative set of finite values, signed
zeros, Inf, and NaN (under both ANSI and non-ANSI modes), and rounding
(especially at the precision boundary, e.g. results exceeding 16/34 digits,
ties).
* Round-trip through at least one built-in file source (Parquet) with a stable
encoding, even if the upstream Parquet annotation is still landing.
* No behavioral change to existing DECIMAL / DOUBLE workloads when the new
type is unused.
*Final exam (~9-12 months):*
* Feature parity with other numeric types for the agreed v1 function set (see
Appendix C sketch).
* Interoperable Parquet read/write against an independent implementation (e.g.
parquet-java ↔ parquet-rs) once the logical type is specified.
* Connect, JDBC, and PySpark can create, query, and collect DECFLOAT columns.
* Documented ANSI / IEEE compliance notes and migration guidance from DECIMAL
/ DOUBLE / string workarounds.
> SPIP: Add the DECFLOAT data type
> --------------------------------
>
> Key: SPARK-58820
> URL: https://issues.apache.org/jira/browse/SPARK-58820
> Project: Spark
> Issue Type: Umbrella
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Uroš Bojanić
> Assignee: Uroš Bojanić
> Priority: Major
> Labels: SPIP
>
> *Q1. What are you trying to do? Articulate your objectives using absolutely
> no jargon.*
> Add a new Spark SQL data type *DECFLOAT* that stores decimal numbers with a
> flexible decimal point (floating point decimal numbers). Each value keeps up
> to a fixed number of significant decimal digits and its own exponent, so one
> column can hold both very large integers and very small fractions without
> picking a single fixed scale for the whole column.
> The type follows the IEEE 754 decimal floating-point formats:
> ||*SQL type*||*IEEE format*||*Significant digits*||*Storage width*||
> |DECFLOAT(16)|decimal64|16|8 bytes|
> |DECFLOAT(34)|decimal128|34|16 bytes|
> Bare DECFLOAT means DECFLOAT(34).
> Unlike DECIMAL(p,s), there is no column-wide scale. Unlike FLOAT / DOUBLE,
> arithmetic is done in base 10, so values such as 0.1 are exact.
> Because each value carries at most 16 or 34 significant digits, a result
> needing more digits is rounded to the format's precision. Rounding is decimal
> (base 10), unlike binary rounding in FLOAT/DOUBLE. The v1 fixed default is
> IEEE roundTiesToEven / HALF_EVEN (banker's rounding), which differs from
> Spark's existing DECIMAL arithmetic (rounds HALF_UP). This can be
> configurable in future iterations.
> The type also supports IEEE special values: signed zero, +/-Infinity, and
> quiet NaN.
> *Q2. What problem is this proposal NOT designed to solve?*
> * *Arbitrary-precision / unbounded decimals.* Extending Spark's fixed-point
> DECIMAL beyond 38 digits, or adding a PostgreSQL-style unbounded NUMERIC, is
> out of scope. Those needs are different from IEEE decimal floating point.
> * *Wider-than-IEEE formats in v1.* Formats such as a 70-digit / 256-bit
> decimal float (sometimes discussed as DECFLOAT(70) / decimal256) are out of
> scope for the first delivery. The type and storage designs should not
> preclude them later.
> * *Replacing* DECIMAL *or* DOUBLE{*}.{*} Existing fixed-point and
> binary-float types remain unchanged. DECFLOAT is additive.
> * *Non-IEEE decimal-float dialects as the native type.* Engines such as
> Snowflake, Oracle, and Teradata expose a decimal float that is not a strict
> IEEE width (e.g. 38 digits, no NaN/Inf). Spark's native type targets IEEE 754
> decimal64 and decimal128. Mapping those dialects into Spark is a connector /
> cast concern, not a second native type.
> * *ORC / CSV / JSON as first-class DECFLOAT storage in v1.* Persistence focus
> for v1 is Parquet (and table formats that sit on Parquet). Other formats may
> round-trip via string or binary until separately specified.
> *Q3. How is it done today, and what are the limits of current practice?*
> Spark SQL today offers two numeric families for real numbers:
> # DECIMAL(p,s) *(fixed-point).* Precision and scale are fixed for the
> column. Spark caps precision at 38. Mixed magnitudes force a tradeoff: many
> fractional digits leave few integer digits (and the reverse). Chains of
> arithmetic expand precision quickly and often require manual casts.
> # FLOAT */* DOUBLE *(binary floating-point).* Wide range, but many common
> decimal fractions are not exact. About 15-16 significant decimal digits for
> DOUBLE, with binary rounding error.
> Workarounds in use today:
> ||*Workaround*||*Limitation*||
> |Map high-precision source decimals to DECIMAL(38,s)|Overflow / out-of-range
> errors for large integers; silent rounding of long fractions|
> |Cast to DOUBLE|Binary rounding; unsuitable when exact decimal fractions
> matter|
> |Store as STRING|Loses numeric semantics; arithmetic and aggregation must be
> rebuilt|
> |Emulate with (unscaled DECIMAL, scale INT) structs|Not a first-class type;
> poor pushdown, stats, and ecosystem support|
> Spark also cannot recognize IEEE decimal floating-point values in Parquet (or
> other built-in file sources) as a distinct logical type today, so even when
> another system wrote such values, Spark cannot load them as decimals with
> per-value exponents.
> *Q4. What is new in your approach and why do you think it will be successful?*
> The approach is deliberately conventional and layered, following prior Spark
> type introductions (TIMESTAMP_NTZ, ANSI intervals, TIME):
> # *Standard semantics first.* Align the SQL type with IEEE 754-2019
> decimal64 / decimal128 and with existing SQL DECFLOAT practice (DB2,
> Firebird, MongoDB). Prefer IEEE default exception handling (produce Inf/NaN),
> rather than inventing a Spark-only numeric model. Same goes for rounding,
> roundTiesToEven is chosen because it's the IEEE default and libbid/DB2
> behaviour.
> # *Concrete physical representation.* Use the IEEE binary integer decimal
> (BID) interchange encoding in memory and on disk, with a well-known math
> library (Intel Decimal Floating-Point Math Library / libbid) for arithmetic
> kernels. This avoids a logical-only typedef over DECIMAL / STRING that would
> diverge across implementations.
> # *Open storage path.* Propose a
> [Parquet|https://docs.google.com/document/d/104397AVUqg_JSlzGBdpa3D98X6Dd-RONIgABn3omBcw/edit?tab=t.0]
> logical type for decimal floating point (width-extensible
> FIXED_LEN_BYTE_ARRAY annotation; initially decimal64 and decimal128) so
> Spark, other engines, and table formats can interoperate without Arrow-footer
> private conventions.
> # *Additive rollout.* Gate the type behind a config during incubation
> (pattern used for TIME), then enable by default once coverage matches peer
> numeric types.
> We expect this to succeed because:
> * The user-visible type matches what migrators already know from DB2 /
> Firebird / MongoDB Decimal128, and closes a clear gap versus warehouses that
> already ship a decimal-float type.
> * The encoding and math library are industry-proven rather than bespoke.
> * Spark already has a playbook for introducing a new atomic type end-to-end
> (parser → Catalyst → execution → datasources → Connect / PySpark / JDBC).
> *Q5. Who cares? If you are successful, what difference will it make?*
> * *Users migrating from PostgreSQL, Oracle, DB2, BigQuery, and similar
> systems* who today hit Spark's DECIMAL(38,*) ceiling or lose precision via
> DOUBLE / STRING workarounds. Unconstrained or high-precision source numerics
> map cleanly to a first-class numeric type.
> * *Financial, actuarial, and crypto / fintech workloads* that need exact
> decimal fractions and mixed magnitudes in one column (rates, notionals,
> micro-quantities and large balances together).
> * *Existing Spark users* who need to read or write Parquet datasets produced
> by systems that already use IEEE decimal128 / DECFLOAT.
> * *The wider storage ecosystem* (Parquet, and eventually Iceberg / Delta
> consumers) gains a portable decimal-float logical type rather than
> engine-private encodings.
> Success means: declare DECFLOAT columns, run SQL arithmetic and aggregations
> without leaving the numeric domain, and round-trip values through Parquet
> with IEEE semantics preserved (including cohort / quantum where the format
> retains it).
> *Q6. What are the risks?*
> ||*Risk*||*Mitigation*||
> |Parquet / table-format standardization lagging Spark SQL|Ship Spark SQL type
> with a documented on-disk encoding; upstream Parquet logical type in
> parallel; avoid relying on undocumented footer keys|
> |Surprising type coercion (DECFLOAT outranking DOUBLE / interacting with
> DECIMAL)|Document precedence explicitly; add golden SQL tests; match IEEE /
> SQL expectations rather than silent demotion to binary float|
> |Special-value semantics (NaN, Inf) surprising users coming from engines
> without them (e.g. some warehouse DECFLOAT dialects)|Document clearly;
> default IEEE non-stop handling; ANSI mode provides fail-fast behavior
> consistent with other numeric types.|
> |External API choice for Inf/NaN (Java BigDecimal cannot represent them)|Pick
> an explicit external type story early (see Appendix B) and test Connect /
> PySpark / JDBC thoroughly|
> |Scope creep into arbitrary precision or DECFLOAT(70)|Keep v1 strictly at
> precisions 16 and 34; record wider formats as follow-ons|
> |Implementation overhead and ensuring full behavioural consistency in a
> custom JVM implementation versus existing libraries|Implement DECFLOAT via
> JNI calls to the existing libbid library (which offers fast, exact math for
> decimal floating-point numbers)|
> *Q7. How long will it take?*
> Rough estimate: *on the order of 9-12 months* for feature-complete parity
> with peer numeric types, based on TIME (SPARK-51162), TIMESTAMP_NTZ
> (SPARK-35662), and ANSI intervals (SPARK-27790), plus extra time for the math
> library integration and Parquet logical type.
> Suggested work split (can become JIRA sub-tasks):
> # *Base type: ~1 month* DecFloatType, parser / DDL, literals (DECFLOAT
> '...', optional DF suffix), casts to/from string and numeric types, etc.
> # *Arithmetic and comparison kernels: ~2 months* + - * /, unary minus,
> comparisons, ordering (totalOrder for sort), hashing / grouping equality
> (canonicalization for =-equal values), codegen / interpreted execution.
> # *Functions and aggregates: ~2 months* Core scalars (abs, sign, floor,
> ceil, round, sqrt, isnan, …), DECFLOAT-specific helpers (quantize,
> same_quantum, total_order), sum / avg / min / max / count, window variants.
> # *Persistence: ~3 months* Parquet read/write with logical annotation,
> partition values, stats / predicate pushdown, caching / shuffle; coordinate
> with Parquet format RFC.
> # *Clients: ~1.5 months* Spark Connect proto, JDBC / Thrift / Hive result
> mapping, catalog / information_schema.
> # *PySpark / Arrow: ~1.5 months* DataFrame API, pandas / Arrow interchange,
> Python UDFs.
> # *Docs, golden tests, benchmarks: ~1 month (overlaps)*
> [OPEN] Confirm estimate after sketching the Parquet dependency and whether
> Arrow needs a parallel extension type for transport.
> *Q8. What are the mid-term and final "exams" to check for success?*
> *Mid-term (~4-5 months):*
> * DecFloatType(16|34) usable in SQL: literals, DDL, casts, arithmetic,
> comparisons, basic aggregates.
> * Correct IEEE behavior for a representative set of finite values, signed
> zeros, Inf, and NaN (under both ANSI and non-ANSI modes), and rounding
> (especially at the precision boundary, e.g. results exceeding 16/34 digits,
> ties).
> * Round-trip through at least one built-in file source (Parquet) with a
> stable encoding, even if the upstream Parquet annotation is still landing.
> * No behavioral change to existing DECIMAL / DOUBLE workloads when the new
> type is unused.
> *Final exam (~9-12 months):*
> * Feature parity with other numeric types for the agreed v1 function set
> (see Appendix C sketch).
> * Interoperable Parquet read/write against an independent implementation
> (e.g. parquet-java ↔ parquet-rs) once the logical type is specified.
> * Connect, JDBC, and PySpark can create, query, and collect DECFLOAT columns.
> * Documented ANSI / IEEE compliance notes and migration guidance from
> DECIMAL / DOUBLE / string workarounds.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
