[
https://issues.apache.org/jira/browse/SPARK-58820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18105266#comment-18105266
]
Uroš Bojanić commented on SPARK-58820:
--------------------------------------
SPIP Document:
https://docs.google.com/document/d/1qnLXm0ldHSwPSJs_Q5SzGyyTKMMEqeX5rm8Fh4rvV3E/edit?tab=t.0#heading=h.zarpams0nxzq.
> SPIP: Add the DECFLOAT data type
> --------------------------------
>
> Key: SPARK-58820
> URL: https://issues.apache.org/jira/browse/SPARK-58820
> Project: Spark
> Issue Type: Umbrella
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Uroš Bojanić
> Assignee: Uroš Bojanić
> Priority: Major
> Labels: SPIP
>
> *Q1. What are you trying to do? Articulate your objectives using absolutely
> no jargon.*
> Add a new Spark SQL data type *DECFLOAT* that stores decimal numbers with a
> flexible decimal point (floating point decimal numbers). Each value keeps up
> to a fixed number of significant decimal digits and its own exponent, so one
> column can hold both very large integers and very small fractions without
> picking a single fixed scale for the whole column.
> The type follows the IEEE 754 decimal floating-point formats:
> ||*SQL type*||*IEEE format*||*Significant digits*||*Storage width*||
> |DECFLOAT(16)|decimal64|16|8 bytes|
> |DECFLOAT(34)|decimal128|34|16 bytes|
> Bare DECFLOAT means DECFLOAT(34).
> Unlike DECIMAL(p,s), there is no column-wide scale. Unlike FLOAT / DOUBLE,
> arithmetic is done in base 10, so values such as 0.1 are exact.
> Because each value carries at most 16 or 34 significant digits, a result
> needing more digits is rounded to the format's precision. Rounding is decimal
> (base 10), unlike binary rounding in FLOAT/DOUBLE. The default is IEEE
> roundTiesToEven / HALF_EVEN (banker's rounding), which differs from Spark's
> existing DECIMAL arithmetic (rounds HALF_UP).
> The type also supports IEEE special values: signed zero, +/-Infinity, and
> quiet NaN.
> *Q2. What problem is this proposal NOT designed to solve?*
> * *Arbitrary-precision / unbounded decimals.* Extending Spark's fixed-point
> DECIMAL beyond 38 digits, or adding a PostgreSQL-style unbounded NUMERIC, is
> out of scope. Those needs are different from IEEE decimal floating point.
> * *Wider-than-IEEE formats in v1.* Formats such as a 70-digit / 256-bit
> decimal float (sometimes discussed as DECFLOAT(70) / decimal256) are out of
> scope for the first delivery. The type and storage designs should not
> preclude them later.
> * *Replacing* DECIMAL *or* DOUBLE{*}.{*} Existing fixed-point and
> binary-float types remain unchanged. DECFLOAT is additive.
> * *Non-IEEE decimal-float dialects as the native type.* Engines such as
> Snowflake, Oracle, and Teradata expose a decimal float that is not a strict
> IEEE width (e.g. 38 digits, no NaN/Inf). Spark's native type targets IEEE 754
> decimal64 and decimal128. Mapping those dialects into Spark is a connector /
> cast concern, not a second native type.
> * *ORC / CSV / JSON as first-class DECFLOAT storage in v1.* Persistence focus
> for v1 is Parquet (and table formats that sit on Parquet). Other formats may
> round-trip via string or binary until separately specified.
> *Q3. How is it done today, and what are the limits of current practice?*
> Spark SQL today offers two numeric families for real numbers:
> # DECIMAL(p,s) *(fixed-point).* Precision and scale are fixed for the
> column. Spark caps precision at 38. Mixed magnitudes force a tradeoff: many
> fractional digits leave few integer digits (and the reverse). Chains of
> arithmetic expand precision quickly and often require manual casts.
> # FLOAT */* DOUBLE *(binary floating-point).* Wide range, but many common
> decimal fractions are not exact. About 15-16 significant decimal digits for
> DOUBLE, with binary rounding error.
> Workarounds in use today:
> ||*Workaround*||*Limitation*||
> |Map high-precision source decimals to DECIMAL(38,s)|Overflow / out-of-range
> errors for large integers; silent rounding of long fractions|
> |Cast to DOUBLE|Binary rounding; unsuitable when exact decimal fractions
> matter|
> |Store as STRING|Loses numeric semantics; arithmetic and aggregation must be
> rebuilt|
> |Emulate with (unscaled DECIMAL, scale INT) structs|Not a first-class type;
> poor pushdown, stats, and ecosystem support|
> Spark also cannot recognize IEEE decimal floating-point values in Parquet (or
> other built-in file sources) as a distinct logical type today, so even when
> another system wrote such values, Spark cannot load them as decimals with
> per-value exponents.
> *Q4. What is new in your approach and why do you think it will be successful?*
> The approach is deliberately conventional and layered, following prior Spark
> type introductions (TIMESTAMP_NTZ, ANSI intervals, TIME):
> # *Standard semantics first.* Align the SQL type with IEEE 754-2019
> decimal64 / decimal128 and with existing SQL DECFLOAT practice (DB2,
> Firebird, MongoDB). Prefer IEEE default exception handling (produce Inf/NaN),
> rather than inventing a Spark-only numeric model. Same goes for rounding,
> roundTiesToEven is chosen because it's the IEEE default and libbid/DB2
> behaviour.
> # *Concrete physical representation.* Use the IEEE binary integer decimal
> (BID) interchange encoding in memory and on disk, with a well-known math
> library (Intel Decimal Floating-Point Math Library / libbid) for arithmetic
> kernels. This avoids a logical-only typedef over DECIMAL / STRING that would
> diverge across implementations.
> # *Open storage path.* Propose a
> [Parquet|https://docs.google.com/document/d/104397AVUqg_JSlzGBdpa3D98X6Dd-RONIgABn3omBcw/edit?tab=t.0]
> logical type for decimal floating point (width-extensible
> FIXED_LEN_BYTE_ARRAY annotation; initially decimal64 and decimal128) so
> Spark, other engines, and table formats can interoperate without Arrow-footer
> private conventions.
> # *Additive rollout.* Gate the type behind a config during incubation
> (pattern used for TIME), then enable by default once coverage matches peer
> numeric types.
> We expect this to succeed because:
> * The user-visible type matches what migrators already know from DB2 /
> Firebird / MongoDB Decimal128, and closes a clear gap versus warehouses that
> already ship a decimal-float type.
> * The encoding and math library are industry-proven rather than bespoke.
> * Spark already has a playbook for introducing a new atomic type end-to-end
> (parser → Catalyst → execution → datasources → Connect / PySpark / JDBC).
> *Q5. Who cares? If you are successful, what difference will it make?*
> * *Users migrating from PostgreSQL, Oracle, DB2, BigQuery, and similar
> systems* who today hit Spark's DECIMAL(38,*) ceiling or lose precision via
> DOUBLE / STRING workarounds. Unconstrained or high-precision source numerics
> map cleanly to a first-class numeric type.
> * *Financial, actuarial, and crypto / fintech workloads* that need exact
> decimal fractions and mixed magnitudes in one column (rates, notionals,
> micro-quantities and large balances together).
> * *Existing Spark users* who need to read or write Parquet datasets produced
> by systems that already use IEEE decimal128 / DECFLOAT.
> * *The wider storage ecosystem* (Parquet, and eventually Iceberg / Delta
> consumers) gains a portable decimal-float logical type rather than
> engine-private encodings.
> Success means: declare DECFLOAT columns, run SQL arithmetic and aggregations
> without leaving the numeric domain, and round-trip values through Parquet
> with IEEE semantics preserved (including cohort / quantum where the format
> retains it).
> *Q6. What are the risks?*
> ||*Risk*||*Mitigation*||
> |Parquet / table-format standardization lagging Spark SQL|Ship Spark SQL type
> with a documented on-disk encoding; upstream Parquet logical type in
> parallel; avoid relying on undocumented footer keys|
> |Surprising type coercion (DECFLOAT outranking DOUBLE / interacting with
> DECIMAL)|Document precedence explicitly; add golden SQL tests; match IEEE /
> SQL expectations rather than silent demotion to binary float|
> |Special-value semantics (NaN, Inf) surprising users coming from engines
> without them (e.g. some warehouse DECFLOAT dialects)|Document clearly;
> default IEEE non-stop handling; ANSI mode provides fail-fast behavior
> consistent with other numeric types.|
> |External API choice for Inf/NaN (Java BigDecimal cannot represent them)|Pick
> an explicit external type story early (see Appendix B) and test Connect /
> PySpark / JDBC thoroughly|
> |Scope creep into arbitrary precision or DECFLOAT(70)|Keep v1 strictly at
> precisions 16 and 34; record wider formats as follow-ons|
> |Implementation overhead and ensuring full behavioural consistency in a
> custom JVM implementation versus existing libraries|Implement DECFLOAT via
> JNI calls to the existing libbid library (which offers fast, exact math for
> decimal floating-point numbers)|
> *Q7. How long will it take?*
> Rough estimate: *on the order of 9-12 months* for feature-complete parity
> with peer numeric types, based on TIME (SPARK-51162), TIMESTAMP_NTZ
> (SPARK-35662), and ANSI intervals (SPARK-27790), plus extra time for the math
> library integration and Parquet logical type.
> Suggested work split (can become JIRA sub-tasks):
> # *Base type: ~1 month* DecFloatType, parser / DDL, literals (DECFLOAT
> '...', optional DF suffix), casts to/from string and numeric types, session
> rounding mode.
> # *Arithmetic and comparison kernels: ~2 months* + - * /, unary minus,
> comparisons, ordering (totalOrder for sort), hashing / grouping equality
> (canonicalization for =-equal values), codegen / interpreted execution.
> # *Functions and aggregates: ~2 months* Core scalars (abs, sign, floor,
> ceil, round, sqrt, isnan, …), DECFLOAT-specific helpers (quantize,
> same_quantum, total_order), sum / avg / min / max / count, window variants.
> # *Persistence: ~3 months* Parquet read/write with logical annotation,
> partition values, stats / predicate pushdown, caching / shuffle; coordinate
> with Parquet format RFC.
> # *Clients: ~1.5 months* Spark Connect proto, JDBC / Thrift / Hive result
> mapping, catalog / information_schema.
> # *PySpark / Arrow: ~1.5 months* DataFrame API, pandas / Arrow interchange,
> Python UDFs.
> # *Docs, golden tests, benchmarks: ~1 month (overlaps)*
> [OPEN] Confirm estimate after sketching the Parquet dependency and whether
> Arrow needs a parallel extension type for transport.
> *Q8. What are the mid-term and final "exams" to check for success?*
> *Mid-term (~4-5 months):*
> * DecFloatType(16|34) usable in SQL: literals, DDL, casts, arithmetic,
> comparisons, basic aggregates.
> * Correct IEEE behavior for a representative set of finite values, signed
> zeros, Inf, and NaN (under both ANSI and non-ANSI modes), and rounding
> (especially at the precision boundary, e.g. results exceeding 16/34 digits,
> ties).
> * Round-trip through at least one built-in file source (Parquet) with a
> stable encoding, even if the upstream Parquet annotation is still landing.
> * No behavioral change to existing DECIMAL / DOUBLE workloads when the new
> type is unused.
> *Final exam (~9-12 months):*
> * Feature parity with other numeric types for the agreed v1 function set
> (see Appendix C sketch).
> * Interoperable Parquet read/write against an independent implementation
> (e.g. parquet-java ↔ parquet-rs) once the logical type is specified.
> * Connect, JDBC, and PySpark can create, query, and collect DECFLOAT columns.
> * Documented ANSI / IEEE compliance notes and migration guidance from
> DECIMAL / DOUBLE / string workarounds.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
