Hi again, Stefan, Serge, Shrirang, Russell, Ian, Dongjoon, and Thomas, Please see the latest reply on the thread [DISCUSS] SPIP: Add the DECFLOAT data type (https://lists.apache.org/thread/wyfnsj45s3cn95thrgw4oho74zbcc5vm).
In short, we agree to move away from the initial JNI / libbid proposal. The new proposal is to use a custom Java native IEEE 754 compliant port. Please refer to the updated SPIP document for additional details: https://docs.google.com/document/d/1qnLXm0ldHSwPSJs_Q5SzGyyTKMMEqeX5rm8Fh4rvV3E. Here is a brief TL;DR: - DECFLOAT should expose +/-Inf and NaN to allow exception free data processing without separate need for try_* operators collapsing all exceptions NULL or special casing open brackets. - Python, a priority Spark consumer, is IEEE 754 compatible. Friction free conversion of Python to/from Spark's runtime is important to Spark's ecosystem. - Similarly it is important to Spark's ecosystem to enable, rather than hinder other projects it is related to, such as Velox, Comet, or Parquet. None of which can rely on BigDecimal (which is Java only). - Intel's IEEE 754's library is BSD-3-Clause licensed, giving full access to Spark to port (SPARK-59111), optimize, validate, and extend the implementation in Java. Spark's eco system is empowered to do the same. The inverse is not true. GPLv2 with the Classpath Exception does not permit frictionless ports, easy optimization and extension within our ecosystem. - Agreeing that NaN and +/-Inf are necessary, implies reliance on BigDecimal becomes risky as it requires emulation. - IEEE 754 provides a clean, well defined, and easily defended contract across the entire runtime and storage ecosystem. - Such a contract becomes even more important as there are efforts in Iceberg and Delta to support expressions such as CHECK constraints and GENERATED columns. Rallying around a standard increases interoperability in the data lake. - Spark and its users have experienced the pain of relying on implied dependencies out of its control: Unexpected ICU changes, and obscure rules for Decimal conversion to String come to mind. The latter being BigDecimal specific. - The proposed IEEE 754 port is already faster (more performant) than BigDecimal across most operations, proving its potential. In particular, please refer to the tab "The case for IEEE 754" which includes the technical justification for conforming to the language-neutral IEEE 754 decimal-floating contract: https://docs.google.com/document/d/1qnLXm0ldHSwPSJs_Q5SzGyyTKMMEqeX5rm8Fh4rvV3E/edit?tab=t.g49bu2698eoe. On another note, we invite the Arrow community (cc: Ian Cook <[email protected]>) to discuss more about interoperability of the new DECFLOAT data type across Arrow. We are open to collaborate and engage on any details! Best, Uroš On 2026/08/25 11:18:58 Uroš Bojanić wrote: > Hi all, > > Discussion regarding DECFLOAT data type has settled, so I'm calling a vote on > the SPIP "Add the DECFLOAT data type". The SPIP proposes to add a new Spark > SQL data type: DECFLOAT (IEEE 754 decimal64 / decimal128), for base-10 > floating-point decimals with per-value exponents. The SPIP document has been > updated along the way to address the feedback raised in the doc & discussion > thread. > > DISCUSS thread: > https://lists.apache.org/thread/wyfnsj45s3cn95thrgw4oho74zbcc5vm > Spark JIRA ID: https://issues.apache.org/jira/browse/SPARK-58820 > SPIP document: > https://docs.google.com/document/d/1qnLXm0ldHSwPSJs_Q5SzGyyTKMMEqeX5rm8Fh4rvV3E > > The vote is open for at least the next 72 hours. > > [ ] +1: Accept the proposal as an official SPIP > [ ] +0 > [ ] -1: I don't think this is a good idea because... > > Thank you! > > Best, > Uroš > > --------------------------------------------------------------------- > To unsubscribe e-mail: [email protected] > > --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
