jiangxintong created SPARK-58831:
------------------------------------
Summary: Add bitmap scalar set operation functions
Key: SPARK-58831
URL: https://issues.apache.org/jira/browse/SPARK-58831
Project: Spark
Issue Type: New Feature
Components: SQL
Affects Versions: 5.0.0
Reporter: jiangxintong
h2. Problem
Spark's flat-bitmap infrastructure provides functions for constructing,
aggregating, and counting
bitmaps, including {{bitmap_construct_agg}}, {{bitmap_or_agg}},
{{bitmap_and_agg}},
{{bitmap_count}}, {{bitmap_bucket_number}}, and {{bitmap_bit_position}}.
However, Spark does not provide scalar set operations for combining two
precomputed bitmaps from
the same row. Users currently need a UDF or application-side processing to
calculate bitmap
intersection, union, difference, or symmetric difference. A native
implementation keeps these
operations visible to Catalyst and whole-stage codegen and provides consistent
SQL, Scala,
PySpark, and Spark Connect APIs.
h2. Proposed Functions
||Function||Signature||Description||
|bitmap_and|bitmap_and(BINARY, BINARY) -> BINARY|Returns the intersection of
two bitmaps.|
|bitmap_or|bitmap_or(BINARY, BINARY) -> BINARY|Returns the union of two
bitmaps.|
|bitmap_andnot|bitmap_andnot(BINARY, BINARY) -> BINARY|Returns the bits present
in the left bitmap but not in the right bitmap.|
|bitmap_xor|bitmap_xor(BINARY, BINARY) -> BINARY|Returns the symmetric
difference of two bitmaps.|
h2. Semantics
* Both arguments use Spark's existing flat-bitmap Binary representation, not a
RoaringBitmap
serialization.
* Each input may contain between 0 and 4096 bytes.
* Missing bytes in a shorter input are treated as zero.
* The result is always a new 4096-byte Binary value.
* If either input is NULL, the result is NULL.
* An input longer than 4096 bytes raises the structured error
{{BITMAP_INPUT_TOO_LARGE}}.
* {{bitmap_andnot(left, right)}} is directional and computes
{{left AND NOT right}}.
* These are scalar functions that combine two bitmaps from the same row.
Existing
{{bitmap_*_agg}} functions continue to combine bitmaps across rows.
h2. Examples
{code:sql}
SELECT substring(hex(bitmap_and(X 'F0', X '70')), 0, 2);
-- 70
SELECT substring(hex(bitmap_or(X '10', X '20')), 0, 2);
-- 30
SELECT substring(hex(bitmap_andnot(X 'F0', X '70')), 0, 2);
-- 80
SELECT substring(hex(bitmap_xor(X 'F0', X '70')), 0, 2);
-- 80
{code}
h2. API Surface
* Spark SQL functions: {{bitmap_and}}, {{bitmap_or}}, {{bitmap_andnot}}, and
{{bitmap_xor}}.
* Scala DataFrame functions under {{org.apache.spark.sql.functions}}.
* PySpark classic and Spark Connect functions under {{pyspark.sql.functions}}.
h2. Scope and Compatibility
* The change is additive and does not alter the behavior of existing bitmap
functions.
* It does not introduce a new bitmap storage format or change bucket-number or
bit-position
mapping.
* It does not add or modify aggregate bitmap functions.
* Existing SQL queries and APIs are unaffected.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
