[
https://issues.apache.org/jira/browse/IMPALA-14586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Borók-Nagy updated IMPALA-14586:
---------------------------------------
Description:
Iceberg doesn't use 64-bit RoaringBitmaps for some reason...
They use an array of 32-bit RoarngBitmaps to cover the range of 0..2^64-1.
See
[https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/deletes/RoaringPositionBitmap.java]
-They have a custom serialization and deserializtion algorithm as well. For
best performance, we should implement our Roaring bitmap similarly.-
They use the portable serialization format of the roaring bitmaps:
[https://github.com/RoaringBitmap/RoaringFormatSpec?tab=readme-ov-file#extension-for-64-bit-implementations]
So we only need to implement serialization/deserializtion for our bitmaps.
was:
Iceberg doesn't use 64-bit RoaringBitmaps for some reason...
They use an array of 32-bit RoarngBitmaps to cover the range of 0..2^64-1.
See
[https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/deletes/RoaringPositionBitmap.java]
They have a custom serialization and deserializtion algorithm as well. For best
performance, we should implement our Roaring bitmap similarly.
> Implement Iceberg-conform Roaring bitmap
> ----------------------------------------
>
> Key: IMPALA-14586
> URL: https://issues.apache.org/jira/browse/IMPALA-14586
> Project: IMPALA
> Issue Type: Sub-task
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: impala-iceberg
>
> Iceberg doesn't use 64-bit RoaringBitmaps for some reason...
> They use an array of 32-bit RoarngBitmaps to cover the range of 0..2^64-1.
> See
> [https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/deletes/RoaringPositionBitmap.java]
> -They have a custom serialization and deserializtion algorithm as well. For
> best performance, we should implement our Roaring bitmap similarly.-
> They use the portable serialization format of the roaring bitmaps:
> [https://github.com/RoaringBitmap/RoaringFormatSpec?tab=readme-ov-file#extension-for-64-bit-implementations]
> So we only need to implement serialization/deserializtion for our bitmaps.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
