Lukasz Antoniak created CASSANALYTICS-199:
---------------------------------------------
Summary: Utilize SAI indexes for filtering in bulk read path
Key: CASSANALYTICS-199
URL: https://issues.apache.org/jira/browse/CASSANALYTICS-199
Project: Apache Cassandra Analytics
Issue Type: Improvement
Components: Reader
Reporter: Lukasz Antoniak
h2. Objective
Improve bulk readers' performance by leveraging SAI indexes to scan only
partition keys that potentially satisfy filter condition of the Spark data set.
Currently, library supports only push-down filters for partition key columns.
Filters based on non-partition key columns do not limit the result set returned
from Cassandra Analytics to Spark. Appropriate filtering is performed by Spark
engine.
{code:java}
bulkReaderDataFrame(TABLE).load()
.filter("age > 30")
.select("name")
.collectAsList();{code}
h2. Approach
The bulk reader opens all SSTables participating in the requested token range
and passes them through the normal compaction / reconciliation path.
_SSTableReader_ already supports restricting the scan to a supplied list of
partition key filters, which are applied consistently across the participating
SSTables.
The proposed SAI integration opens the relevant SAI indexes and creates a
long-lived, lazy iterator of matching primary keys. Cassandra Analytics
converts this stream into batches of unique partition keys, for example up to
1024 keys per batch. Each batch is then passed to {_}SSTableReader{_}, which
reads only those candidate partitions from all participating SSTables before
normal reconciliation is performed.
If additional SAI matches remain, the same iterator is advanced to obtain the
next batch, without reopening the SAI indexes.
{code:java}
Spark WHERE
│
├── age > 30
└── gender = 'female'
│
V
QueryPlanner
│
┌────────────┴───────────┐
│ │
age expression gender expression
│ │
V V
For each SSTable:
age segment 1 gender segment 1
age segment 2 gender segment 2
age segment 3 gender segment 3
│ │
V V
UNION segments UNION segments
│ │
V V
age result gender result
per SSTable per SSTable
│ │
every predicate is evaluated
independently across all SSTables
│ │
V V
UNION across UNION across
ALL SSTables ALL SSTables
│ │
└──────────┬────────────┘
│
V
INTERSECT
│
│ lazy / long-lived
│ KeyRangeIterator
│
V
batch of up to 1024
partition keys
│
V
PartitionKeyFilters
│
V
read candidate partitions from
the participating Data.db SSTables
│
V
CompactionStreamScanner
reconciliation across SSTables
tombstones / updates / TTLs
│
V
reconstructed rows
│
V
Spark filtering
│
V
final result rows{code}
h2. Implementation details
# Unclear if we can re-use SAI {_}QueryController{_}, _QueryViewBuilder_ and
{_}StorageAttachedIndexSearcher{_}, because those may query memtables and
trigger read commands.
# Re-use _IndexSegmentSearcher_ to read SAI index for given column and return
partition keys. Opening multiple SAI segments needs to be reconciled with
_KeyRangeUnionIterator_ before passing the keys to {_}SSTableReader{_}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
