github-actions[bot] commented on code in PR #69528:
URL: https://github.com/apache/doris/pull/69528#discussion_r4371468958
##########
be/src/format_v2/table/lance_reader.cpp:
##########
@@ -628,6 +887,10 @@ Status LanceTableReader::_ensure_dataset_open(const
TFileRangeDesc& range,
} else if (!_opened_dataset_key.has_value() || *_opened_dataset_key !=
key) {
return Status::InvalidArgument(
"Lance reader cannot mix dataset snapshots or storage
options");
+ } else {
+ // Revalidate reused handles before scanners, indexes or row-id reads.
Fresh handles
+ // are checked inside _open_dataset before they become reader state.
+ RETURN_IF_ERROR(_validate_dataset_storage(_dataset,
key.storage_policy));
Review Comment:
[P2] Reuse remote storage admission for an unchanged snapshot
For a normal S3 Lance scan with F fragments and B registered remote bases,
`TVFScanNode.getLanceSplits()` emits one fragment per split, and each split
returns here with the same `_dataset`/`DatasetKey`. This call nevertheless
reconstructs, sorts, serializes, parses and validates every base through the
new JSON API again. The pinned SDK's [public add_bases
API](https://github.com/lance-format/lance/blob/68c12dfd7efe02f90ad2d7f3a239a7eb64884e57/rust/lance/src/dataset.rs#L2013-L2043)
can register many distinct remote bases, including unused ones, so 1,000
fragments and 1,000 registrations cause about a million base descriptions
instead of one admission pass per opened handle. The added work scales with F
times B, plus sorting, despite an unchanged immutable snapshot and policy.
Cache successful REMOTE validation for that handle/key after `_open_dataset`,
preserving LOCAL rechecks needed for filesystem changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
