github-actions[bot] commented on code in PR #69277:
URL: https://github.com/apache/doris/pull/69277#discussion_r4323296683
##########
be/src/format_v2/lance/lance_session_manager.cpp:
##########
@@ -231,9 +237,46 @@ Status LanceSessionManager::open_dataset(const char* uri,
const char* const* sto
*dataset = lance_dataset_open_with_session(uri, storage_options, version,
_session);
if (*dataset == nullptr) {
+ // Constructing an SDK error Status can itself log the provider
message. Redact before
+ // that construction, not only when returning the prewarm RPC response.
+ if (redact_errors) {
+ return Status::InternalError(
+ "Unable to open the pinned Lance snapshot for index
prewarm");
+ }
return lance_error("open Lance dataset with shared session");
}
return Status::OK();
}
+Status LanceSessionManager::prewarm_index(const char* uri, const char* const*
storage_options,
+ uint64_t version, const char*
index_name) {
+ if (uri == nullptr || *uri == '\0' || version == 0 || index_name ==
nullptr ||
+ *index_name == '\0') {
+ return Status::InvalidArgument(
+ "Lance index prewarm requires URI, index name and a fixed
version");
+ }
+ // Bound concurrent SDK prewarm IO per BE without blocking all RPC workers
behind a mutex.
+ // A failed or timed-out caller may retry after the existing SDK operation
finishes.
+ std::unique_lock lock(_prewarm_mutex, std::try_to_lock);
+ if (!lock.owns_lock()) {
+ return Status::Cancelled("Another Lance index prewarm is running on
this backend");
+ }
+ LanceDataset* raw_dataset = nullptr;
+ auto status = _open_dataset(uri, storage_options, version, &raw_dataset,
true);
+ std::unique_ptr<LanceDataset, decltype(&lance_dataset_close)>
dataset(raw_dataset,
+
lance_dataset_close);
+ if (!status.ok()) {
+ // SDK storage errors can include credentials or signed URLs. Return
operation context only.
+ return Status::InternalError("Unable to open the pinned Lance snapshot
for index prewarm");
+ }
+ if (lance_dataset_version(dataset.get()) != version) {
+ return Status::InternalError("Lance index prewarm opened an unexpected
snapshot");
+ }
+ if (lance_dataset_prewarm_index(dataset.get(), index_name) != 0) {
Review Comment:
[P2] Resolve legacy scalar fields before SDK prewarm
On a cold BE, a Lance v0.18.0 BTREE on `id` normally has logical name
`id_idx` and no `index_details`. FE now correctly discovers it, but this call
still fails: pinned Lance-C delegates to `Dataset::prewarm_index("id_idx")`
when no schema field is named `id_idx`; the SDK passes that name through
`open_generic_index`/`open_scalar_index` to
[infer_scalar_index_details](https://github.com/lance-format/lance/blob/68c12dfd7efe02f90ad2d7f3a239a7eb64884e57/rust/lance/src/index/scalar.rs#L689-L709),
which requires `schema.field("id_idx")`. Thus both `WARM UP SELECT id ...` and
`SELECT *` fail for a valid index that ordinary scans open using `id`. This is
a BE field-resolution gap after successful discovery, distinct from the
previous FE omission. Resolve scalar fields from segment metadata as well while
retaining SDK prewarm/FTS validation, and cover a cold-session legacy index
whose name differs from its column.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
