github-actions[bot] commented on code in PR #70727:
URL: https://github.com/apache/doris/pull/70727#discussion_r4623358418


##########
thirdparty/patches/lance-c-foyer.patch:
##########
@@ -6776,32 +6818,50 @@ index 
0000000000000000000000000000000000000000..430b4c891f91d31f202e2318e18db64c
 +    Ok(0)
 +}
 diff --git a/src/dataset.rs b/src/dataset.rs
-index 
cc1f87ce3f7dfe8dd33aeb6ab7f54d02e31a220a..76fd39ef9e456d1fab6978b6b2cb3c27fab1c2f3
 100644
+index 
cc1f87ce3f7dfe8dd33aeb6ab7f54d02e31a220a..34ce3e4f4d43037b73b1126308f05402c3dbaeeb
 100644
 --- a/src/dataset.rs
 +++ b/src/dataset.rs
-@@ -14,6 +14,7 @@ use lance::Dataset;
+@@ -14,15 +14,18 @@ use lance::Dataset;
  use lance::dataset::builder::DatasetBuilder;
  use lance_core::Result;
 
 +use crate::data_cache::DatasetDataCache;
  use crate::error::{ffi_try, swallow_unwind};
  use crate::helpers;
  use crate::runtime::block_on;
-@@ -23,6 +24,7 @@ use crate::stream_guard::guarded_ffi_stream_from_reader;
+ use crate::session::LanceSession;
++use crate::session_cache::storage_identity;
+ use crate::stream_guard::guarded_ffi_stream_from_reader;
+
  /// Opaque handle representing an opened Lance dataset.
  pub struct LanceDataset {
      pub(crate) inner: RwLock<Arc<Dataset>>,
 +    pub(crate) data_cache: Option<Arc<dyn DatasetDataCache>>,
  }
 
  impl LanceDataset {
-@@ -182,8 +184,16 @@ unsafe fn open_dataset_inner(
+@@ -170,6 +173,9 @@ unsafe fn open_dataset_inner(
+
+     let opts = unsafe { helpers::parse_storage_options(storage_options)? };
+
++    let environment = std::env::vars_os()
++        .filter_map(|(key, value)| Some((key.into_string().ok()?, 
value.into_string().ok()?)));
++    let identity = storage_identity(&opts, environment);

Review Comment:
   [P1] Include profile-resolved endpoints in the storage identity
   
   This hashes only raw options and environment values, but the pinned Lance 
AWS provider also resolves `endpoint_url` from the selected AWS profile 
(`providers/aws.rs`, `ResolvedS3StorageOptions::new` and `new_store`). With 
`AWS_PROFILE` and `AWS_CONFIG_FILE` unchanged, edit that profile from endpoint 
A to B and restart using the same disk cache. Both opens get the same identity, 
`s3$bucket` prefix and data-file keys, so a file present at the same 
bucket/path on both endpoints can be served from A's recovered bytes while 
querying B. The explicit-option/environment tests cannot detect this remaining 
cross-origin wrong-result case. Resolve and include the effective provider 
endpoint before choosing the namespace, and add a profile-file change/reopen 
regression.



##########
thirdparty/patches/lance-c-foyer.patch:
##########
@@ -8313,14 +8853,377 @@ index 
60a16234557eaeb93e8eaad971a9eba73a562a83..9ed8cfe4b5207afc78c65163e8cc4c7d
  ) -> Result<*mut LanceSession> {
      let index_cache_size_bytes = u64_to_usize(index_cache_size_bytes, 
"index_cache_size_bytes")?;
      let metadata_cache_size_bytes =
-@@ -58,6 +68,7 @@ fn session_new_inner(
-     );
+         u64_to_usize(metadata_cache_size_bytes, "metadata_cache_size_bytes")?;
+-    let session = Session::new(
+-        index_cache_size_bytes,
+-        metadata_cache_size_bytes,
+-        Default::default(),
+-    );
++    let index_cache_backend = SharedSessionCache::new(index_cache_size_bytes);
++    let metadata_cache_backend = 
SharedSessionCache::new(metadata_cache_size_bytes);
      Ok(Box::into_raw(Box::new(LanceSession {
-         inner: Arc::new(session),
+-        inner: Arc::new(session),
++        sessions: Mutex::new(HashMap::new()),
 +        data_cache_factory,
++        index_cache_backend,
++        metadata_cache_backend,
      })))
  }
 
+@@ -97,8 +111,8 @@ unsafe fn session_get_cache_stats_inner(
+     }
+     let session = unsafe { &*session };
+     let (index, metadata) = block_on(async {
+-        let index = session.inner.index_cache_stats().await;
+-        let metadata = session.inner.metadata_cache_stats().await;
++        let index = session.index_cache_backend.stats().await;
++        let metadata = session.metadata_cache_backend.stats().await;
+         (index, metadata)
+     });
+     unsafe {
+@@ -126,3 +140,73 @@ fn u64_to_usize(value: u64, field: &'static str) -> 
Result<usize> {
+         )
+     })
+ }
++
++/// Clear shared cache entries without replacing the session or invalidating
++/// already-open datasets. Concurrent reads can populate new entries 
afterward.
++#[unsafe(no_mangle)]
++pub unsafe extern "C" fn lance_session_clear_caches(session: *const 
LanceSession) -> i32 {
++    ffi_try!(unsafe { session_clear_caches_inner(session) }, neg)
++}
++
++unsafe fn session_clear_caches_inner(session: *const LanceSession) -> 
Result<i32> {
++    // A null handle is a caller error, just as it is for the statistics API.
++    if session.is_null() {
++        return Err(lance_core::Error::invalid_input_source(
++            "session must not be NULL".into(),
++        ));
++    }
++    let session = unsafe { &*session };
++    block_on(async {
++        session.index_cache_backend.clear().await;
++        session.metadata_cache_backend.clear().await;
++    });
++    if let Some(factory) = &session.data_cache_factory {
++        factory.clear()?;
++    }
++    Ok(0)
++}
++
++impl LanceSession {
++    pub(crate) fn for_storage(&self, identity: &str) -> Arc<Session> {
++        // Lance scopes these caches by URI alone. The same s3://bucket/table
++        // on two endpoints must also isolate manifests, deletions and 
indices,
++        // before a data-cache wrapper is attached after dataset loading.
++        // Lance's live-store reuse key omits provider environment values.
++        // Keep registries within the same identity as all cache tiers, while
++        // retaining reuse for live datasets that address the same origin.
++        let mut sessions = self
++            .sessions
++            .lock()
++            .unwrap_or_else(|error| error.into_inner());
++        if let Some(session) = sessions.get(identity).and_then(Weak::upgrade) 
{
++            return session;
++        }
++        sessions.retain(|_, session| session.strong_count() > 0);
++        let session = Arc::new(Session::with_cache_backends(
++            CacheSpec::Backend(self.index_cache_backend.namespaced(identity)),
++            
CacheSpec::Backend(self.metadata_cache_backend.namespaced(identity)),
++            Default::default(),
++        ));
++        sessions.insert(identity.to_owned(), Arc::downgrade(&session));

Review Comment:
   [P2] Keep cache state alive for retained legacy IVF indexes
   
   Only a weak reference to this `Session` survives here. After the last 
dataset/scan for an identity closes, its `LanceCache` state can therefore die 
even though `SharedSessionCache` retains cached legacy IVF objects. In the 
pinned Lance code, legacy index formats 0.0/0.1/0.2 cache the live `IVFIndex`, 
which holds `WeakLanceCache`; reopening returns that same object rather than 
rebinding its cache. Its partition lookups then always miss and insertions 
become no-ops, forcing repeated partition reads/decoding until that index is 
evicted. Modern IVF state reconstruction avoids this, so the existing modern 
prewarm/reopen test misses the regression. Preserve the originating cache 
state's lifetime or reconstruct legacy entries with the current state, and 
cover close-last-handle/reopen queries on a legacy IVF index.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to