Gabriel39 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5289814826
One further scope clarification after comparing v5 with the original issue: metadata reads are needed here only to manage indexes. This issue should not grow into a query-engine, general external-transaction, or permanent reconciliation project for 4.2. Please narrow the 4.2 implementation and release gates to the original lifecycle requirements. **Keep for 4.2:** - Lance metadata remains the source of truth. - Read one pinned snapshot for the fields needed by index management. - `SHOW INDEX` exposes logical name, columns, type, and bounded properties. - At most one additional physical inspection surface exposes UUID and dataset version when they do not fit the existing SHOW schema. - `IF EXISTS`, `IF NOT EXISTS`, and REPLACE use the same authoritative name/type/column comparison. - Admission records and revalidates the required dataset version and schema contract. - Directory CREATE, CREATE OR REPLACE, and DROP run asynchronously on one selected BE in a resource-isolated worker. - There is no automatic mutation retry and no cancellation after dispatch. - A lost or ambiguous result becomes UNKNOWN and retains the same-name fence. - Provide one explicit audited force-release path and refresh metadata after known completion or operator resolution. - Keep the basic privilege, bounded-resource, quota, crash-safety, and vector/scalar end-to-end tests needed to prove the created indexes are usable. **Defer to follow-up issues unless already trivial and independently releasable:** - the duplicate logical `lance_indexes()` TVF when `SHOW INDEX` already represents logical indexes; - exact row counts, indexed/unindexed row or fragment coverage, and FE `countRows()` work in normal SHOW paths; - a permanent automatic UNKNOWN watcher/reconciliation daemon; - ACK plus multiple FORCE variants, permanent tombstones, and the full operator-resolution event framework beyond one force-release operation; - a full provider-backed table-incarnation subsystem and normalization-version upgrade protocol; - broad generic catalog-wide external-DDL transaction machinery beyond protecting the stable locator and same index name needed by this feature; - REST read support as a prerequisite for Directory mutation. REST inspection may ship independently, but it must not gate the original Directory lifecycle; - future REST mutation profiles; - query-side architecture such as distributed fragment-parallel vector TopN, candidate merging, coverage-freshness SLA, or a reusable query I/O recorder. One focused end-to-end test confirming that the existing query path consumes a newly created vector/scalar index is sufficient for this issue; - incremental BUILD, progress, running cancellation, distributed index construction, and other capabilities already listed as non-goals. The durable distinction between mutation outcome and metadata refresh, and the UNKNOWN same-name fence from the previous review, must remain. However, the rest of the five-dimensional lifecycle machinery should not automatically become a 4.2 delivery requirement when a smaller state model can satisfy the original issue safely. Please update the release boundary or provide a smaller implementation plan reflecting this split. The advanced design can remain useful future work, but the current deliverable should focus on create, inspect, replace/rebuild, drop, the minimum safe failure semantics, tests, and documentation requested by the original issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
