Gabriel39 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5290050744
This is a scope clarification and should be used as the 4.2 implementation boundary. For 4.2, the mutation requirement is intentionally small: > Safely execute one one-shot Lance mutation. If the result is uncertain, do not retry it, do not report a false success/failure/cancellation, retain the same-name fence, and allow an explicit audited manual release. The advanced mechanisms described in v5 are useful design material, but they are not requirements for the current 4.2 code or release gates. ## Required in 4.2 ### Original index lifecycle - Directory Catalog CREATE for the initial verified vector and scalar index types. - CREATE OR REPLACE as a full rebuild. - DROP by logical name. - `SHOW INDEX` for logical index metadata. - One bounded physical inspection surface only if needed to expose UUID and dataset version that do not fit the existing SHOW schema. - Basic `IF EXISTS` / `IF NOT EXISTS` behavior using authoritative Lance metadata. - Table SHOW/ALTER privileges, basic same-name serialization, metadata refresh, tests, and documentation. ### Minimum safe one-shot execution - Persist the job before dispatch. - Execute at most once on one selected BE through a resource-isolated worker. - Do not automatically retry after dispatch may have happened. - Do not offer running cancellation. - A complete typed result may report COMMITTED or NOT_COMMITTED. - A missing, partial, lost, crashed, timed-out, or otherwise ambiguous post-dispatch result becomes UNKNOWN. - UNKNOWN retains the same-name fence. - Provide one explicit audited FORCE RELEASE operation so an operator can accept the late-commit risk and unblock the name. - Keep mutation outcome and metadata refresh distinct so refresh success never fabricates a mutation result. - Revalidate the required dataset version and a small versioned schema contract before invocation. - Add simple unresolved-job quotas, allow credential rotation, and do not reuse a hard possible-live worker slot until termination proof or manual force release. - Include focused OOM/panic/failover tests and one end-to-end check that each created index type is consumable by the existing query path. No new query architecture is required. A compact lifecycle such as `PENDING -> RUNNING -> COMMITTED | NOT_COMMITTED | UNKNOWN`, plus refresh status and one manual force-release path, is sufficient for the current implementation. ## Move to follow-up issues ### Permanent UNKNOWN automation - automatic watcher/reconciliation daemons; - persisted backoff, jitter, periodic observations, and watcher metrics; - automatic long-term handling of permanently UNKNOWN jobs; - ACKNOWLEDGED plus multiple FORCE variants, including FORCE WITHOUT REFRESH; - a general immutable resolution-event framework; - permanent UNKNOWN tombstones and advanced archival/replay behavior beyond retaining the unresolved job and fence needed for the manual operation. ### External object identity and compatibility framework - provider-backed table/dataset incarnation; - detecting external deletion and recreation of another dataset at the same URI; - normalization-version migration and mixed-version fence protocols; - a general cross-release schema canonicalization framework beyond the small versioned schema contract needed by the current operation; - broad catalog-wide external-DDL transaction/guard machinery beyond preventing changes to the active target locator and same index name. For 4.2, external replacement of a dataset at the same URI may be documented as unsupported while a mutation is active or UNKNOWN. ### Extended metadata and REST surfaces - duplicate logical inspection APIs when `SHOW INDEX` already provides the logical view; - exact row counts, indexed/unindexed row coverage, fragment coverage, and FE `countRows()` work in normal SHOW paths; - rich physical consistency/provenance diagnostics beyond the fields required by the original issue; - REST inspection as a prerequisite for Directory mutation; - any REST mutation profile or REST mutation lifecycle. REST inspection may be delivered independently later, but it must not gate the Directory lifecycle. ### Distributed and advanced index/query capabilities - distributed index construction and segment commit; - distributed fragment-parallel vector TopN and candidate merging; - incremental BUILD or coverage repair; - build progress and running cancellation; - automatic retry or idempotency protocols; - continuously appended-table freshness SLA; - reusable query I/O recorder infrastructure; - additional index types, composite indexes, nullable-column support, and richer physical metadata. Please revise v5 or its implementation plan so that the follow-up sections above are explicitly non-normative for 4.2, are removed from current release gates, and are tracked by separate issues if we decide to build them. The current code should focus on the original index lifecycle plus the minimum safe one-shot UNKNOWN/manual-release semantics described above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
