LiJie20190102 opened a new pull request, #13904:
URL: https://github.com/apache/gravitino/pull/13904
### What changes were proposed in this pull request?
Adds `StaleRegistrationReconciler`, a server-side periodic task that detects
stale schema registrations of non-managed catalogs, following the
`RelationalGarbageCollector` pattern: it owns a daemon single-thread
`ScheduledThreadPoolExecutor` with `start()`/`close()`, wired in `GravitinoEnv`.
Each round enumerates the schema registrations of every metalake's
non-managed catalogs and confirms absence with an explicit `schemaExists`
probe. A registration is reported stale only when all of the following hold:
- the catalog does not manage storage for the schema scope
(`capabilities().managedStorage(SCHEMA)`);
- the probe explicitly reports the object absent;
- the probe did not fail for infrastructural reasons — a probe failure marks
the catalog unreachable and skips it entirely for the round ("never
report/delete on doubt").
Four configuration keys are added, all under `gravitino.reconcile.*`:
`enabled` (false), `intervalSecs` (3600), `catalogJitterSecs` (60) and `policy`
(report-only). A per-catalog random jitter spreads probe load, and a failing
catalog is logged and skipped without aborting the round.
For hierarchical namespaces, detected stale names are reduced to the
outermost ones so a future cascading drop of a parent carries its descendants
along.
An internal report model (`StaleCatalogReport`, `StaleSchemaEntry`) carries
the per-catalog results for a future REST surface. Nothing is removed in this
first step: every entry is reported with `removed == false`, and the
auto-remove policy is applied by a follow-up change through the dispatcher
chain.
Out of scope: removal itself (subtask 2), topic reconcile (subtask 3), REST
endpoints (subtask 4), and tables/views/filesets.
### Why are the changes needed?
For non-managed catalogs the source system is the source of truth and
Gravitino keeps only a registration row per entity. When an object is dropped
out-of-band in the source, the registration stays behind: it keeps showing up
in `list*` results and makes `load*` return 404 for catalog-walking consumers.
The existing ad-hoc workaround removes rows via `EntityStore.delete` directly,
bypassing the dispatcher chain and skipping secret cleanup, authorization
plugin revocation, events and tree locks.
This is the first step of the stale-registration reconcile epic: detect and
report safely first, remove later.
Fix: #13903
### Does this PR introduce _any_ user-facing change?
No, by default. The task is disabled (`gravitino.reconcile.enabled` is
`false`), and when enabled it defaults to the `report-only` policy: nothing is
deleted, and no REST API is exposed in this step. Enabling it introduces the
four new `gravitino.reconcile.*` configuration keys and per-round log output.
### How was this patch tested?
- Unit tests for scheduling: no task is scheduled when disabled, a round is
scheduled when enabled, and jitter stays within `[0, catalogJitterSecs * 1000)`.
- Unit tests for the three stale conditions: managed catalogs and catalogs
without schema operations are skipped, probe infrastructure failures mark the
catalog unreachable, and a confirmed-absent registration is reported.
- Unit tests for per-catalog error isolation: a failing catalog does not
block the reconcile of the others.
- Unit tests for outermost stale selection over hierarchical, mixed, flat
and empty name sets.
- End-to-end unit tests: a registration present in the entity store with no
source object is reported stale with `removed == false`, and policy/config
defaults are asserted.
- `./gradlew spotlessApply` and `:core:test` pass on top of the latest main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
