Ystk-hsn opened a new pull request, #13506:
URL: https://github.com/apache/gravitino/pull/13506
### What changes were proposed in this pull request?
- Creating a `MANAGED` fileset on a location that already exists and is not
empty
is now rejected with a message pointing to `EXTERNAL` or to the new
`allow-existing-location-as-managed` property. Empty existing directories
are
still adopted as before.
- New mutable fileset property `delete-data-on-drop` (default `true`).
`dropFileset` and the cascade branch of `dropSchema` share one check,
`deletesStorageOnDrop`, and skip the storage delete when it is `false`.
- Both properties are declared in `FilesetPropertiesMetadata`; docs updated
in
`fileset-catalog.md` and `filesets.md`.
### Why are the changes needed?
A managed fileset takes its storage with it on drop. Pointing one at a
directory that already holds data, which the docs already say should be
registered as `EXTERNAL`, was accepted silently and deleted that data on
drop.
Rejecting the non-empty case enforces the documented contract at the source,
mirroring the rule Spark applies to managed tables since 2.4, and
`delete-data-on-drop` gives a per-fileset safety net for filesets that
already
exist.
Fix: #12550
### Does this PR introduce _any_ user-facing change?
Yes.
- Creating a `MANAGED` fileset on an existing non-empty location now fails
unless `allow-existing-location-as-managed=true` is set.
- New fileset properties `allow-existing-location-as-managed` (default
`false`,
immutable) and `delete-data-on-drop` (default `true`).
Filesets that do not hit the non-empty case behave exactly as before.
### How was this patch tested?
Added to `TestFilesetCatalogOperations`:
- `testCreateManagedFilesetRejectsNonEmptyExistingLocation`
- `testCreateManagedFilesetAdoptsEmptyExistingLocation`
- `testCreateManagedFilesetOnNonEmptyLocationWithOverride`
- `testDropFilesetKeepsStorageWhenDeleteDataOnDropIsFalse`
- `testDropFilesetKeepsStorageWhenDeleteDataOnDropIsSetAfterCreation`
- `testDropSchemaCascadeKeepsStorageWhenDeleteDataOnDropIsFalse`
`./gradlew :catalogs:catalog-fileset:test -PskipITs` — 192 tests, 0 failures.
`./gradlew :api:build :catalogs:catalog-fileset:build -PskipITs -x
:catalogs:catalog-fileset:test`
(spotlessCheck, checkstyle, javadoc) passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
