JAEKWANG97 opened a new issue, #13099:
URL: https://github.com/apache/seatunnel/issues/13099
## Background
FieldEncrypt reads `key` from configuration and keeps it in a String field.
The first call to `transformRow()` finds an `Encryptor` and calls `init(String
key)`. The initialized encryptor is transient. The existing interface also
defines `support(String)`, `encrypt(String)`, and `decrypt(String)`.
At commit `67941ddefacebbcadab6ba72f7e3877d10cbaee9`, two other parts of the
code affect startup. `AbstractMultiCatalogTransform` does not forward `open()`
or `close()` to its children. Zeta's `TransformFlowLifeCycle.open()` catches
initialization exceptions and logs them without rethrowing them. An `open()`
override in `FieldEncryptTransform` alone cannot ensure that key preparation
succeeds before row processing starts.
## Motivation
Users need to get data keys from an external key system. They should not
need to put a raw data key in the job configuration. FieldEncrypt should
support this choice while existing static-key jobs continue to work.
## Goals
- Keep existing options, defaults, ciphertext, and `Encryptor`
implementations compatible.
- Separate key preparation from local encryption and decryption.
- Prepare external keys before processing rows. Fail startup if preparation
fails.
- Recover the same keys after restart without remote calls during row
processing.
- Define a public interface for official and custom providers before
implementing it.
- Agree on the new ciphertext format, credential handling, and module layout
before adding a reference provider.
## Non-Goals
- Generate a new data key each time a worker starts.
- Add a SeaTunnel key database or require a particular sink.
- Refresh keys during row processing or automatically encrypt stored data
again.
- Fetch an unconfigured historical key when a row requests it.
- Export KMS master keys or call remote encryption APIs for each field.
- Add several vendor integrations in the first change.
## Design
### User configuration
I propose two paths. Without `key_provider`, FieldEncrypt uses the existing
`key` option and ciphertext format. A missing or blank `key` fails validation.
It keeps the current algorithm defaults and first-row initialization. We can
extract that initialization into a helper without changing when it runs.
With `key_provider`, FieldEncrypt prepares keys in `open()` and uses the new
format. This example proposes the configuration shape. These options and the
`example` provider do not exist yet.
```hocon
transform {
FieldEncrypt {
fields = ["phone"]
mode = "encrypt"
algorithm = "AES_GCM"
key_provider = "example"
key_provider_config {
endpoint = "https://keys.example.invalid"
credential_source = "worker"
write_key = { id = "K2", reference = "phone-key/version-2" }
read_keys = [
{ id = "K1", reference = "phone-key/version-1" },
{ id = "K2", reference = "phone-key/version-2" }
]
}
ciphertext_format = "v1"
ciphertext_prefix = "ENC:"
}
}
```
The transform owns `key_provider`, `key_provider_config`,
`ciphertext_format`, and `ciphertext_prefix`. Each new option needs a SeaTunnel
`Option` definition. The proposed defaults are no provider, an empty provider
block, `legacy` format, and `ENC:` prefix. The prefix applies only to the new
format. A provider requires explicit `ciphertext_format = "v1"`; other
combinations fail validation.
The selected provider defines and validates every field inside
`key_provider_config`. The example uses versioned references and credentials
from the worker environment. It contains no raw key or credential. A real
provider must document its authentication method and recovery inputs.
When both `key_provider` and `key` exist, the provider takes priority. A
provider failure fails startup; it never selects the static key as a fallback.
The ignored `key` can still enter existing job state. Review must settle this
mixed-configuration risk before the provider release.
### Proposed interfaces
The public SPI lets users add a provider without changing FieldEncrypt. SPI
means an interface that external implementations can use. The following
signatures are a proposal for review. They are not existing classes or an
implementation patch.
```java
public interface KeyProviderFactory {
String identifier();
void validate(ReadonlyConfig config) throws KeyProviderException;
KeyProvider create();
}
public interface KeyProvider extends AutoCloseable {
PreparedKeys prepare(ReadonlyConfig config, KeyRequest request)
throws KeyProviderException;
void close() throws KeyProviderException;
}
public interface PreparedKeys extends AutoCloseable {
Optional<PreparedKey> writeKey();
Map<String, PreparedKey> readKeys();
void close() throws KeyProviderException;
}
public interface DataCipher extends AutoCloseable {
String identifier();
void init(PreparedKeys keys, CipherOptions options) throws
CipherException;
String encrypt(String value) throws CipherException;
String decrypt(String value) throws CipherException;
void close() throws CipherException;
}
```
`KeyProviderFactory.identifier()` supplies the name used by `key_provider`.
Service discovery selects one factory by that name. Duplicate names fail
validation. `validate()` checks configuration without remote calls. `create()`
creates a worker-local provider without contacting the service.
`KeyRequest` contains the mode and algorithm. `prepare()` opens the provider
session and gets the required keys once per transform instance. If it fails,
the provider releases resources that it created before throwing a sanitized
exception.
`PreparedKey` contains a logical ID and a `java.security.Key` usable by the
selected cipher. It may contain a local cryptographic handle. It must not
contain an exported KMS master key. `PreparedKeys` returns an immutable map
indexed by logical ID. Reject conflicting recovery references for the same ID.
Encrypt mode requires a write key and includes it in the read map. Decrypt mode
requires read keys but no write key.
`CipherOptions` contains the algorithm, format, prefix, and size limits. It
contains no provider credentials or SDK clients. `DataCipher` checks format
headers and performs local cryptographic operations. Unknown keys fail through
the local map. It cannot fetch more keys.
The transform owns the provider, prepared keys, and cipher. It closes the
cipher first, then the prepared keys, then the provider. The cipher borrows
keys and must not destroy them. `PreparedKeys.close()` releases key resources.
`KeyProvider.close()` releases sessions and clients. Each close operation must
tolerate partial initialization and repeated calls. Exceptions must identify
the operation without exposing secret values or unsafe nested exceptions.
I propose a separate `DataCipher` SPI for the new path. The existing
`Encryptor` interface and service registration stay unchanged. Separate service
discovery selects a new cipher by its `identifier()` and the existing
`algorithm` option. Duplicate cipher identifiers fail validation. Existing
custom `Encryptor` implementations continue to work on the legacy path. They
need a `DataCipher` implementation to support provider keys and the new format.
### Startup, rows, and cleanup
`FactoryUtil` validates the option rules from
`FieldEncryptTransformFactory`. The factory returns a creator for the
multi-table wrapper. Neither contacts the key service or calls `open()`. The
proposed worker flow is:
```text
Zeta calls open() on the runtime transform
-> outer wrappers forward open(), if present
-> FieldEncryptMultiCatalogTransform calls each child's open()
-> the child creates a KeyProvider
-> prepare() gets the configured keys
-> DataCipher.init() checks and accepts the keys
-> the child becomes ready
For each row
-> transformRow() uses the ready DataCipher
-> the cipher selects a prepared key and processes the value locally
On shutdown
-> wrappers forward close() to their children
-> the child closes the cipher, prepared keys, and provider
```
I recommend separate prerequisite changes for common lifecycle forwarding
and Zeta startup failure propagation. Forwarding belongs in the common wrappers
and chains because they own their children. A FieldEncrypt-only override would
leave the same problem in composed transform chains. These changes need
regression tests for other transforms, Map/FlatMap chains, and error-handling
wrappers.
If one child fails, it cleans up its partial initialization. The wrapper
closes earlier children and rethrows the startup failure. Cleanup continues if
another close operation fails. The engine must stop startup before processing
rows. A child rejects rows before readiness, after failure, and after close.
Each child owns its resources. There is no shared JVM key cache. Mutable
cipher state must be isolated between concurrent calls. Providers need finite
timeouts, bounded retries, and cancellation support. Their limits need
agreement before implementation. Engine adapters and schema-change paths need
explicit verification before they are listed as supported.
### Restart and key rotation
A data encryption key, or DEK, encrypts field values. A KMS master key
protects that DEK and stays inside the key system. Both the old and new paths
use worker memory. Restart creates new runtime objects from configuration and
the key system.
For v1, users supply references to existing keys. The provider must recover
the same DEK from each reference after restart. Some providers also need an
immutable version or a protected data-key blob. Their configuration must supply
those recovery inputs. SeaTunnel adds no database for them.
In the example, `K1` and `K2` are IDs that the provider returns with the
keys. The versioned references tell the provider how to recover them. New
writes use K2. Reads can use K1 or K2 because startup prepares both keys. To
rotate again, users change the write reference and restart the job. They keep
older read references only while older data needs decryption.
The write key stays fixed during that execution. An unknown ciphertext key
ID fails locally. The provider does not generate a new key on each restart.
Review must set limits on the read-key count, startup duration, and memory use.
Revoking remote access does not remove a key already held in worker memory.
Provider documentation must explain the required stop/restart procedure.
Cleanup attempts to release key resources but cannot promise complete JVM
memory erasure.
### Ciphertext format and prefix
Legacy output stays unchanged. For the provider path, I propose a prefix
followed by a Base64-encoded envelope. The envelope contains a version, logical
key ID, algorithm ID, nonce, encrypted payload, and authentication tag. The
prefix must contain a character outside the legacy Base64 alphabet. The default
`ENC:` meets this rule.
The cipher must authenticate the prefix, header, and payload. AES-GCM is the
built-in choice for the new path. Custom ciphers must provide equivalent
authentication and local processing. Legacy AES-CBC stays available through the
existing path. A header does not add authentication to CBC.
I propose these default behaviors:
- Encrypt unmarked input as plaintext.
- If input has the configured prefix, parse and authenticate it with the
prepared read keys before preserving it unchanged.
- Reject malformed marked input, unknown keys or versions, and
authentication failures.
- In decrypt mode, require the configured prefix and format. Reject a
mismatch.
- Select legacy decoding through the legacy configuration. Do not guess
between formats or silently return plaintext.
Plaintext that starts with the reserved prefix therefore fails unless it is
a valid authenticated envelope. This restriction needs documentation. A mixed
legacy/new dataset needs separate processing or an agreed migration tool. The
binary field layout, encoding rules, and parser limits still need format review
before implementation.
### Security and modules
Only recovery references and approved non-secret options belong in
serialized provider configuration. Workers resolve credentials and create
runtime key objects. SDK clients, prepared keys, and live handles must not
enter job or checkpoint state. Each provider must define an allowed
configuration schema and reject inline secrets for this path.
Existing literal-key state and Zeta's row DEBUG/dry-run output need separate
review. The current code does not satisfy the issue's full secrecy requirement.
Check configuration output, exception causes, suppressed exceptions, metrics,
and error-row storage as well. The provider release must address these paths
before claiming that they expose no secrets.
I recommend a small encryption SPI jar separate from `seatunnel-api` and
vendor SDKs. A proposed name is `seatunnel-transform-encrypt-api`. The
transform and optional provider jars depend on it. Vendor SDKs belong only in
optional provider jars, such as `seatunnel-transform-encrypt-provider-<id>`.
These are proposed artifacts; the first code change creates none of them.
Packaged-jar tests must check discovery, classloader isolation, dependency
relocation, and licenses. The first reference provider remains a review choice.
A provider-free test double must first prove the lifecycle and zero remote
calls during row processing.
### Delivery phases
1. Agree on this design before implementation.
2. Extract static-key preparation without changing existing public
interfaces or first-row behavior. Add no cloud SDK, provider module, or public
SPI.
3. Review lifecycle forwarding and startup failure propagation in separate
prerequisite changes. Prove the flow with an internal provider-free test double.
4. After design acceptance, add the agreed public interfaces and artifacts,
then one reference provider. Add other providers separately.
### Decisions requested before acceptance
1. Accept the proposed configuration and interfaces, including separate
legacy and provider paths?
2. Accept common lifecycle fixes as separate prerequisites, and define the
engines and schema changes that v1 must support?
3. Accept configured key recovery and rotation by restart, and select one
reference provider?
4. Agree on the binary format, prefix behavior, and resource limits?
5. Resolve mixed-configuration secrets, credential transport, and row
diagnostics before releasing a provider?
## Compatibility, Deprecation, and Migration Plan
Keep every existing option, default, `Encryptor` method, package, and
registration path. Existing AES-CBC/AES-GCM data must remain decryptable.
Repeated encryption can produce different bytes because it uses random IVs.
Compatibility requires the same encoding and successful cross-decryption.
The new format requires explicit selection. Older releases may not read it.
Deploy compatible readers before changing writers and document rollback
restrictions. Keep the current `max_field_length` behavior. Define separate
limits for new encoded input and output.
Schema changes must preserve field selection and resource ownership.
FieldEncrypt computes field indexes in its constructor. Test index updates or
safely reject unsupported changes before processing affected rows.
Restart/restore must reconstruct transient readiness, keys, and cipher state.
Tests must cover same-version recovery; cross-version saved-job compatibility
needs separate evidence.
## Test Plan
Extend existing FieldEncrypt/Encryptor tests and `TestFieldEncryptIT`. Cover
legacy fixtures, cross-decryption, plugin loading, and null, empty, and long
values. Exercise the factory, wrappers, and worker together. Include multiple
tables, schema changes, initialization failures, cleanup failures,
cancellation, and restart/restore.
For 0, 1, and 10,000 rows, count calls during startup and row processing
separately. Assert zero remote calls during row processing, including failures
for unknown keys. After restart, decrypt K1 data with K1/K2 prepared and K2
selected for writes.
Test header and tag tampering, prefix collisions, authenticated
double-encryption checks, and resource limits. Use synthetic secret values to
detect exposure in logs, exceptions, metrics, and state. Test the reference
provider's recovery and installation with packaged jars and E2E tests. Update
English and Chinese documentation with the implementation.
These are planned tests. This draft reports no implementation, Maven build,
provider call, E2E result, or performance measurement. The implementation must
measure startup cost and throughput.
## Alternatives Considered
- Per-row KMS calls add remote access to row processing. Startup preparation
avoids this dependency.
- New keys on each restart require durable storage and recovery. Existing
versioned references avoid that addition in v1.
- A SeaTunnel key store adds another service to operate. Existing key
systems and deployment configuration can supply recovery inputs.
- Mandatory legacy initialization in `open()` changes direct calls and
failure timing for empty input. Keep first-row behavior initially.
- A JVM-wide cache can reduce startup calls but needs shared ownership,
isolation, and cleanup rules.
## References
- Feature and discussion: https://github.com/apache/seatunnel/issues/12407
- Implementation sequencing guidance:
https://github.com/apache/seatunnel/issues/12407#issuecomment-6246034419
- dev@ discussion request:
https://github.com/apache/seatunnel/issues/12407#issuecomment-5771146218
- Contribution process:
https://seatunnel.apache.org/community/contribution_guide/STIP/
- Current transform:
https://github.com/apache/seatunnel/blob/67941ddefacebbcadab6ba72f7e3877d10cbaee9/seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/encrypt/FieldEncryptTransform.java
- Existing Encryptor contract:
https://github.com/apache/seatunnel/blob/67941ddefacebbcadab6ba72f7e3877d10cbaee9/seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/encrypt/encryptor/Encryptor.java
- Multi-table lifecycle:
https://github.com/apache/seatunnel/blob/67941ddefacebbcadab6ba72f7e3877d10cbaee9/seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/common/AbstractMultiCatalogTransform.java
- Zeta lifecycle/diagnostics:
https://github.com/apache/seatunnel/blob/67941ddefacebbcadab6ba72f7e3877d10cbaee9/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/task/flow/TransformFlowLifeCycle.java
- Configuration baseline:
https://github.com/apache/seatunnel/blob/67941ddefacebbcadab6ba72f7e3877d10cbaee9/seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/encrypt/FieldEncryptTransformConfig.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]