wolfboys opened a new pull request, #4512:
URL: https://github.com/apache/streampark/pull/4512
## Purpose
This PR establishes a clean architecture baseline for StreamPark 3.0.
The change intentionally removes obsolete modules, duplicated
implementations, unused dependencies, and historical compatibility layers. It
also rebuilds several core boundaries so that subsequent backend, frontend, and
runtime improvements can be developed on a simpler and more maintainable
foundation.
This is a deliberate 3.0 cleanup. Deprecated internal structures are removed
instead of being carried forward indefinitely.
## Major Changes
### 1. Simplify the module structure
- Remove the obsolete `streampark-spark-cli` module.
- Remove the standalone `streampark-flink-udf` module.
- Move `FlinkShimsProxy` into the Flink client API and remove the standalone
`streampark-flink-proxy` module.
- Rename `streampark-common-scala-bridge` to the clearer `streampark-scala`
module.
- Reorganize Flink client APIs into explicit request, response,
configuration, and implementation packages.
- Remove unused classes, utilities, Maven dependencies, and outdated package
structures.
### 2. Retire the legacy E2E implementation
- Remove the deprecated `streampark-e2e` modules and their associated
workflow.
- Remove E2E infrastructure that is tightly coupled to the previous frontend
and backend architecture.
- Reserve the E2E workflow for reconstruction after the StreamPark 3.0
frontend and backend boundaries are stabilized.
The removal is intentional and temporary at the project level. A new E2E
workflow will be designed around the refactored architecture.
### 3. Rebuild the common configuration framework
Replace the historical `common.conf` implementation with a typed and
immutable configuration model under `common.configuration`.
The new configuration framework provides:
- Typed `ConfigOption` declarations.
- Immutable configuration snapshots.
- Explicit configuration source precedence.
- Configuration origin tracking and actionable conversion errors.
- Structured YAML, HOCON, properties, environment, system property,
command-line, and runtime sources.
- Module-owned option catalogs without global option registries.
- Atomic process-level configuration publication.
- Constant-style workspace access backed by a validated configuration
snapshot.
- A dedicated Console-side Spring configuration initializer.
Common configuration remains independent of Flink and Spark runtime APIs.
### 4. Fix Flink configuration compatibility
Reimplement Flink configuration discovery and parsing according to the
target Flink version instead of the StreamPark compile-time Flink version.
The supported behavior is:
- Flink versions before 1.19 load `flink-conf.yaml` with the legacy Flink
line parser.
- Flink 1.19 and 1.20 support both filenames and prefer `flink-conf.yaml`
when both files exist.
- `flink-conf.yaml` uses legacy parsing, while `config.yaml` uses standard
YAML parsing.
- Flink 2.0 and later load only `config.yaml` with standard nested YAML
parsing.
- Nested maps are flattened into dot-separated Flink configuration keys.
- YAML collections are preserved using the representation expected by Flink.
This resolves the historical compatibility issues when StreamPark manages
Flink 1.x and 2.x installations from the same deployment.
### 5. Consolidate the Flink shims architecture
- Remove the duplicated `streampark-flink-shims-base-v2` module.
- Consolidate shared Flink behavior into a single shims-base implementation.
- Remove the obsolete shims test module and redundant implementations.
- Remove the unsupported Flink 1.17 shim.
- Make Flink 1.18, 1.19, 1.20, 2.0, 2.1, 2.2, and 2.3 explicit reactor
modules.
- Keep only APIs supported by every version in shared abstract classes.
- Keep legacy APIs such as `registerDataStream` and table-function
`registerFunction` overloads only in compatible Flink 1.x implementations.
- Preserve classloader isolation through serializable StreamPark request and
response contracts.
No general utilities or target-independent configuration logic are placed in
the shims layer.
### 6. Clarify Console responsibilities
- Move Spring configuration composition into
`SpringConfigurationInitializer`.
- Keep Console-specific options in the Console module.
- Move Flink environment and application configuration behavior out of
persistence entities.
- Keep database entities focused on table mappings and persisted state.
- Introduce typed REST response envelopes while retaining the existing wire
format.
- Separate request DTOs, response DTOs, assemblers, services, and entities
more clearly.
### 7. Improve HTTP client and proxy handling
- Introduce a shared OkHttp client with connection pooling and bounded
retries for idempotent requests.
- Rework Console proxy handling to preserve multi-value headers and response
status.
- Filter hop-by-hop and connection-declared headers correctly.
- Ensure upstream responses are always closed.
- Avoid creating a new HTTP client for every proxy request.
### 8. Improve repository and developer tooling
- Add a root development launcher for building, starting, stopping, and
inspecting a source checkout.
- Align build and distribution banners with the StreamPark 3.0 version.
- Refresh repository metadata and remove the retired E2E required check and
label mapping.
- Update NOTICE files for 2026 and ignore the local pnpm store.
### 9. Improve maintainability
- Remove stale and inaccurate comments.
- Add class-level and core implementation documentation.
- Simplify method names and keep new production and test method names within
40 characters.
- Remove redundant and overlapping tests.
- Update `AGENTS.md` to describe the current module boundaries and
architecture rules.
- Preserve Apache license, formatting, Checkstyle, and RAT requirements.
## Compatibility Notes
This PR intentionally introduces internal breaking changes for StreamPark
3.0:
- The legacy `org.apache.streampark.common.conf` package is replaced by
`org.apache.streampark.common.configuration`.
- Obsolete modules and internal APIs are removed.
- Flink 1.17 shims are removed.
- The previous E2E implementation is removed and will be rebuilt separately.
- Flink client request and response types are reorganized around explicit
package boundaries.
User-facing configuration keys retain their defined compatibility aliases
where required.
## Verification
The following checks have been completed:
- `streampark-common`: 102 tests passed.
- Console configuration, proxy, Flink environment, SQL, and savepoint tests:
12 tests passed.
- Flink client request resolution tests: 11 tests passed.
- Full Flink shims reactor build passed for Flink 1.18 through 2.3.
- Spotless and Checkstyle passed for the tested modules.
- Apache RAT passed for the common, client, and shims modules.
- Flink configuration parsing was verified with:
- Flink 1.16.3 `flink-conf.yaml`
- Flink 1.19.1 `config.yaml`
- Flink 2.3.0 `config.yaml`
## Follow-up Work
- Rebuild the E2E framework and CI workflow against the finalized 3.0
frontend and backend contracts.
- Continue reducing deprecated compatibility APIs after downstream
migrations are complete.
- Run the complete release CI matrix before the StreamPark 3.0 release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
