taozhiyi13 commented on issue #12923: URL: https://github.com/apache/seatunnel/issues/12923#issuecomment-6603223045
@DanielLeens Thanks for pointing out the wider use of `MySqlTypeConverter`. After tracing the affected catalog, dialect, and sink paths, I agree that changing its default `GEOMETRY` mapping would expand the first-version impact beyond the intended scope. ## Compatibility inventory The current `GEOMETRY -> BYTES` behavior is used by more than the MySQL JDBC Source: - `MySqlCatalog` uses `MySqlTypeConverter` for JDBC schema discovery. - MySQL-CDC also calls the converter through `MySqlTypeUtils`. - `TiDBCatalog` extends `MySqlCatalog`. - `StarRocksDialect` extends `MysqlDialect`. - Every `jdbc:mysql:` URL selects the MySQL dialect unless another compatible mode changes the dialect implementation. - `MysqlCreateTableSqlBuilder` can reuse the source column type when auto-creating a MySQL sink table. This means the existing MySQL JDBC behavior is not only an incomplete spatial abstraction. It is also a usable opaque-binary transport contract: ```text MySQL GEOMETRY -> SeaTunnel BYTES -> file, Kafka, JDBC, or a custom binary-capable sink ``` In particular, a MySQL-to-MySQL job can currently carry the JDBC-returned MySQL geometry bytes as `BYTES`, auto-create a `GEOMETRY` column from the preserved source type, and write the bytes through the JDBC sink. Changing the shared default mapping to `GEOMETRY` would therefore affect MySQL-to-MySQL jobs, TiDB catalog paths, the StarRocks JDBC-compatible dialect, and other existing JDBC pipelines whose sinks do not support the new type. `ST_AsWKB(column)` is not a drop-in replacement for the MySQL-to-MySQL path. It returns standard WKB without MySQL's four-byte SRID prefix, so it preserves a `BYTES` schema but not the historical MySQL internal representation. The native StarRocks connector and OceanBase MySQL mode use separate type converters and are not affected by this specific mapping change. ## Difference between JDBC and CDC The JDBC and CDC compatibility situations are different. For JDBC, the existing `BYTES` path can successfully transport opaque geometry values even though it does not expose their spatial semantics. For MySQL-CDC, a non-null spatial value is received as a Debezium geometry structure. The existing `BYTES` conversion path cannot decode that structure into a usable byte array, so there is no working historical end-to-end contract for non-null CDC spatial values. Introducing `GeometryType` and `Geometry` on the CDC path is therefore a new capability. Changing the JDBC default would instead replace an existing binary transport contract. ## Revised first-version decision Based on this inventory, I propose to revise the earlier JDBC compatibility decision and remove MySQL JDBC spatial-type support from the first version of STIP-42. The first version will be limited to: ```text MySQL-CDC -> GeometryType / Geometry -> Zeta -> Elasticsearch/OpenSearch ``` The expected behavior will be: - MySQL JDBC Source continues to expose `GEOMETRY` as `BYTES`. - The default/JDBC mode of `MySqlTypeConverter` remains unchanged. - MySQL-to-MySQL binary transport remains unchanged. - TiDB and StarRocks JDBC-compatible paths remain unchanged. - MySQL JDBC Sink remains outside the spatial contract. - MySQL-CDC explicitly builds a `GeometryType` schema and converts Debezium spatial values into `Geometry`. - Elasticsearch/OpenSearch consumes that shared geometry contract. - The strict full-cluster upgrade boundary still applies to CDC spatial jobs because the new schema and row classes cross Zeta serialization boundaries. This keeps the first implementation focused on the original unsupported path rather than changing all existing JDBC consumers. ## Tests The revised test plan will verify that: 1. MySQL JDBC generic `GEOMETRY` remains `BYTES`. 2. Existing JDBC binary row conversion remains unchanged. 3. MySQL-to-MySQL does not receive the new `Geometry` value. 4. TiDB and StarRocks JDBC-compatible mappings do not implicitly adopt `GeometryType`. 5. MySQL-CDC maps all supported MySQL spatial types to `GeometryType`. 6. MySQL-CDC snapshot and binlog values are converted to `Geometry`. 7. Elasticsearch/OpenSearch correctly consumes the CDC geometry schema and values. The previous `BYTES -> GEOMETRY(GEOMETRY)` JDBC migration contract, `ST_AsWKB` migration requirement, and JDBC before/after schema test will be removed because the first version no longer changes JDBC behavior. ## Future JDBC support JDBC spatial support can be proposed separately after the shared CDC-to-sink contract is established. A future design could either require an explicit JDBC source mode or introduce a planner-level sink capability contract so that `GeometryType` is enabled only for an end-to-end topology whose downstream components all support it. That broader source-to-sink negotiation is not required for the first version. ## Confirmation requested Could you please confirm whether deferring MySQL JDBC spatial support and limiting the first version to MySQL-CDC -> Zeta -> Elasticsearch/OpenSearch is an acceptable scope for STIP-42? If this revised scope is acceptable, I will update the normative STIP body with: 1. the compatibility inventory and the decision to keep the existing JDBC `GEOMETRY -> BYTES` behavior unchanged; 2. the MySQL-CDC-only spatial source contract; 3. the previously agreed strict full-cluster upgrade boundary and restore behavior; 4. the revised test plan and implementation scope. Please let me know whether this narrower first-version scope resolves the JDBC compatibility concern. If so, I will update the normative STIP body as described above, then start the `[DISCUSS][STIP-42]` thread on `[email protected]` and link the mailing-list discussion back to this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
