goutamadwant commented on code in PR #12776:
URL: https://github.com/apache/seatunnel/pull/12776#discussion_r4278424585
##########
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-mysql-e2e/src/test/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/MysqlCDCIT.java:
##########
@@ -38,4 +37,25 @@ public MysqlCDCIT() {
new UniqueDatabase(
MYSQL_CONTAINER, MYSQL_DATABASE, "mysqluser",
"mysqlpw", MYSQL_DATABASE);
}
+
+ @TestTemplate
+ @DisabledOnContainer(
+ value = {},
+ type = {EngineType.SEATUNNEL, EngineType.FLINK},
+ disabledReason = "Covered by testMysqlCdcCheckDataE2e")
+ public void testMysqlCdcCheckDataE2eOnSpark(TestContainer container) {
Review Comment:
Thanks for checking @Carl-Zhou-CN I agree that Spark should behave like Zeta
and Flink. this gap isn't specific to CDC, though:
- The Spark starter only uses `spark.read()` and never `readStream`, so no
source gets checkpoint restore on Spark today, and STREAMING jobs run as one
long batch task. The Spark 4.1 PR #11174 documents the same: checkpoint
settings such as `checkpoint.interval` are not applied by the Spark starter.
- That's also why the shared JDBC-sink tests can't pass on Spark unchanged:
they rely on checkpoint-triggered flushes that never happen there.
- This PR changes CDC on Spark from failing on the first snapshot split to
running correctly, without changing Zeta or Flink, and documents the restore
limit.
For matching semantics, I see two options:
1. Land this as a first step, and follow up with micro-batch streaming on
Spark (enumerator and reader state saved per batch, sink committed per batch),
so CDC and other sources get checkpoint restore.
2. Keep this PR open and add that work over here.
Which one should we prefer?
@davidzollo @Carl-Zhou-CN
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
