nzw921rx commented on PR #11077:
URL: https://github.com/apache/seatunnel/pull/11077#issuecomment-6845260477

   I ran a local validation on the PR base (`9cd7defe`), without this PR’s 
changes, using 200 same-schema FakeSource tables, one Sink subtask, and 
`multi_table_sink_replica = 1`.
   
   | Pipeline | Paimon Sink instances | Paimon Writers |
   |---|---:|---:|
   | Source → Paimon | 200 | 200 |
   | Source → TableMerge → Paimon | 1 | 1 |
   
   With `TableMerge`, 2,000 records were successfully written, committed, and 
read back from Paimon. All 200 source tables were represented, with no missing 
or duplicate records.
   
   This test used Paimon’s local filesystem catalog. It does not reproduce the 
original MySQL + Hive Metastore environment, but it confirms that `TableMerge` 
eliminates the per-source-table Sink/Writer creation in this same-schema case.
   
   Given the substantial changes to the core multi-table sink lifecycle, could 
you first verify whether **Source → TableMerge → Paimon** resolves the original 
issue?
   
   If `TableMerge` cannot cover a specific requirement, please explain that 
limitation. We could then consider extending it or introducing a dedicated 
transform to handle the merging before the Sink, keeping the change within the 
Transform layer where possible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to