yidaimi opened a new pull request, #13200:
URL: https://github.com/apache/seatunnel/pull/13200

   ### Purpose of this pull request
   
   Yes — this pull request is a contribution to SeaTunnel.
   
   Fixes #13198. Related to the Amazon DocumentDB sink discussion in #10443.
   
   `MongodbSinkFactory` only passed `buffer-flush.max-rows`, 
`buffer-flush.interval`, `retry.max` and `retry.interval` to 
`MongodbWriterOptions.Builder` when the options were explicitly configured. 
When they were omitted, the builder kept its Java zero values instead of the 
defaults declared in `MongodbSinkOptions`:
   
   | Option | Declared / documented default | Effective value before this PR 
when omitted |
   |---|---|---|
   | `buffer-flush.max-rows` | 1000 | 0 |
   | `buffer-flush.interval` | 30000 ms | 0 |
   | `retry.max` | 3 | 0 |
   | `retry.interval` | 1000 ms | 0 |
   
   With `flushSize = 0`, `MongodbWriter.isOverMaxBatchSizeLimit()` 
(`bulkRequests.size() >= 0`) is always true, so the writer issued one 
`bulkWrite` per row; with `retryMax = 0`, failed bulk writes were never retried.
   
   This PR always reads the four options with `readonlyConfig.get(...)`, so an 
omitted option falls back to its declared default and an explicit value keeps 
working as before. No option name, declared default or other option handling is 
changed.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes, for jobs that do **not** set these options. Their behavior now matches 
the documented defaults (`docs/en|zh/connectors/sink/MongoDB.md` already list 
1000 / 30000 / 3 / 1000, so no doc change is needed):
   
   1. **Batching**: rows are written in batches of up to 1000 rows, or after 30 
s, or on checkpoint / timer flush / close, instead of one `bulkWrite` per row. 
This is a large throughput improvement on remote servers (20,000 inserts 
against Amazon DocumentDB in the same VPC: 56 s -> 3 s engine time, see 
#13198), but data becomes visible in the target in batches rather than row by 
row.
   2. **Retries**: a failed bulk write is now retried up to 3 times with the 
documented back-off, instead of failing immediately. Retrying re-sends the 
whole ordered batch, which is the same behavior users already get today when 
they set `retry.max` explicitly; in insert mode (`upsert-enable = false`) a 
retry after a partially applied batch can therefore surface duplicate-key 
errors.
   
   Jobs that set the options explicitly are not affected. Reviewers: please let 
me know if you would like an entry in `incompatible-changes.md` for these two 
points.
   
   ### How was this patch tested?
   
   New unit test `MongodbSinkOptionDefaultsTest`. It creates the sink through 
`MongodbSinkFactory` exactly as the engine does and observes the writer 
behavior against a mocked collection (no server needed):
   
   | Test | Before this PR | After this PR |
   |---|---|---|
   | `omittedBufferOptionsUseDeclaredDefaults` (1000 rows, bulk writes) | 1000 
| 1 |
   | `explicitBufferOptionsAreStillApplied` (control, max-rows = 10) | 100 | 
100 |
   | `omittedRetryMaxUsesDeclaredDefault` (attempts) | 1 | 4 |
   | `explicitRetryMaxIsStillApplied` (control, retry.max = 1) | 2 | 2 |
   | `omittedRetryIntervalUsesDeclaredDefault` (back-off before the retry) | ~2 
ms | >= 1000 ms |
   
   The "before" column was produced by running the new test against unmodified 
`dev`.
   
   Local verification on the rebased branch:
   
   ```shell
   ./mvnw spotless:apply -pl seatunnel-connectors-v2/connector-mongodb
   ./mvnw -pl seatunnel-connectors-v2/connector-mongodb test      # Tests run: 
48, Failures: 0, Errors: 0
   ./mvnw -q -DskipTests verify -pl seatunnel-connectors-v2/connector-mongodb
   ```
   
   The behavior was also measured end to end against a live Amazon DocumentDB 
5.0.2 endpoint and a local MongoDB 7 with the released 3.0.0 sink (cases P01 / 
P01b / X1a): 
https://github.com/yidaimi/seatunnel-cosmos-to-documentdb/blob/v0.2-delete_insert/test/sink-compat/evidence-excerpts.md
   
   This change was prepared with the help of Claude Code (AI assistant) and 
reviewed by me; the commit carries a `Generated-by` trailer as recommended by 
the ASF generative tooling guidance.
   
   ### Check list
   
   * [x] No new Jar binary package is added.
   * [x] Documentation already describes the corrected defaults 
(`docs/en/connectors/sink/MongoDB.md`, `docs/zh/connectors/sink/MongoDB.md`); 
no change needed.
   * [ ] `incompatible-changes.md`: not updated yet, see the question above.
   * [x] Not a new connector, so `plugin-mapping.properties`, 
`seatunnel-dist/pom.xml`, `label-scope-conf.yml`, e2e module and 
`plugin_config` are unchanged.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to