yidaimi opened a new pull request, #13200: URL: https://github.com/apache/seatunnel/pull/13200
### Purpose of this pull request Yes — this pull request is a contribution to SeaTunnel. Fixes #13198. Related to the Amazon DocumentDB sink discussion in #10443. `MongodbSinkFactory` only passed `buffer-flush.max-rows`, `buffer-flush.interval`, `retry.max` and `retry.interval` to `MongodbWriterOptions.Builder` when the options were explicitly configured. When they were omitted, the builder kept its Java zero values instead of the defaults declared in `MongodbSinkOptions`: | Option | Declared / documented default | Effective value before this PR when omitted | |---|---|---| | `buffer-flush.max-rows` | 1000 | 0 | | `buffer-flush.interval` | 30000 ms | 0 | | `retry.max` | 3 | 0 | | `retry.interval` | 1000 ms | 0 | With `flushSize = 0`, `MongodbWriter.isOverMaxBatchSizeLimit()` (`bulkRequests.size() >= 0`) is always true, so the writer issued one `bulkWrite` per row; with `retryMax = 0`, failed bulk writes were never retried. This PR always reads the four options with `readonlyConfig.get(...)`, so an omitted option falls back to its declared default and an explicit value keeps working as before. No option name, declared default or other option handling is changed. ### Does this PR introduce _any_ user-facing change? Yes, for jobs that do **not** set these options. Their behavior now matches the documented defaults (`docs/en|zh/connectors/sink/MongoDB.md` already list 1000 / 30000 / 3 / 1000, so no doc change is needed): 1. **Batching**: rows are written in batches of up to 1000 rows, or after 30 s, or on checkpoint / timer flush / close, instead of one `bulkWrite` per row. This is a large throughput improvement on remote servers (20,000 inserts against Amazon DocumentDB in the same VPC: 56 s -> 3 s engine time, see #13198), but data becomes visible in the target in batches rather than row by row. 2. **Retries**: a failed bulk write is now retried up to 3 times with the documented back-off, instead of failing immediately. Retrying re-sends the whole ordered batch, which is the same behavior users already get today when they set `retry.max` explicitly; in insert mode (`upsert-enable = false`) a retry after a partially applied batch can therefore surface duplicate-key errors. Jobs that set the options explicitly are not affected. Reviewers: please let me know if you would like an entry in `incompatible-changes.md` for these two points. ### How was this patch tested? New unit test `MongodbSinkOptionDefaultsTest`. It creates the sink through `MongodbSinkFactory` exactly as the engine does and observes the writer behavior against a mocked collection (no server needed): | Test | Before this PR | After this PR | |---|---|---| | `omittedBufferOptionsUseDeclaredDefaults` (1000 rows, bulk writes) | 1000 | 1 | | `explicitBufferOptionsAreStillApplied` (control, max-rows = 10) | 100 | 100 | | `omittedRetryMaxUsesDeclaredDefault` (attempts) | 1 | 4 | | `explicitRetryMaxIsStillApplied` (control, retry.max = 1) | 2 | 2 | | `omittedRetryIntervalUsesDeclaredDefault` (back-off before the retry) | ~2 ms | >= 1000 ms | The "before" column was produced by running the new test against unmodified `dev`. Local verification on the rebased branch: ```shell ./mvnw spotless:apply -pl seatunnel-connectors-v2/connector-mongodb ./mvnw -pl seatunnel-connectors-v2/connector-mongodb test # Tests run: 48, Failures: 0, Errors: 0 ./mvnw -q -DskipTests verify -pl seatunnel-connectors-v2/connector-mongodb ``` The behavior was also measured end to end against a live Amazon DocumentDB 5.0.2 endpoint and a local MongoDB 7 with the released 3.0.0 sink (cases P01 / P01b / X1a): https://github.com/yidaimi/seatunnel-cosmos-to-documentdb/blob/v0.2-delete_insert/test/sink-compat/evidence-excerpts.md This change was prepared with the help of Claude Code (AI assistant) and reviewed by me; the commit carries a `Generated-by` trailer as recommended by the ASF generative tooling guidance. ### Check list * [x] No new Jar binary package is added. * [x] Documentation already describes the corrected defaults (`docs/en/connectors/sink/MongoDB.md`, `docs/zh/connectors/sink/MongoDB.md`); no change needed. * [ ] `incompatible-changes.md`: not updated yet, see the question above. * [x] Not a new connector, so `plugin-mapping.properties`, `seatunnel-dist/pom.xml`, `label-scope-conf.yml`, e2e module and `plugin_config` are unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
