bridgeDream opened a new pull request #13: URL: https://github.com/apache/incubator-doris-flink-connector/pull/13
# Proposed changes ## Problem Summary: 目前，dorisSink 刷入 batch 的时机依赖 "batchSize" 和 "batchIntervalMs"，一般情形下是能够符合预期的；但是当数据源流量或者单条消息数据量极大时，可能会造成单个 batch 的数据很大，进而会导致数据刷入 doris 时长时间阻塞影响实时性能，甚至请求 BE 超时无法正常写入的问题。 出于上诉原因，提供依据 "maxBatchBytes" 刷 batch 到 doris 中，修改点： * 类 `DorisExecutionOptions`: 增加一个可配置选项 `maxBatchBytes` * 类 `DorisDynamicOutputFormat`: 估计当前 batch 的数据量 `batchBytes` 达到阈值 `maxBatchBytes` 时，刷入 doris。 ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (No Need) 3. Has document been added or modified: (No Need) 4. Does it need to update dependencies: (No) 5. Are there any changes that cannot be rolled back: (No) ## Further comments -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
