madongz opened a new pull request #12: URL: https://github.com/apache/incubator-doris-flink-connector/pull/12
# Proposed changes Issue Number: close #xxx ## Problem Summary: At present, Doris has supported 2pc. On this basis, we have implemented doris-flink-connector based on Flink's unified sink API, DorisWriter uploads data via http in streaming mode and DorisCommitter commit the transaction after checkpoint completed. ## Main Change Points 1. Replace output format with DorisSink. 2. Unified serialization interface, user could implement DorisRecordSerializer to define how to serialize record. 3. Upload data in streaming mode. ## Data processing flow 1. when job start or checkpoint, doris writer send new http request to doris for load. 2. write data into data stream. 3. before next checkpoint, mark data load finished and get the http response. 4. get txnId from response and send to doris committer. 5. after checkpoint completed, doris committer commit the transactions. ## Checklist(Required) 1. Does it affect the original behavior: (Yes) 7. Has unit tests been added: (Yes) 8. Has document been added or modified: (Todo) 9. Does it need to update dependencies: (No) 10. Are there any changes that cannot be rolled back: (No) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
