Wilson-BT commented on issue #11258: URL: https://github.com/apache/doris/issues/11258#issuecomment-1198066436
> > Yes, encapsulate such a data structure, dynamically load by spelling the URL on the sink side, and add a keyby operator before sink > > At present, flink-doris-connector initiates the stream load of the table when the flink task starts, instead of doing the stream load when the upstream data is received. How to do this dynamic stream load? Please describe your design in detail～ Many users put all the canal logs of all tables in the business library into one topic, which needs to be distributed before they can use doris-flink-connector. His idea is to edit a task to synchronize the entire library. Because currently doris-flink-connector uses http inputstream, that is, a checkpoint opens a stream, and a streamLoad url is strongly bound. In this case, we can only use the flink side to cache data, and then a table generates a buffer, and bind the corresponding table-streamload-url, set a threshold, such as rows number or batch size to submit tasks, just like doris-datax-writer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
