morningman opened a new issue #6803: URL: https://github.com/apache/incubator-doris/issues/6803
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description Currently, even if there is no more data in kafka, the routine load job will still send task to try to consume from the kafka topic. And you will see a lot of `kafka consume timeout` in be.INFO. And finally this task is aborted, and the `abortedTaskNum` of routine load will increase by 1. So I think we can try to fetch the latest offset of kafka partition before deciding whether to send task to consume. This will bring 2 benefits: 1. Avoid unnecessary tasks running on BE, which occupying working thread. 2. Avoid lots of aborted transactions. ### Use case _No response_ ### Related issues _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
