JNSimba opened a new pull request, #70467: URL: https://github.com/apache/doris/pull/70467
### Background The job scheduler submits ordinary insert tasks and streaming job control tasks to the same insert executor. Streaming data tasks already execute through a separate scheduler, while periodic control tasks handle metadata, split progress and lifecycle checks. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Bugfix Problem Summary: Long-running insert jobs can occupy the insert executor and delay streaming control tasks. Add an independent bounded executor for jobs whose existing execution mode is STREAMING, covering both timer dispatch and immediate submission. Keep streaming jobs classified as INSERT so existing job queries and metrics retain their classification. Most of the diff adds isolation tests. ### Details - **Frequency:** When insert workers and their queue are occupied, streaming control work competes for the same capacity. - **Root cause:** Executor selection uses JobType alone; both ordinary insert jobs and streaming insert jobs return INSERT. - **Why existing tests missed it:** The existing manager test covers queue-size normalization, without exercising contention between insert and streaming tasks. - **Fix:** Register a streaming control TaskProcessor and select it using the existing JobExecuteType.STREAMING in both dispatch paths. Add startup-only master configuration: job_streaming_task_consumer_thread_num defaults to 10, and streaming_task_queue_size defaults to 1024. - **Other paths and branches:** Ordinary insert and MTMV routing, task rejection callbacks and pending-task cleanup keep their existing behavior. No persisted job fields or protocol values change. The common timer/dispatch stage remains shared. ### Testing On the development machine, against master 74bcd1aa1735350decf959f586df6728de7b7955 plus this patch: - `bash run-fe-ut.sh --run org.apache.doris.job.manager.TaskDisruptorGroupManagerTest,org.apache.doris.job.executor.TaskProcessorTest,org.apache.doris.job.base.AbstractJobStatusTest,org.apache.doris.job.base.JobExecutionConfigurationTest`: BUILD SUCCESS; 28 tests, 0 failures, 0 errors, 0 skipped. The manager isolation test accounts for 6 tests. - `mvn checkstyle:check -pl fe-common,fe-core`: BUILD SUCCESS; 0 violations. - `git diff --check`: passed. No live-cluster regression or throughput benchmark was run. ### Release note Streaming job control tasks use a separate executor from ordinary insert jobs. Operators can configure its worker count and bounded waiting queue with job_streaming_task_consumer_thread_num and streaming_task_queue_size; both settings require an FE restart. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: Yes. Add the two startup configuration entries to the FE configuration reference in a follow-up documentation PR. - Pre-submission status: Partially verified: focused FE unit tests and Checkstyle passed; live-cluster regression and throughput measurements were not run. ### AI Disclosure - Tool: Codex (GPT-6.1). - Generated: Code, unit tests and this description. - Human review: The author reviewed and approved the design; final diff review is pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
