bzhang-obsec opened a new pull request, #70649:
URL: https://github.com/apache/doris/pull/70649

   ### Background
   
   An OLAP `INSERT INTO ... SELECT` creates an `InsertLoadJob` and registers it 
with the master FE's `LoadManager` in PENDING when
   `OlapInsertExecutor` is constructed. The job becomes FINISHED or CANCELLED 
only through `LoadManager.recordFinishedLoadJob`, which
   `OlapInsertExecutor.afterExec` / `onFail` call once the insert has run. 
`SHOW LOAD` and the `doris_fe_job` metric read these jobs, and
   the label cleaner removes only completed ones.
   
   ### What problem does this PR solve?
   
   Issue Number: close #70632
   Related PR: #62282 (fixed the same leak for other insert paths), #62889 (its 
backport)
   PR Type: Bugfix
   
   Problem Summary:
   
   If the planner reduces the source of an OLAP `INSERT INTO ... SELECT` to an 
empty set, the load job is registered but never finished. Two
   examples are partition pruning on a LIST-partitioned table that has no 
partition for the filtered value, and `WHERE 1 = 0`. The job stays
   PENDING in the master FE until it restarts, so a workload that often hits 
this leaks heap without bound. The fix records the registered job
   as FINISHED with 0 rows on that early-return path. After upgrading, these 
inserts show up in `SHOW LOAD` as FINISHED with 0 rows instead of
   PENDING.
   
   ### Details
   
   - **Frequency:** Every OLAP `INSERT ... SELECT` whose source is planned as a 
single `VEMPTYSET`, except `INSERT ... VALUES`, transaction
     mode and group commit, which do not register a job. The reporter saw about 
140k stuck jobs per day on a production master. That filled a
     16 GiB heap and caused a stop-the-world full GC about every 5 minutes. 
Once more than `label_num_threshold` jobs were stuck, every run of
     the label cleaner also deleted all finished load jobs, so `SHOW LOAD` kept 
less than an hour of history.
   - **Root cause:** `InsertIntoTableCommand.runInternal` returns early when 
`insertExecutor.requiresTransaction()` is false, which is the
     case for an empty source without Table Stream offset updates. On that path 
neither `afterExec` nor `onFail` runs, so the job registered
     in the constructor is never finished. `LoadJob.isExpired()` is false for 
uncompleted jobs and `removeOverLimitLoadJob` removes only
     completed ones, so the job is never cleaned up. Registering a job doesn't 
write an edit-log entry, so it exists only in the master's memory
     and disappears on restart.
   - **Why existing tests missed it:** No test inserted from a source the 
planner reduces to an empty set and then checked the job's state.
   - **Fix:** Add `AbstractInsertExecutor.onSkipExecution(StmtExecutor)`, a 
hook that does nothing by default. `runInternal` calls it on the
     no-transaction early return. `OlapInsertExecutor` overrides it to call the 
same `recordLoadJob` that `afterExec` uses, but only when its
     constructor registered the job. No transaction is faked: the job is 
recorded with transaction id `-1`, and the client response is
     unchanged.
   - **Other paths and branches:** `OlapTxnInsertExecutor`, 
`OlapGroupCommitInsertExecutor`, `RemoteOlapInsertExecutor` and the
     external-table executors don't register a job, so for them the hook does 
nothing. The empty-source paths in `RewriteTableCommand` and
     `RowLevelDmlCommand` don't use a registering executor. The same leak 
exists on branch-4.1 (4.1.3 and 4.1.5), where the early return
     checks `shouldExecuteEmptyInsert()`, so this fix should be backported.
   - **Behavior change:** These inserts now appear in `SHOW LOAD` as FINISHED 
with 0 rows. Previously they showed
     as PENDING until the FE restarted. Each one now also writes one 
`logCreateLoadJob` edit-log entry, the same as an executed insert.
   
   ### Testing
   
   - `OlapInsertExecutorTest`: `testSkipExecutionFinishesRegisteredLoadJob` 
checks that a registered empty insert calls
     `recordFinishedLoadJob` with no fail message, which makes the job 
FINISHED. `testSkipExecutionRecordsNothingForUnregisteredLoadJob` checks
     that an executor which didn't register a job doesn't touch `LoadManager`.
   - `load_p0/insert/test_insert_statistic`: runs the issue's repro. The source 
is an auto LIST-partitioned table filtered on a value with no
     partition, and the test asserts its plan contains `VEMPTYSET`. It also 
runs a `WHERE 1 = 0` source. For each labelled insert it asserts
     `SHOW LOAD` reports FINISHED, then asserts no job in the database is 
PENDING. Before this fix both labels report PENDING.
   
   None of these were run locally. They are left to CI.
   
   ### Notes for Reviewers
   
   I left the issue's suggested safeguard, expiring old PENDING insert jobs in 
the label cleaner, out of this PR. A live insert is also
   PENDING until it completes, so a job's age alone doesn't show that it's 
orphaned.
   
   ### Release note
   
   Fix `INSERT INTO ... SELECT` statements whose source is planned empty, such 
as one filtered on a partition that doesn't exist, leaving
   PENDING load jobs in the master FE that leaked memory until restart.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: No
   - Pre-submission status: Partially verified: the new unit test and 
regression case have not been run locally; they rely on CI.
   
   ### AI Disclosure
   
   - Tool: Claude Code (Claude Opus 5.5)
   - Generated: Code, tests, and this description
   - Human review: 
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to