bzhang-obsec opened a new pull request, #70649:
URL: https://github.com/apache/doris/pull/70649
### Background
An OLAP `INSERT INTO ... SELECT` creates an `InsertLoadJob` and registers it
with the master FE's `LoadManager` in PENDING when
`OlapInsertExecutor` is constructed. The job becomes FINISHED or CANCELLED
only through `LoadManager.recordFinishedLoadJob`, which
`OlapInsertExecutor.afterExec` / `onFail` call once the insert has run.
`SHOW LOAD` and the `doris_fe_job` metric read these jobs, and
the label cleaner removes only completed ones.
### What problem does this PR solve?
Issue Number: close #70632
Related PR: #62282 (fixed the same leak for other insert paths), #62889 (its
backport)
PR Type: Bugfix
Problem Summary:
If the planner reduces the source of an OLAP `INSERT INTO ... SELECT` to an
empty set, the load job is registered but never finished. Two
examples are partition pruning on a LIST-partitioned table that has no
partition for the filtered value, and `WHERE 1 = 0`. The job stays
PENDING in the master FE until it restarts, so a workload that often hits
this leaks heap without bound. The fix records the registered job
as FINISHED with 0 rows on that early-return path. After upgrading, these
inserts show up in `SHOW LOAD` as FINISHED with 0 rows instead of
PENDING.
### Details
- **Frequency:** Every OLAP `INSERT ... SELECT` whose source is planned as a
single `VEMPTYSET`, except `INSERT ... VALUES`, transaction
mode and group commit, which do not register a job. The reporter saw about
140k stuck jobs per day on a production master. That filled a
16 GiB heap and caused a stop-the-world full GC about every 5 minutes.
Once more than `label_num_threshold` jobs were stuck, every run of
the label cleaner also deleted all finished load jobs, so `SHOW LOAD` kept
less than an hour of history.
- **Root cause:** `InsertIntoTableCommand.runInternal` returns early when
`insertExecutor.requiresTransaction()` is false, which is the
case for an empty source without Table Stream offset updates. On that path
neither `afterExec` nor `onFail` runs, so the job registered
in the constructor is never finished. `LoadJob.isExpired()` is false for
uncompleted jobs and `removeOverLimitLoadJob` removes only
completed ones, so the job is never cleaned up. Registering a job doesn't
write an edit-log entry, so it exists only in the master's memory
and disappears on restart.
- **Why existing tests missed it:** No test inserted from a source the
planner reduces to an empty set and then checked the job's state.
- **Fix:** Add `AbstractInsertExecutor.onSkipExecution(StmtExecutor)`, a
hook that does nothing by default. `runInternal` calls it on the
no-transaction early return. `OlapInsertExecutor` overrides it to call the
same `recordLoadJob` that `afterExec` uses, but only when its
constructor registered the job. No transaction is faked: the job is
recorded with transaction id `-1`, and the client response is
unchanged.
- **Other paths and branches:** `OlapTxnInsertExecutor`,
`OlapGroupCommitInsertExecutor`, `RemoteOlapInsertExecutor` and the
external-table executors don't register a job, so for them the hook does
nothing. The empty-source paths in `RewriteTableCommand` and
`RowLevelDmlCommand` don't use a registering executor. The same leak
exists on branch-4.1 (4.1.3 and 4.1.5), where the early return
checks `shouldExecuteEmptyInsert()`, so this fix should be backported.
- **Behavior change:** These inserts now appear in `SHOW LOAD` as FINISHED
with 0 rows. Previously they showed
as PENDING until the FE restarted. Each one now also writes one
`logCreateLoadJob` edit-log entry, the same as an executed insert.
### Testing
- `OlapInsertExecutorTest`: `testSkipExecutionFinishesRegisteredLoadJob`
checks that a registered empty insert calls
`recordFinishedLoadJob` with no fail message, which makes the job
FINISHED. `testSkipExecutionRecordsNothingForUnregisteredLoadJob` checks
that an executor which didn't register a job doesn't touch `LoadManager`.
- `load_p0/insert/test_insert_statistic`: runs the issue's repro. The source
is an auto LIST-partitioned table filtered on a value with no
partition, and the test asserts its plan contains `VEMPTYSET`. It also
runs a `WHERE 1 = 0` source. For each labelled insert it asserts
`SHOW LOAD` reports FINISHED, then asserts no job in the database is
PENDING. Before this fix both labels report PENDING.
None of these were run locally. They are left to CI.
### Notes for Reviewers
I left the issue's suggested safeguard, expiring old PENDING insert jobs in
the label cleaner, out of this PR. A live insert is also
PENDING until it completes, so a job's age alone doesn't show that it's
orphaned.
### Release note
Fix `INSERT INTO ... SELECT` statements whose source is planned empty, such
as one filtered on a partition that doesn't exist, leaving
PENDING load jobs in the master FE that leaked memory until restart.
### Check List (For Author)
- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Partially verified: the new unit test and
regression case have not been run locally; they rely on CI.
### AI Disclosure
- Tool: Claude Code (Claude Opus 5.5)
- Generated: Code, tests, and this description
- Human review:
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]