mrhhsg opened a new pull request, #67498:
URL: https://github.com/apache/doris/pull/67498
### What problem does this PR solve?
Issue Number: None
Problem Summary:
Spill creates one directory per query and per operator plus one file per
part, so under high concurrency a spill disk can run out of inodes long
before it runs out of bytes. The spill capacity checks and the periodic
`spill gc time` log only report bytes, so an inode leak or a growing
`spill_gc` backlog cannot be diagnosed from the BE log or metrics.
This PR adds inode monitoring to the spill file manager:
- `LocalFileSystem::get_inode_info()` returns the total and available
inode count of the file system holding a path via `statvfs`.
- `SpillDataDir::update_capacity()` refreshes the inode statistics and
exposes them as `spill_disk_inode_total` and
`spill_disk_inode_available` metrics. A throttled warning is logged
when the inode usage reaches `storage_flood_stage_usage_percent`.
- The spill gc summary log now prints, per spill store, the inode usage,
the number of query directories still pending under `spill_gc`, the
number of directories/files deleted in this round and the number of
failed deletions. Deletion failures are no longer silently discarded
but logged with a throttled warning.
### Release note
None
### Check List (For Author)
- Test:
- Unit Test: `LocalFileSystemTest.GetInodeInfo`,
`SpillFileTest.UpdateCapacityTracksInodeUsage`,
`SpillFileTest.GCCleansUpGcRootBacklog`
- Regression test: No
- Behavior changed: No
- Does this need documentation: No
### Check List (For Reviewer who merge this PR)
- Confirm the release note
- Confirm test cases
- Confirm document
- Add branch pick label
https://claude.ai/code/session_0149x6DMH4C9RUmHLjNKDFxS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
