[
https://issues.apache.org/jira/browse/NIFI-16416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael W Moser updated NIFI-16416:
-----------------------------------
Status: Patch Available (was: In Progress)
> Optimize WriteAheadProvenanceRepository storage maintenance and bug fixes
> -------------------------------------------------------------------------
>
> Key: NIFI-16416
> URL: https://issues.apache.org/jira/browse/NIFI-16416
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Reporter: Michael W Moser
> Assignee: Michael W Moser
> Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h3. Background
> WriteAheadProvenanceRepository has a background maintenance task that checks
> if the max.storage.time and max.storage.size is exceeded and cleanup needs to
> happen. The maintenance task performs many provenance repo directory
> listings, including a new listing after each file is deleted during
> max.storage.size checks. This process repeats until enough files are deleted
> to bring the provenance repository within storage limits. (Task 1) This
> could be optimized.
> h3. Bug Reports
> Users have also reported various other issues with provenance repo
> maintenance, including:
> * (Task 2) an orphaned events file with a large event ID "3437277653.prov"
> could never get cleaned up because files with lower event IDs "572531.prov"
> are cleaned up first
> * (Task 3) junk files in the prov_repo directory count toward the max
> storage size limit, which can cause the maintenance task to get stuck in an
> infinite loop if 0 files can be deleted
> * (Task 4) .toc files can be removed if the corresponding .prov/.prov.gz
> file could not be removed
> h3. Goals
> * (Task 1) We can improve the provenance repo maintenance by performing only
> one directory listing, then reusing that list throughout the maintenance task.
> * (Task 2) Sort event files by lastModified instead of event ID in filename
> * (Task 3) Avoid infinite loop when prov_repo size exceeds max.storage.size,
> but event files cannot be deleted
> * (Task 4) Properly handle removing .toc files only if its .prov file is
> removed
> h3. Stretch Goal
> * Remove events files when disk partition usage exceeds 98% (NiFi does this
> in FileSystemRepository when content_repo archiving is on, removing archived
> data). This could help avoid filling the disk partition where prov_repo
> stores data.
> h3. Other Observations
> * When using multiple provenance repo partitions, the cleanup task deletes
> one file from each prov_repo partition per pass. This can delete files that
> are not the oldest events overall. This is a minor issue, though, because
> deleted events are still very close in time to older events in other
> partitions.
> This was discussed in Slack in May 2026 and August 2026. Feedback on
> proposed goals appreciated.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)