[
https://issues.apache.org/jira/browse/NIFI-16599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18146941#comment-18146941
]
ASF subversion and git services commented on NIFI-16599:
--------------------------------------------------------
Commit f7e73872481f474c217d27fc2a86447d6d156ce7 in nifi's branch
refs/heads/main from Mark Payne
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f7e73872481 ]
NIFI-16599 Preserve FlowFile order across restart for queues that are swapping
(#11936)
- Restore FlowFiles in priority order on recovery when there are no swap files,
sharing the logic used when swap files exist
- Keep the place of a swap file rewritten by a selective drop by reusing the
timestamp and counter of the replaced swap file
- Add a counter to swap file names so that swap files written in the same
millisecond are recovered in the order written
Co-authored-by: Cursor <[email protected]>
Signed-off-by: David Handermann <[email protected]>
> Preserve FlowFile order across restart for queues that are swapping
> -------------------------------------------------------------------
>
> Key: NIFI-16599
> URL: https://issues.apache.org/jira/browse/NIFI-16599
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When a connection holds more FlowFiles than its swap threshold, FlowFiles can
> come out of the queue in the wrong order after NiFi restarts. There are three
> causes.
> If more FlowFiles are restored from the FlowFile Repository than the active
> queue can hold, but there are no swap files, whichever FlowFiles are restored
> first fill the active queue. The restore order is arbitrary, so newer
> FlowFiles can be processed ahead of older ones. NIFI-16580 fixed this only
> for the case where swap files exist.
> A selective drop rewrites a swap file under a new name that uses the current
> time. On restart, swap files are sorted by the time in their names, so the
> rewritten file is placed behind swap files that hold newer FlowFiles.
> Swap files written within the same millisecond are ordered by a random UUID
> in their names, so their order after a restart is random.
> SwappablePriorityQueue should restore FlowFiles in priority order when there
> are no swap files. FileSystemSwapManager should name swap files so that a
> rewritten file keeps its place and files written in the same millisecond sort
> in the order they were written.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)