[ 
https://issues.apache.org/jira/browse/NIFI-16599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18146941#comment-18146941
 ] 

ASF subversion and git services commented on NIFI-16599:
--------------------------------------------------------

Commit f7e73872481f474c217d27fc2a86447d6d156ce7 in nifi's branch 
refs/heads/main from Mark Payne
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f7e73872481 ]

NIFI-16599 Preserve FlowFile order across restart for queues that are swapping 
(#11936)

- Restore FlowFiles in priority order on recovery when there are no swap files, 
sharing the logic used when swap files exist
- Keep the place of a swap file rewritten by a selective drop by reusing the 
timestamp and counter of the replaced swap file
- Add a counter to swap file names so that swap files written in the same 
millisecond are recovered in the order written

Co-authored-by: Cursor <[email protected]>
Signed-off-by: David Handermann <[email protected]>

> Preserve FlowFile order across restart for queues that are swapping
> -------------------------------------------------------------------
>
>                 Key: NIFI-16599
>                 URL: https://issues.apache.org/jira/browse/NIFI-16599
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Core Framework
>            Reporter: Mark Payne
>            Assignee: Mark Payne
>            Priority: Major
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> When a connection holds more FlowFiles than its swap threshold, FlowFiles can 
> come out of the queue in the wrong order after NiFi restarts. There are three 
> causes.
> If more FlowFiles are restored from the FlowFile Repository than the active 
> queue can hold, but there are no swap files, whichever FlowFiles are restored 
> first fill the active queue. The restore order is arbitrary, so newer 
> FlowFiles can be processed ahead of older ones. NIFI-16580 fixed this only 
> for the case where swap files exist.
> A selective drop rewrites a swap file under a new name that uses the current 
> time. On restart, swap files are sorted by the time in their names, so the 
> rewritten file is placed behind swap files that hold newer FlowFiles.
> Swap files written within the same millisecond are ordered by a random UUID 
> in their names, so their order after a restart is random.
> SwappablePriorityQueue should restore FlowFiles in priority order when there 
> are no swap files. FileSystemSwapManager should name swap files so that a 
> rewritten file keeps its place and files written in the same millisecond sort 
> in the order they were written.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to