markap14 opened a new pull request, #11936:
URL: https://github.com/apache/nifi/pull/11936

   # Summary
   
   [NIFI-16599](https://issues.apache.org/jira/browse/NIFI-16599)
   
   When a connection holds more FlowFiles than its swap threshold, FlowFiles 
could come out of the queue in the wrong order after NiFi restarts. This change 
fixes three causes.
   
   Restart with no swap files. On restart, the FlowFiles that were held in 
memory are loaded from the FlowFile Repository in no particular order. If there 
are more of them than the active queue holds, whichever ones are loaded first 
fill the active queue, and the rest go to the swap queue. NIFI-16580 fixed this 
when swap files exist, by keeping in the active queue only the FlowFiles that 
come before the oldest swap file. This change replaces that method with 
`SwappablePriorityQueue.restoreFlowFilesInPriorityOrder`, which handles both 
cases. It sorts the restored FlowFiles using the queue's prioritizers. With 
swap files, it keeps the same rule as NIFI-16580. Without swap files, it keeps 
the same number of FlowFiles in the active queue, chosen from the highest 
priority ones.
   
   Selective drop. A selective drop rewrites a swap file with the remaining 
FlowFiles under a new name. Swap files are recovered on restart in the order of 
the timestamp in their names, so the rewritten file moved behind swap files 
holding newer FlowFiles. `FlowFileSwapManager` now has a default method 
`swapOut(flowFiles, queue, partitionName, replacedSwapLocation)` that falls 
back to the existing `swapOut`. `FileSystemSwapManager` overrides it so that 
the new swap file takes the timestamp and counter of the file it replaces, and 
`SwappablePriorityQueue` uses it when rewriting a swap file for a selective 
drop.
   
   Swap files written in the same millisecond. Swap files with the same 
timestamp were ordered by the random UUID in their names. Swap file names now 
include a counter, `<timestamp>-<queue id>-<counter>-<random uuid>`, and swap 
files are recovered in order of timestamp, then counter. Names written by 
earlier versions have no counter and still sort by timestamp. Earlier versions 
can still read the new names, because the timestamp and queue identifier are in 
the same positions.
   
   ### Tests
   
   - `TestSwappablePriorityQueue.testRecoverSwappedFlowFilesPreservesOrder` now 
also restores more FlowFiles than the active queue holds with no swap file, and 
verifies that every FlowFile is polled in order.
   - `TestSwappablePriorityQueue.testSelectiveDropPreservesSwapFileOrdering` 
now uses a real `FileSystemSwapManager` and verifies that, after a selective 
drop, a new swap manager recovers the rewritten swap file ahead of the swap 
file that holds newer FlowFiles.
   - `TestFileSystemSwapManager.testRecoverSwapLocationsInOrderWritten` writes 
100 swap files and verifies that a new swap manager recovers them in the order 
they were written.
   
   Each test fails without its change and passes with it.
   
   # Tracking
   
   Please complete the following tracking steps prior to pull request creation.
   
   ### Issue Tracking
   
   - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue 
created
   
   ### Pull Request Tracking
   
   - [x] Pull Request title starts with Apache NiFi Jira issue number, such as 
`NIFI-00000`
   - [x] Pull Request commit message starts with Apache NiFi Jira issue number, 
as such `NIFI-00000`
   - [x] Pull request contains [commits 
signed](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits)
 with a registered key indicating `Verified` status
   
   ### Pull Request Formatting
   
   - [x] Pull Request based on current revision of the `main` branch
   - [x] Pull Request refers to a feature branch with one commit containing 
changes
   
   # Verification
   
   Please indicate the verification steps performed prior to pull request 
creation.
   
   ### Build
   
   - [ ] Build completed using `./mvnw clean install -P contrib-check`
     - [ ] JDK 21
     - [ ] JDK 25
   
   ### Licensing
   
   - [ ] New dependencies are compatible with the [Apache License 
2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License 
Policy](https://www.apache.org/legal/resolved.html)
   - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` 
files
   
   ### Documentation
   
   - [ ] Documentation formatting appears as expected in rendered files
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to