[
https://issues.apache.org/jira/browse/NIFI-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847910#comment-15847910
]
ASF GitHub Bot commented on NIFI-3373:
--------------------------------------
GitHub user ijokarumawak opened a pull request:
https://github.com/apache/nifi/pull/1460
NIFI-3373: Add nifi.flow.configuration.archive.max.count
- Add 'nifi.flow.configuration.archive.max.count' in nifi.properties
- Change default archive limit so that it uses archive max time(30 days)
and storage (500MB) if no limitation is specified
- Simplified logic to delete old archives
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [x] Is your initial contribution a single, squashed commit?
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ijokarumawak/nifi nifi-3373
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/1460.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1460
----
commit 05dce3f1d3081b737a2a3112795156d5cbd640a2
Author: Koji Kawamura <[email protected]>
Date: 2017-01-24T00:23:01Z
NIFI-3373: Add nifi.flow.configuration.archive.max.count
- Add 'nifi.flow.configuration.archive.max.count' in nifi.properties
- Change default archive limit so that it uses archive max time(30 days)
and storage (500MB) if no limitation is specified
- Simplified logic to delete old archives
----
> Add nifi.flow.configuration.archive.max.count property
> ------------------------------------------------------
>
> Key: NIFI-3373
> URL: https://issues.apache.org/jira/browse/NIFI-3373
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Reporter: Koji Kawamura
>
> Currently we can limit the number of flow.xml.gz archive files by:
> * total archive size (nifi.flow.configuration.archive.max.storage)
> * archive file age (nifi.flow.configuration.archive.max.time)
> In addition to these conditions to manage old archives, there's a demand that
> simply limiting number of archive files regardless time or size constraint.
> https://lists.apache.org/thread.html/4d2d9cec46ee896318a5492bf020f60c28396e2850c077dad40d45d2@%3Cusers.nifi.apache.org%3E
> We can provide that by adding new property
> 'nifi.flow.configuration.archive.max.count', so that If specified, only N
> latest config files can be archived.
> Make those properties optional, and process in following order:
> - If max.count is specified, any archive other than the latest (N-1) is
> removed
> - If max.time is specified, any archive that is older than max.time is removed
> - If max.storage is specified, old archives are deleted while total size is
> greater than the configuration
> - Create new archive, keep the latest archive regardless of above limitations
> To illustrate how flow.xml archiving works, here are simulations with the
> updated logic, where the size of flow.xml keeps increasing:
> h3. CASE-1
> archive.max.storage=10MB
> archive.max.count = 5
> ||Time || flow.xml || archives || archive total ||
> |t1 | f1 5MB | f1 | 5MB|
> |t2 | f2 5MB | f1, f2 | 10MB|
> |t3 | f3 5MB | f2, f3 | 10MB|
> |t4 | f4 10MB | f4 | 10MB|
> |t5 | f5 15MB | f5 | 15MB|
> |t6 | f6 20MB | f6 | 20MB|
> |t7 | f7 25MB | t7 | 25MB|
> * t3: The oldest f1 is removed, because f1 + f2 + f3 > 10MB.
> * t5: Even if flow.xml size exceeds max.storage, the latest archive is
> created. f4 is removed because f4 + f5 > 10MB. WAR message is logged because
> f5 is greater than 10MB.
> In this case, NiFi will keep logging WAR message
> indicating archive storage size is exceeding limit, from t5.
> After t5, NiFi will only keep the latest flow.xml.
> h3. CASE-2
> If at least 5 archives need to be kept no matter what, then set
> blank max.storage and max.time.
> archive.max.storage=
> archive.max.time=
> archive.max.count = 5 // Only limit archives by count
> |Time || flow.xml || archives || archive total ||
> |t1 | f1 5MB | f1 | 5MB|
> |t2 | f2 5MB | f1, f2 | 10MB|
> |t3 | f3 5MB | f1, f2, f3 | 15MB|
> |t4 | f4 10MB | f1, f2, f3, f4 | 25MB|
> |t5 | f5 15MB | f1, f2, f3, f4, f5 | 40MB|
> |t6 | f6 20MB | f2, f3, f4, f5, f6 | 55MB|
> |t7 | f7 25MB | f3, f4, f5, f6, (f7) | 50MB, (75MB)|
> |t8 | f8 30MB | f3, f4, f5, f6 | 50MB|
> * From t6, oldest archive is removed to keep number of archives <= 5
> * At t7, if the disk has only 60MB space, f7 won't be archived. And
> after this point, archive mechanism stop working (Trying to create new
> archive, but keep getting exception: no space left on device).
> In either case above, once flow.xml has grown to that size, some human
> intervention would be needed
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
