[
https://issues.apache.org/jira/browse/NIFI-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15869006#comment-15869006
]
ASF GitHub Bot commented on NIFI-3373:
--------------------------------------
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1460#discussion_r101427041
--- Diff:
nifi-commons/nifi-properties/src/main/java/org/apache/nifi/util/NiFiProperties.java
---
@@ -972,11 +973,15 @@ public Integer getFlowElectionMaxCandidates() {
}
public String getFlowConfigurationArchiveMaxTime() {
- return getProperty(FLOW_CONFIGURATION_ARCHIVE_MAX_TIME,
DEFAULT_FLOW_CONFIGURATION_ARCHIVE_MAX_TIME);
+ return getProperty(FLOW_CONFIGURATION_ARCHIVE_MAX_TIME, null);
--- End diff --
@joewitt It used to have defaults here, for time 30 days, for storage
500MB. These defaults are still effective if none of those are specified in
nifi.properties. I just moved the code applying default values to here:
```java
// FlowConfigurationArchiveManager
public FlowConfigurationArchiveManager(final Path flowConfigFile, final
NiFiProperties properties) {
(...)
if (maxCount == null && StringUtils.isBlank(maxTime) &&
StringUtils.isBlank(maxStorage)) {
// None of limitation is specified, fall back to the default
configuration;
maxTime =
NiFiProperties.DEFAULT_FLOW_CONFIGURATION_ARCHIVE_MAX_TIME;
maxStorage =
NiFiProperties.DEFAULT_FLOW_CONFIGURATION_ARCHIVE_MAX_STORAGE;
logger.info("None of archive max limitation is specified, fall
back to the default configuration, maxTime={}, maxStorage={}",
maxTime, maxStorage);
}
```
https://github.com/apache/nifi/pull/1460/files#diff-00c47dfbe82807a330d4f2f9432bb54dR70
The purpose of this is to let user only specify what they would like to use
to limit the archiving. Some user want to limit by only archive file count and
don't want to limit by size ... etc.
To avoid keep archiving without any limit, the above code uses default time
and storage size when no limit is configured.
Does this sound reasonable?
> Add nifi.flow.configuration.archive.max.count property
> ------------------------------------------------------
>
> Key: NIFI-3373
> URL: https://issues.apache.org/jira/browse/NIFI-3373
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Reporter: Koji Kawamura
> Assignee: Koji Kawamura
>
> Currently we can limit the number of flow.xml.gz archive files by:
> * total archive size (nifi.flow.configuration.archive.max.storage)
> * archive file age (nifi.flow.configuration.archive.max.time)
> In addition to these conditions to manage old archives, there's a demand that
> simply limiting number of archive files regardless time or size constraint.
> https://lists.apache.org/thread.html/4d2d9cec46ee896318a5492bf020f60c28396e2850c077dad40d45d2@%3Cusers.nifi.apache.org%3E
> We can provide that by adding new property
> 'nifi.flow.configuration.archive.max.count', so that If specified, only N
> latest config files can be archived.
> Make those properties optional, and process in following order:
> - If max.count is specified, any archive other than the latest (N-1) is
> removed
> - If max.time is specified, any archive that is older than max.time is removed
> - If max.storage is specified, old archives are deleted while total size is
> greater than the configuration
> - Create new archive, keep the latest archive regardless of above limitations
> To illustrate how flow.xml archiving works, here are simulations with the
> updated logic, where the size of flow.xml keeps increasing:
> h3. CASE-1
> archive.max.storage=10MB
> archive.max.count = 5
> ||Time || flow.xml || archives || archive total ||
> |t1 | f1 5MB | f1 | 5MB|
> |t2 | f2 5MB | f1, f2 | 10MB|
> |t3 | f3 5MB | f2, f3 | 10MB|
> |t4 | f4 10MB | f4 | 10MB|
> |t5 | f5 15MB | f5 | 15MB|
> |t6 | f6 20MB | f6 | 20MB|
> |t7 | f7 25MB | t7 | 25MB|
> * t3: The oldest f1 is removed, because f1 + f2 + f3 > 10MB.
> * t5: Even if flow.xml size exceeds max.storage, the latest archive is
> created. f4 is removed because f4 + f5 > 10MB. WAR message is logged because
> f5 is greater than 10MB.
> In this case, NiFi will keep logging WAR message
> indicating archive storage size is exceeding limit, from t5.
> After t5, NiFi will only keep the latest flow.xml.
> h3. CASE-2
> If at least 5 archives need to be kept no matter what, then set
> blank max.storage and max.time.
> archive.max.storage=
> archive.max.time=
> archive.max.count = 5 // Only limit archives by count
> |Time || flow.xml || archives || archive total ||
> |t1 | f1 5MB | f1 | 5MB|
> |t2 | f2 5MB | f1, f2 | 10MB|
> |t3 | f3 5MB | f1, f2, f3 | 15MB|
> |t4 | f4 10MB | f1, f2, f3, f4 | 25MB|
> |t5 | f5 15MB | f1, f2, f3, f4, f5 | 40MB|
> |t6 | f6 20MB | f2, f3, f4, f5, f6 | 55MB|
> |t7 | f7 25MB | f3, f4, f5, f6, (f7) | 50MB, (75MB)|
> |t8 | f8 30MB | f3, f4, f5, f6 | 50MB|
> * From t6, oldest archive is removed to keep number of archives <= 5
> * At t7, if the disk has only 60MB space, f7 won't be archived. And
> after this point, archive mechanism stop working (Trying to create new
> archive, but keep getting exception: no space left on device).
> In either case above, once flow.xml has grown to that size, some human
> intervention would be needed
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
