[
https://issues.apache.org/jira/browse/NIFI-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinyu Wang reassigned NIFI-16440:
---------------------------------
Assignee: Xinyu Wang
> Provided Parameters stay empty after restart once a failed Parameter Provider
> fetch has dropped their value from flow.json
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-16440
> URL: https://issues.apache.org/jira/browse/NIFI-16440
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 2.0.0, 2.12.0
> Reporter: Xinyu Wang
> Assignee: Xinyu Wang
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h3. Problem
> NIFI-13560 (first released in 2.0.0) stopped persisting values fetched from a
> Parameter Provider. The flow configuration only keeps a reference
> ({{provided:parameter}}) and NiFi fetches the values again when it starts. If
> the Parameter Provider cannot be reached during that startup fetch, the
> reference is lost and the Parameters cannot recover on later restarts, even
> after the provider is healthy again.
> h3. Steps to reproduce
> # Configure a Parameter Provider and apply its Parameter Group so that a
> Parameter Context is bound to it. All Parameters have values.
> # Make the provider unreachable and restart NiFi. The startup fetch fails,
> {{Fetched Parameters [0]}} is logged, and every provided Parameter is empty.
> This part is expected.
> # Make the provider reachable again.
> # Restart NiFi, as many times as you like.
> h3. Actual behavior
> On every restart in step 4 the provider returns all Parameters ({{Fetched
> Parameters [4]}}), but every Parameter stays empty. The only way back is
> *Fetch Parameters* followed by *Apply Changes*.
> In flow.json the provided Parameters were serialized as
> {code}{"name":"key1","sensitive":false,"provided":true,"value":"provided:parameter"}{code}
> before step 2, and after step 2 they have *no {{value}} field at all*:
> {code}{"name":"key1","sensitive":false,"provided":true}{code}
> h3. Expected behavior
> Once the provider is reachable again, the next restart fetches the values and
> the Parameters are populated. A failed fetch should not remove the reference
> to the provider from the flow configuration.
> h3. Root cause
> Two null checks run before the provided check:
> # {{StandardParameterValueMapper.getMapped}} evaluates {{value == null}}
> before {{parameter.isProvided()}} (added in NIFI-13577 to avoid an NPE in the
> sensitive value encryptor). After a failed fetch the Parameter value is null,
> so the mapper returns null and the {{value}} field is omitted when flow.json
> is written.
> # {{VersionedFlowSynchronizer}} (the method that builds the Parameter map
> from the serialized Parameter Context) evaluates {{rawValue == null}} before
> the provided check. A provided Parameter without a stored value is set to
> null without looking at the values fetched from the provider.
> The failed fetch itself is only logged ({{Fetching Parameters for Group [...]
> from Provider [...] failed}}), then each provided Parameter logs {{Provided
> Parameter [x] not found}} and gets a null value. This happens for any
> provider whose {{fetchParameters}} throws, for example when the external
> service is unavailable during startup.
> h3. Proposed fix
> Evaluate {{isProvided()}} first in both places:
> * {{StandardParameterValueMapper.getMapped}}: return {{provided:parameter}}
> for provided Parameters even when the value is null.
> * {{VersionedFlowSynchronizer}}: for provided Parameters always use the value
> from the provider, regardless of the stored value.
> With this change the flow configuration keeps {{provided:parameter}} through
> a failed fetch, and the Parameters recover on the next restart after the
> provider is back (verified on 2.12.0, see below). Non-provided Parameters are
> not affected.
> h3. Verification
> Tested on stock {{apache/nifi}} images, standalone, with a Parameter Provider
> that throws {{IOException}} when its backend is down. The "without fix"
> column was reproduced on both 2.0.0 and 2.12.0 with identical results; the
> "with fix" column was tested on 2.12.0:
> || Scenario || Without fix || With fix ||
> | provider down, restart | Parameters empty, {{value}} dropped from flow.json
> | Parameters empty, {{provided:parameter}} kept in flow.json |
> | provider up again, restart (repeat 3 times) | Parameters stay empty every
> time | Parameters populated after one restart |
> | flow.json already missing {{value}} | stays empty | populated after one
> restart |
> h3. Notes
> * Existing unit test {{TestStandardParameterValueMapper}} covers a null value
> for a non-provided Parameter only; a provided Parameter with a null value is
> not covered.
> * Related: NIFI-13560 (design), NIFI-13577 (null handling in the mapper),
> NIFI-13688 (startup ordering of provider validation).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
