Vincenzo Lombardo created NIFI-16602:
----------------------------------------

             Summary: NifiDiscriminator throws NullPointerException for every 
log event without key/value pairs
                 Key: NIFI-16602
                 URL: https://issues.apache.org/jira/browse/NIFI-16602
             Project: Apache NiFi
          Issue Type: Bug
            Reporter: Vincenzo Lombardo


h3. Description

{{org.apache.nifi.logging.NifiDiscriminator}} (module 
{{nifi-per-process-group-logging}}) iterates over 
{{ILoggingEvent.getKeyValuePairs()}} without checking for {{null}}. Logback 
returns {{null}} for events that carry no key/value pairs, which covers every 
framework log message and every component log message from Process Groups 
without a Log File Suffix. So for the vast majority of events the discriminator 
throws a {{NullPointerException}}:

{noformat}
ERROR in ch.qos.logback.classic.sift.SiftingAppender[DEDICATED_LOGGING] - 
Appender [DEDICATED_LOGGING] failed to append.
java.lang.NullPointerException: Cannot invoke "java.util.List.iterator()" 
because the return value of 
"ch.qos.logback.classic.spi.ILoggingEvent.getKeyValuePairs()" is null
    at 
org.apache.nifi.logging.NifiDiscriminator.getDiscriminatingValue(NifiDiscriminator.java:30)
    at 
ch.qos.logback.core.sift.SiftingAppenderBase.append(SiftingAppenderBase.java:113)
    at ch.qos.logback.core.AppenderBase.doAppend(AppenderBase.java:85)
{noformat}

The exception is caught by {{AppenderBase.doAppend}}, so the event is not 
written to the dedicated file and the visible output is correct. However:
* an exception is created and thrown for every log event that goes through the 
{{DEDICATED_LOGGING}} appender, which is enabled by default in the NiFi 
{{logback.xml}};
* "failed to append" errors are recorded in the logback status;
* the correct output relies on the exception rather than on explicit logic.

h3. Why a null check in the discriminator is not a valid fix

If {{getDiscriminatingValue}} returns {{null}}, {{SiftingAppenderBase.append}} 
does not skip the event. It routes the event to an appender created for the 
undefined key. All events without a suffix then end up in an additional 
{{nifi-app-logFileSuffix_IS_UNDEFINED.log}} file, which duplicates the whole 
application log and doubles the write I/O.

h3. Proposed fix

Add a small {{Filter<ILoggingEvent>}} to {{nifi-per-process-group-logging}} 
that returns {{DENY}} for events without the {{logFileSuffix}} key/value pair 
and {{NEUTRAL}} otherwise. Then reference it from the {{DEDICATED_LOGGING}} 
appender:

{code:xml}
<appender name="DEDICATED_LOGGING" 
class="ch.qos.logback.classic.sift.SiftingAppender">
    <filter class="org.apache.nifi.logging.LogFileSuffixFilter"/>
    <discriminator class="org.apache.nifi.logging.NifiDiscriminator"/>
    ...
</appender>
{code}

Filters run in {{AppenderBase.doAppend}} before {{append}}, so events without a 
suffix never reach the discriminator.

Affected configuration files:
* 
{{nifi-framework-bundle/.../nifi-resources/src/main/resources/conf/logback.xml}}
* MiNiFi {{minifi-resources/src/main/resources/conf/logback.xml}} (the block is 
commented out, see NIFI-16572)

The discriminator should keep its current semantics. Existing installations 
often keep their {{logback.xml}} across upgrades and therefore won't have the 
new filter, and making the discriminator return {{null}} would produce the 
{{IS_UNDEFINED}} file described above.

h3. Measurements

Standalone test with logback 1.6.6 and slf4j 2.0.20: 1M INFO events to a file 
appender, plus a {{SiftingAppender}} using {{NifiDiscriminator}}. Average of 3 
runs:

||Configuration||Time||Files||
|No dedicated appender|~2.62 s|app.log|
|Dedicated appender, current discriminator (NPE path)|~2.66 s|app.log, 
app-<suffix>.log|
|Dedicated appender, null check in discriminator|~5.2 s|app.log, 
app-<suffix>.log, app-logFileSuffix_IS_UNDEFINED.log (all events)|
|Dedicated appender, proposed filter|~2.65 s|app.log, app-<suffix>.log|

On a modern JIT the performance difference between the current behavior and the 
filter is within noise, because hot implicit NPEs are optimized. The main 
benefit is correctness and clean logback status, not throughput.

h3. Related
NIFI-16572




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to