[
https://issues.apache.org/jira/browse/NIFI-16521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18133500#comment-18133500
]
Paweł Rzepiński commented on NIFI-16521:
----------------------------------------
Adding some research from my side:
||Processing Strategy||Output Strategy||Can preserve tombstones in Flow Files||
|FLOW_FILE|-|Yes, it is aready preserved it through KAFKA_TOMBSTONE attribute
on Flow Files|
|DEMARCATOR|-|Not easily, it would need to emit Flow File with single record
(instead of multiple) and set KAFKA_TOMBSTONE attribute on that Flow File|
|RECORD|USE_VALUE|Not easily, it would need to emit Flow File with single
record (instead of multiple) and set KAFKA_TOMBSTONE attribute on that Flow
File|
|RECORD|USE_WRAPPER|Possibly yes, through KAFKA_TOMBSTONE metadata entry|
|RECORD|INJECT_METADATA|Possibly yes, through KAFKA_TOMBSTONE metadata entry
but it would need static "tombstone-only" Avro schema|
|RECORD|INJECT_OFFSET|Possibly yes, through kafkaTombstone field but it would
need static "tombstone-only" Avro schema|
1. Since preserving tombstone would mean passing null values downstream it may
require adjustment in other processors (especially if you use Avro / Protobuf
readers).
2. Emitting single Flow File to indicate tombstone may impact throughput.
Given above, Nifi should expose opt-in property that enables preserving
tombstone but the would be skipped by default.
> Support Kafka tombstones in DEMARCATOR and RECORD processing strategies
> -----------------------------------------------------------------------
>
> Key: NIFI-16521
> URL: https://issues.apache.org/jira/browse/NIFI-16521
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Paweł Rzepiński
> Priority: Minor
>
> Follow up to https://issues.apache.org/jira/browse/NIFI-12371.
> In some scenarios (compaction, transactions, etc), you may have Kafka
> tombstone messages (ie. the message has null for the content) and it may be
> important to have NiFi properly consume and publish those tombstone messages.
> As of now, NIFI properly preserves tombstones only in FLOW_FILE processing
> strategy but it does not in DEMARCATOR and RECORD processing strategies. This
> ticket is about fixing that.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
