[
https://issues.apache.org/jira/browse/NIFI-16610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvind Kandpal updated NIFI-16610:
----------------------------------
Description:
StreamDemarcator does not find a multi-byte delimiter when the bytes just
before it contain a partial match of the delimiter.
In nextToken(), when a byte does not match delimiterBytes[j], the match index j
is reset to 0. It should fall back to the longest part of the delimiter that is
still matched (as in the KMP algorithm). Because of this, a delimiter that
starts inside a partial match is skipped, and the two tokens are returned as
one.
StreamDemarcator is used by PublishKafka (DelimitedStreamKafkaRecordConverter)
when the Message Demarcator property is set. With a multi-byte demarcator,
messages can be silently combined into one Kafka message.
Steps to reproduce (StreamDemarcator with maxDataSize 1000):
{noformat}
data "msg1\r\r\nmsg2" delimiter "\r\n" actual [msg1\r\r\nmsg2] expected
[msg1\r, msg2]
data "x---END--y" delimiter "--END--" actual [x---END--y] expected
[x-, y]
data "helloaaab" delimiter "aab" actual [helloaaab] expected
[helloa]
{noformat}
Single-byte delimiters such as "\n" are not affected.
Code: nifi-utils, StreamDemarcator.java, nextToken()
The existing tests in StreamDemarcatorTest only use delimiters where this case
does not happen (for example "daa").
was:
StreamDemarcator does not find a multi-byte delimiter when the bytes just
before it contain a partial match of the delimiter.
In nextToken(), when a byte does not match delimiterBytes[j], the match index j
is reset to 0, but the current byte is not compared again with
delimiterBytes[0]. If that byte is the start of the real delimiter, the
delimiter is skipped and the two tokens are returned as one.
StreamDemarcator is used by PublishKafka (DelimitedStreamKafkaRecordConverter)
when the Message Demarcator property is set. With a multi-byte demarcator,
messages can be silently combined into one Kafka message.
Steps to reproduce (StreamDemarcator with maxDataSize 1000):
{noformat}
data "msg1\r\r\nmsg2" delimiter "\r\n" actual [msg1\r\r\nmsg2] expected
[msg1\r, msg2]
data "x---END--y" delimiter "--END--" actual [x---END--y] expected
[x-, y]
data "helloaaab" delimiter "aab" actual [helloaaab] expected
[helloa]
{noformat}
Single-byte delimiters such as "\n" are not affected.
Code: nifi-utils, StreamDemarcator.java, nextToken()
The existing tests in StreamDemarcatorTest only use delimiters where this case
does not happen (for example "daa").
> StreamDemarcator misses a multi-byte delimiter that follows a partial match
> ---------------------------------------------------------------------------
>
> Key: NIFI-16610
> URL: https://issues.apache.org/jira/browse/NIFI-16610
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Reporter: Arvind Kandpal
> Assignee: Anshu Singh
> Priority: Major
>
> StreamDemarcator does not find a multi-byte delimiter when the bytes just
> before it contain a partial match of the delimiter.
> In nextToken(), when a byte does not match delimiterBytes[j], the match index
> j is reset to 0. It should fall back to the longest part of the delimiter
> that is still matched (as in the KMP algorithm). Because of this, a delimiter
> that starts inside a partial match is skipped, and the two tokens are
> returned as one.
> StreamDemarcator is used by PublishKafka
> (DelimitedStreamKafkaRecordConverter) when the Message Demarcator property is
> set. With a multi-byte demarcator, messages can be silently combined into one
> Kafka message.
> Steps to reproduce (StreamDemarcator with maxDataSize 1000):
> {noformat}
> data "msg1\r\r\nmsg2" delimiter "\r\n" actual [msg1\r\r\nmsg2] expected
> [msg1\r, msg2]
> data "x---END--y" delimiter "--END--" actual [x---END--y] expected
> [x-, y]
> data "helloaaab" delimiter "aab" actual [helloaaab] expected
> [helloa]
> {noformat}
> Single-byte delimiters such as "\n" are not affected.
> Code: nifi-utils, StreamDemarcator.java, nextToken()
> The existing tests in StreamDemarcatorTest only use delimiters where this
> case does not happen (for example "daa").
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)