[
https://issues.apache.org/jira/browse/NIFI-14696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dariusz Seweryn updated NIFI-14696:
-----------------------------------
Description:
{{KinesisRecordProcessorRecord#processRecord}} method may encounter an
{{IllegalTypeConversionException}} while trying write the record. In this case
the exception is caught in the
{{AbstractKinesisRecordProcessor#attemptProcessRecord}} function.
{{KinesisRecordProcessorRecord#processRecord}} apart from writing records is
responsible for:
# creating FlowFiles and associated objects
# closing the the FlowFile at the last record
# routing problematic records to the Failure Relationship
Current implementation has several bugs:
* in case the problematic record does not conform to writers schema, the
exception is not handled in the class — the record is not routed properly to
the Failure Relationship
* in case the problematic record is the last record in the batch, the FlowFile
is left unfinished, without arguments
The main problem is that the writers schema is determined based on the first
observed record. If the schema is inferred — it is possible a number field in
the first record may get inferred as an Integer and some other as a Long. In
such case, the writers schema should be widened.
Fixes needed:
# Handling of {{IllegalTypeConversionException}}
# Closing FlowFile when last record processing failed
# Implement writer schema widening on incompatible records — possibly output
multiple FlowFiles where each next has wider schema
was:
{{KinesisRecordProcessorRecord#processRecord}} method may encounter an
{{IllegalTypeConversionException}} while trying write the record. In this case
the exception is caught in the
{{AbstractKinesisRecordProcessor#attemptProcessRecord}} function.
{{KinesisRecordProcessorRecord#processRecord}} apart from writing records is
responsible for:
# creating FlowFiles and associated objects
# closing the the FlowFile at the last record
# routing problematic records to the Failure Relationship
Current implementation has several bugs:
* in case the problematic record does not conform to writers schema, the
exception is not handled in the class — the record is not routed properly to
the Failure Relationship
* in case the problematic record is the last record in the batch, the FlowFile
is left unfinished, without arguments
The main problem is that the writers schema is determined based on the first
observed record. If the schema is inferred — it is possible a number field in
the first record may get inferred as an Integer and some other as a Long. In
such case, the writers schema should be widened.
Fixes needed:
# Handling of {{IllegalTypeConversionException}}
# Closing FlowFile when last record processing failed
# Implement writer schema widening on incompatible records
> KinesisRecordProcessorRecord does not handle IllegalTypeConversionException
> ---------------------------------------------------------------------------
>
> Key: NIFI-14696
> URL: https://issues.apache.org/jira/browse/NIFI-14696
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 2.4.0
> Reporter: Dariusz Seweryn
> Assignee: Dariusz Seweryn
> Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {{KinesisRecordProcessorRecord#processRecord}} method may encounter an
> {{IllegalTypeConversionException}} while trying write the record. In this
> case the exception is caught in the
> {{AbstractKinesisRecordProcessor#attemptProcessRecord}} function.
> {{KinesisRecordProcessorRecord#processRecord}} apart from writing records is
> responsible for:
> # creating FlowFiles and associated objects
> # closing the the FlowFile at the last record
> # routing problematic records to the Failure Relationship
> Current implementation has several bugs:
> * in case the problematic record does not conform to writers schema, the
> exception is not handled in the class — the record is not routed properly to
> the Failure Relationship
> * in case the problematic record is the last record in the batch, the
> FlowFile is left unfinished, without arguments
> The main problem is that the writers schema is determined based on the first
> observed record. If the schema is inferred — it is possible a number field in
> the first record may get inferred as an Integer and some other as a Long. In
> such case, the writers schema should be widened.
> Fixes needed:
> # Handling of {{IllegalTypeConversionException}}
> # Closing FlowFile when last record processing failed
> # Implement writer schema widening on incompatible records — possibly output
> multiple FlowFiles where each next has wider schema
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
