[
https://issues.apache.org/jira/browse/NIFI-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Kawamura updated NIFI-4844:
--------------------------------
Status: Patch Available (was: In Progress)
> AvroRecordSetWriter should be able to convert a double having less scale than
> intended target Avro schema instead of throwing an AvroTypeException
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-4844
> URL: https://issues.apache.org/jira/browse/NIFI-4844
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 1.3.0
> Reporter: Koji Kawamura
> Assignee: Koji Kawamura
> Priority: Major
>
> Current AvroTypeUtil conversion logic can throw AvroTypeException when it
> maps double values into Avro decimal logical type fields if the double value
> has less scale than the one defined at the target Avro decimal field schema.
> For example, with following schema:
> {code}
> {
> "type": "record",
> "name": "logicalDecimalTest",
> "fields": [
> {"name": "id", "type": "int"},
> {"name": "name", "type": "string"},
> {
> "name": "price",
> "type": {
> "type": "bytes",
> "logicalType": "decimal",
> "precision": 18,
> "scale": 8
> }}]}
> {code}
> And following CSV records:
> {code}
> id|name|price
> 1|one|1.23
> 2|two|2.34
> {code}
> Would produce this Exception:
> {code}
> 2018-02-06 09:57:27,461 ERROR [Timer-Driven Process Thread-7]
> o.a.n.processors.standard.ConvertRecord
> ConvertRecord[id=6897bc30-0161-1000-a8e7-9ce0ce8eb9ae] Failed to process
> StandardFlowFileRecord[uuid=a97366a0-79bb-42ff-9023-c5d62ecfdbc5,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1517878123416-2, container=default,
> section=2], offset=5, length=48],offset=0,name=220105646548465,size=48]; will
> route to failure: org.apache.avro.AvroTypeException: Cannot encode decimal
> with scale 17 as scale 8
> org.apache.avro.AvroTypeException: Cannot encode decimal with scale 17 as
> scale 8
> at
> org.apache.avro.Conversions$DecimalConversion.toBytes(Conversions.java:86)
> at
> org.apache.nifi.avro.AvroTypeUtil.convertToAvroObject(AvroTypeUtil.java:546)
> at
> org.apache.nifi.avro.AvroTypeUtil.createAvroRecord(AvroTypeUtil.java:457)
> at
> org.apache.nifi.avro.WriteAvroResultWithExternalSchema.writeRecord(WriteAvroResultWithExternalSchema.java:76)
> at
> org.apache.nifi.serialization.AbstractRecordSetWriter.write(AbstractRecordSetWriter.java:59)
> at
> org.apache.nifi.processors.standard.AbstractRecordProcessor$1.process(AbstractRecordProcessor.java:122)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2827)
> at
> org.apache.nifi.processors.standard.AbstractRecordProcessor.onTrigger(AbstractRecordProcessor.java:109)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> The same issue is reported in the Avro project,
> [AVRO-1864|https://issues.apache.org/jira/browse/AVRO-1864]. The recommended
> approach is to adjust the scale at NiFi side. Actually, for BigDecimal input
> values, NiFi already does this, but not with double values. AvroTypeUtil
> should do the same scale adjustment for double values, too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
