[
https://issues.apache.org/jira/browse/NIFI-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353477#comment-16353477
]
ASF GitHub Bot commented on NIFI-4844:
--------------------------------------
GitHub user ijokarumawak opened a pull request:
https://github.com/apache/nifi/pull/2450
NIFI-4844: Adjust BigDecimal scale to the target Avro schema
- Applied the same scale adjustment not only to BigDecimal inputs, but
also to Double values.
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [x] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ijokarumawak/nifi nifi-4844
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/2450.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2450
----
commit 342fcfa2b667b73890e2ed24d74d2ae84fb9acc8
Author: Koji Kawamura <ijokarumawak@...>
Date: 2018-02-06T04:52:48Z
NIFI-4844: Adjust BigDecimal scale to the target Avro schema
- Applied the same scale adjustment not only to BigDecimal inputs, but
also to Double values.
----
> AvroRecordSetWriter should be able to convert a double having less scale than
> intended target Avro schema instead of throwing an AvroTypeException
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-4844
> URL: https://issues.apache.org/jira/browse/NIFI-4844
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 1.3.0
> Reporter: Koji Kawamura
> Assignee: Koji Kawamura
> Priority: Major
>
> Current AvroTypeUtil conversion logic can throw AvroTypeException when it
> maps double values into Avro decimal logical type fields if the double value
> has less scale than the one defined at the target Avro decimal field schema.
> For example, with following schema:
> {code}
> {
> "type": "record",
> "name": "logicalDecimalTest",
> "fields": [
> {"name": "id", "type": "int"},
> {"name": "name", "type": "string"},
> {
> "name": "price",
> "type": {
> "type": "bytes",
> "logicalType": "decimal",
> "precision": 18,
> "scale": 8
> }}]}
> {code}
> And following CSV records:
> {code}
> id|name|price
> 1|one|1.23
> 2|two|2.34
> {code}
> Would produce this Exception:
> {code}
> 2018-02-06 09:57:27,461 ERROR [Timer-Driven Process Thread-7]
> o.a.n.processors.standard.ConvertRecord
> ConvertRecord[id=6897bc30-0161-1000-a8e7-9ce0ce8eb9ae] Failed to process
> StandardFlowFileRecord[uuid=a97366a0-79bb-42ff-9023-c5d62ecfdbc5,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1517878123416-2, container=default,
> section=2], offset=5, length=48],offset=0,name=220105646548465,size=48]; will
> route to failure: org.apache.avro.AvroTypeException: Cannot encode decimal
> with scale 17 as scale 8
> org.apache.avro.AvroTypeException: Cannot encode decimal with scale 17 as
> scale 8
> at
> org.apache.avro.Conversions$DecimalConversion.toBytes(Conversions.java:86)
> at
> org.apache.nifi.avro.AvroTypeUtil.convertToAvroObject(AvroTypeUtil.java:546)
> at
> org.apache.nifi.avro.AvroTypeUtil.createAvroRecord(AvroTypeUtil.java:457)
> at
> org.apache.nifi.avro.WriteAvroResultWithExternalSchema.writeRecord(WriteAvroResultWithExternalSchema.java:76)
> at
> org.apache.nifi.serialization.AbstractRecordSetWriter.write(AbstractRecordSetWriter.java:59)
> at
> org.apache.nifi.processors.standard.AbstractRecordProcessor$1.process(AbstractRecordProcessor.java:122)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2827)
> at
> org.apache.nifi.processors.standard.AbstractRecordProcessor.onTrigger(AbstractRecordProcessor.java:109)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> The same issue is reported in the Avro project,
> [AVRO-1864|https://issues.apache.org/jira/browse/AVRO-1864]. The recommended
> approach is to adjust the scale at NiFi side. Actually, for BigDecimal input
> values, NiFi already does this, but not with double values. AvroTypeUtil
> should do the same scale adjustment for double values, too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
