[
https://issues.apache.org/jira/browse/NIFI-5065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431903#comment-16431903
]
ASF GitHub Bot commented on NIFI-5065:
--------------------------------------
GitHub user ijokarumawak opened a pull request:
https://github.com/apache/nifi/pull/2622
NIFI-5065: HTTP Site-to-Site fails sending data more than 2GB at once
Confirmed HTTP S2S can send files containing more than 2GB. I tested up to
8GB but can be more.
---
Int variables got overflown.
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [x] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ijokarumawak/nifi nifi-5065
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/2622.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2622
----
commit aada04962db9d2e2bb72d5c92a1a5c2306f62b72
Author: Koji Kawamura <ijokarumawak@...>
Date: 2018-04-10T08:15:30Z
NIFI-5065: HTTP Site-to-Site fails sending data more than 2GB at once
Int variables got overflown.
----
> HTTP Site-to-Site fails sending data more than 2GB at once
> ----------------------------------------------------------
>
> Key: NIFI-5065
> URL: https://issues.apache.org/jira/browse/NIFI-5065
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.0.0
> Reporter: Koji Kawamura
> Assignee: Koji Kawamura
> Priority: Major
>
> NiFi Site-to-Site (S2S) is designed to support sending large files. While RAW
> transport protocol works fine with large files (I tested up to 8GB, but it
> can be more), HTTP transport protocol starts failing if more than 2GB data is
> sent at a time, meaning by sending a 2GB FlowFile, or batching multiple
> FlowFiles to make more than 2GB into one transaction.
>
> When it fails, following log messages are written:
> {code}
> 2018-04-10 16:05:45,006 ERROR [I/O dispatcher 25]
> o.a.n.r.util.SiteToSiteRestApiClient Failed to send data to
> http://localhost:8080/nifi-api/data-transfer/input-ports/ad9a3887-0162-1000-e312-dee642179
> c9c/transactions/608f1ce4-56da-4899-9348-d2864e364d40/flow-files due to
> java.lang.RuntimeException: Sending data to
> http://localhost:8080/nifi-api/data-transfer/input-ports/ad9a3887-0162-1000-e312-dee
> 642179c9c/transactions/608f1ce4-56da-4899-9348-d2864e364d40/flow-files has
> reached to its end, but produced : read : wrote byte sizes (659704502 :
> 659704502 : 9249639094) were not equal. Something went wr
> ong.
> java.lang.RuntimeException: Sending data to
> http://localhost:8080/nifi-api/data-transfer/input-ports/ad9a3887-0162-1000-e312-dee642179c9c/transactions/608f1ce4-56da-4899-9348-d2864e364d40/flow-files
> h
> as reached to its end, but produced : read : wrote byte sizes (659704502 :
> 659704502 : 9249639094) were not equal. Something went wrong.
> at
> org.apache.nifi.remote.util.SiteToSiteRestApiClient$4.produceContent(SiteToSiteRestApiClient.java:848)
> at
> org.apache.http.impl.nio.client.MainClientExec.produceContent(MainClientExec.java:262)
> at
> org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.produceContent(DefaultClientExchangeHandlerImpl.java:140)
> at
> org.apache.http.nio.protocol.HttpAsyncRequestExecutor.outputReady(HttpAsyncRequestExecutor.java:241)
> at
> org.apache.http.impl.nio.DefaultNHttpClientConnection.produceOutput(DefaultNHttpClientConnection.java:290)
> at
> org.apache.http.impl.nio.client.InternalIODispatch.onOutputReady(InternalIODispatch.java:86)
> at
> org.apache.http.impl.nio.client.InternalIODispatch.onOutputReady(InternalIODispatch.java:39)
> at
> org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(AbstractIODispatch.java:145)
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:188)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:341)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
> at
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
> at java.lang.Thread.run(Thread.java:745)
> 2018-04-10 16:06:25,009 ERROR [Timer-Driven Process Thread-3]
> o.a.nifi.remote.StandardRemoteGroupPort
> RemoteGroupPort[name=input,targets=http://localhost:8080/nifi] failed to
> communicate with remote NiFi instance due to java.io.IOException: Failed to
> confirm transaction with Peer[url=http://localhost:8080/nifi-api] due to
> java.io.IOException: Awaiting transferDataLatch has been timeout.
> 2018-04-10 16:06:25,009 ERROR [Timer-Driven Process Thread-3]
> o.a.nifi.remote.StandardRemoteGroupPort
> java.io.IOException: Failed to confirm transaction with
> Peer[url=http://localhost:8080/nifi-api] due to java.io.IOException: Awaiting
> transferDataLatch has been timeout.
> at
> org.apache.nifi.remote.AbstractTransaction.confirm(AbstractTransaction.java:264)
> at
> org.apache.nifi.remote.StandardRemoteGroupPort.transferFlowFiles(StandardRemoteGroupPort.java:369)
> at
> org.apache.nifi.remote.StandardRemoteGroupPort.onTrigger(StandardRemoteGroupPort.java:285)
> at
> org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:250)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Awaiting transferDataLatch has been timeout.
> at
> org.apache.nifi.remote.util.SiteToSiteRestApiClient.finishTransferFlowFiles(SiteToSiteRestApiClient.java:938)
> at
> org.apache.nifi.remote.protocol.http.HttpClientTransaction.readTransactionResponse(HttpClientTransaction.java:93)
> at
> org.apache.nifi.remote.AbstractTransaction.confirm(AbstractTransaction.java:239)
> ... 12 common frames omitted
> {code}
> The cause of this issue is that SiteToSiteRestApiClient.openConnectionForSend
> method uses int variables to track bytes it sends. When the total size
> exceeds Integer.MAX_VALUE (2,147,483,647 = 2GB), the int variables get
> overflown. Then the check condition goes wrong.
> https://github.com/apache/nifi/blob/master/nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/util/SiteToSiteRestApiClient.java#L777
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
