We will try your recommendations. thanks RK
On Wed, Oct 3, 2018 at 10:47 AM Bryan Bende <[email protected]> wrote: > Have you tried with smaller batch settings? > > I think 2000 flow files and 1000MB would be a bit high. Might try > something like 100 flow files and 10MB. > > Also, if you concurrent tasks set to 7 on the RPG side, then you will > likely want to increase the concurrent tasks on Input Port on cluster > 2 where the data is being received. > On Tue, Oct 2, 2018 at 5:47 PM roozbeh karimi <[email protected]> > wrote: > > > > The only items that we set are: > > > > Remote Process Groups: > > URLs ... > > Transport Protocol: HTTP > > Communication Timeout: 60 sec > > Yield Duration: 2 sec > > > > Input Ports > > Concurrent Tasks: 7 > > Batch Settings > > Count: 2000 > > Size: 1000MB > > Duration: not set > > > > On Tue, Oct 2, 2018 at 3:47 PM Bryan Bende <[email protected]> wrote: > > > > > It looks like there may have been images attached that didn't make it > > > through. > > > > > > I'm not sure why S2S would be slow, it is made to handle very high > > > throughput. > > > > > > What I meant by #4 was... did you perform any specific configuration > of the > > > RPG on cluster 1 or the input port on cluster 2, such as changing batch > > > settings or modifying concurrent tasks, or did you use all default > values? > > > > > > > > > On Tue, Oct 2, 2018 at 2:01 PM roozbeh karimi <[email protected]> > > > wrote: > > > > > > > Hello Bryan > > > > 1) The communication is site to site between two separate > > > > instances/clusters of NiFi. > > > > 2) We have tried S2S in both raw and http modes, and neither made a > > > > difference. > > > > 3) The data flows but very slow, and after a while the back pressure > > > > reaches the threshold and we start losing data. > > > > 4) We have set up the RPG input port. > > > > 5) If we replace RPG with a posthttp processor on cluster 1 and a > > > listener > > > > processor on cluster 2 (where rpg nodes were), we don't see any back > > > > pressure buildup on cluster 1 and the data flows between the two > > > instances > > > > very fast. > > > > > > > > > > > > [image: image.png] > > > > [image: image.png] > > > > > > > > On Tue, Oct 2, 2018 at 11:55 AM Bryan Bende <[email protected]> > wrote: > > > > > > > >> Hello, > > > >> > > > >> Can you describe the setup a little bit more? > > > >> > > > >> Is this site-to-site between two separate NiFi instances, or is it > > > >> within the same cluster for purposes of redistribution? > > > >> > > > >> If it's separate instances then what type of instances, clustered or > > > >> standalone? > > > >> > > > >> Have you made any changes to the configuration of the RPG, the port > > > >> settings within the RPG, or the Input Port on the remote side? > > > >> > > > >> Is some data being transferred through the RPG and it is just slow, > or > > > >> is no data making it through? > > > >> > > > >> -Bryan > > > >> > > > >> On Mon, Oct 1, 2018 at 11:53 PM roozbeh karimi < > [email protected]> > > > >> wrote: > > > >> > > > > >> > Hello All, > > > >> > > > > >> > Are there any best practices and guides for setting up RPG? With > my > > > set > > > >> up, > > > >> > I am seeing a large volume of back pressure is being built up and > > > hardly > > > >> > any data is being transmitted from the source NiFi node to the > nodes > > > in > > > >> > RPG. If I replace RPG with PostHttp on the source node and a > listener > > > >> > processor on the destination node, no back pressure is built and > the > > > >> data > > > >> > is transmitted smoothly between the source and destination nodes. > > > >> > > > > >> > I'd appreciate any feedback on to go about tuning my RPG set up. > > > >> > > > > >> > Thanks > > > >> > RK > > > >> > > > > > > > >
