Have you tried with smaller batch settings? I think 2000 flow files and 1000MB would be a bit high. Might try something like 100 flow files and 10MB.
Also, if you concurrent tasks set to 7 on the RPG side, then you will likely want to increase the concurrent tasks on Input Port on cluster 2 where the data is being received. On Tue, Oct 2, 2018 at 5:47 PM roozbeh karimi <[email protected]> wrote: > > The only items that we set are: > > Remote Process Groups: > URLs ... > Transport Protocol: HTTP > Communication Timeout: 60 sec > Yield Duration: 2 sec > > Input Ports > Concurrent Tasks: 7 > Batch Settings > Count: 2000 > Size: 1000MB > Duration: not set > > On Tue, Oct 2, 2018 at 3:47 PM Bryan Bende <[email protected]> wrote: > > > It looks like there may have been images attached that didn't make it > > through. > > > > I'm not sure why S2S would be slow, it is made to handle very high > > throughput. > > > > What I meant by #4 was... did you perform any specific configuration of the > > RPG on cluster 1 or the input port on cluster 2, such as changing batch > > settings or modifying concurrent tasks, or did you use all default values? > > > > > > On Tue, Oct 2, 2018 at 2:01 PM roozbeh karimi <[email protected]> > > wrote: > > > > > Hello Bryan > > > 1) The communication is site to site between two separate > > > instances/clusters of NiFi. > > > 2) We have tried S2S in both raw and http modes, and neither made a > > > difference. > > > 3) The data flows but very slow, and after a while the back pressure > > > reaches the threshold and we start losing data. > > > 4) We have set up the RPG input port. > > > 5) If we replace RPG with a posthttp processor on cluster 1 and a > > listener > > > processor on cluster 2 (where rpg nodes were), we don't see any back > > > pressure buildup on cluster 1 and the data flows between the two > > instances > > > very fast. > > > > > > > > > [image: image.png] > > > [image: image.png] > > > > > > On Tue, Oct 2, 2018 at 11:55 AM Bryan Bende <[email protected]> wrote: > > > > > >> Hello, > > >> > > >> Can you describe the setup a little bit more? > > >> > > >> Is this site-to-site between two separate NiFi instances, or is it > > >> within the same cluster for purposes of redistribution? > > >> > > >> If it's separate instances then what type of instances, clustered or > > >> standalone? > > >> > > >> Have you made any changes to the configuration of the RPG, the port > > >> settings within the RPG, or the Input Port on the remote side? > > >> > > >> Is some data being transferred through the RPG and it is just slow, or > > >> is no data making it through? > > >> > > >> -Bryan > > >> > > >> On Mon, Oct 1, 2018 at 11:53 PM roozbeh karimi <[email protected]> > > >> wrote: > > >> > > > >> > Hello All, > > >> > > > >> > Are there any best practices and guides for setting up RPG? With my > > set > > >> up, > > >> > I am seeing a large volume of back pressure is being built up and > > hardly > > >> > any data is being transmitted from the source NiFi node to the nodes > > in > > >> > RPG. If I replace RPG with PostHttp on the source node and a listener > > >> > processor on the destination node, no back pressure is built and the > > >> data > > >> > is transmitted smoothly between the source and destination nodes. > > >> > > > >> > I'd appreciate any feedback on to go about tuning my RPG set up. > > >> > > > >> > Thanks > > >> > RK > > >> > > > > >
