Given the observed behavior, I am in support of removing this property as well. In addition, the fact that this property is on the listening side and not the posting side indicates this is a flow rate limiter. It is not attempting to throttle the transfer rate. Therefore, the appropriate - and arguably more desirable - approach here is to let the data flow handle the rate throttle. As you've suggested, ControlRate and possible back pressure that results is a great solution.
The inclusion of a use case in the processor documentation is a great idea and will assist users who may feel they lost functionality. As Lucas mentions, this change should be clearly documented in the migration guide when it is included in a release. It's not always a great idea to change some base processors that are very heavily utilized. However, the benefits here are significant especially when possible duplication of data may occur. +1 on recommended changes including documentation. On Thu, Mar 13, 2025 at 3:45 PM Lucas Ottersbach <ottersb...@apache.org> wrote: > > Thanks for opening the discussion Michael. > > Separating the concerns of HTTP and rate limiting sounds reasonable to me. > Especially when considering the limitations of the current implementation > you've outlined. > +1 from me on this issue. > > While I'm in favour, we should ensure this change is documented properly in > the migration guide. Otherwise, users might experience undesired loads due > to the limit being migrated away automatically but the corresponding > ControlRate processor missing. > > Best regards, > Lucas > > Michael Moser <moser...@gmail.com> schrieb am Do., 13. März 2025, 20:08: > > > Team, > > > > tl;dr I would like to remove the ListenHTTP "Max Data to Receive per > > Second" capability because it has problems. Thoughts? > > > > I've been experimenting with the ListenHTTP "Max Data to Receive per > > Second" feature and I've found some concerning results. > > If I set "Max Data to Receive per Second" to a low value such as "500 KB" > > and send a 5 MB file using InvokeHTTP, then ListenHTTP receives the file > > after 10 seconds. Great! > > But if the size of the file would take longer than the InvokeHTTP "Socket > > Read Timeout" (default 15 seconds), for example sending a 20 MB file (which > > would take 40 seconds), > > then after 15 seconds InvokeHTTP experiences a SocketTimeoutException and > > routes the file to Failure. After 40 seconds elapses, ListenHTTP completes > > and receives the 20 MB file! If the InvokeHTTP Failure relationship is > > retried, then the receiving system gets duplicate files. > > > > At larger "Max Data to Receive per Second" values such as "10 MB", > > SocketTimeoutExceptions are harder to reproduce but I can still make it > > happen by heavily loading ListenHTTP. > > > > As an alternative to this property, if a ListenHTTP is followed by > > ControlRate with tight back pressure in between, then a limited data rate > > appears to work much better. > > When InvokeHTTP is the client, and ListenHTTP experiences backpressure > > and does things like send a 503 response or deny connections or close > > connections, > > InvokeHTTP experiences various errors. This results in files going to > > either Retry or Failure relationships. > > When those are retried, then the number of files that InvokeHTTP wants to > > send exactly equal the number of files that ListenHTTP receives. Also > > great! > > > > So, I propose removing the ListenHTTP "Max Data to Receive per Second" > > property, with proper migration of properties. And documenting a UseCase > > on ListenHTTP to explain > > the recommended approach (backpressure) to rate limiting. > > > > Thanks for your thoughts, > > -- Mike > >
