+1, and something I was planning to comment on in the Jira issue. Also, if rate limiting could effectively stop the stream, then this could be used solve a common data enrichment issue.
Logically you want to pause one stream (typically the time series data being processed) while another stream (typically the broadcast) is broadcasting an update to enrichment data. Currently you have to buffer the time series data in your enrichment function, but if the rate limiter was pluggable, it could detect when this enrichment update was happening. — Ken > On Jan 31, 2019, at 6:10 PM, Becket Qin <becket....@gmail.com> wrote: > > Hi Jamie, > > Thanks for the explanation. That makes sense to me. I am wondering if there > is a more general way to add a rate limiter to all the connecters rather > than doing that for each individual one. For example, maybe we can have the > rate limiting logic in the Collector / Output, thus all the connectors > (even operators?) could be rate limited. > > Thanks, > > Jiangjie (Becket) Qin > > On Fri, Feb 1, 2019 at 7:23 AM Lakshmi Gururaja Rao <l...@lyft.com.invalid> > wrote: > >> Thanks for adding more context @Jamie Grier <jgr...@lyft.com> . >> >> JIRA for this feature: https://issues.apache.org/jira/browse/FLINK-11501. >> >> Thanks >> Lakshmi >> >> On Thu, Jan 31, 2019 at 3:20 PM Thomas Weise <t...@apache.org> wrote: >> >>> I think it would be reasonable to have a rate limiter option in the >>> consumer, given that others have also looked to solve this. >>> >>> I think for this and other optional features, it would be good to >> implement >>> in a way that overrides are possible. Someone else may want to do the >>> limiting differently, taking into account more/other factors. >>> >>> Both, adding the limiter and making the consumer code more adoptable >> could >>> be split into separate work also. >>> >>> BTW is there a JIRA for this? >>> >>> Thomas >>> >> >> >> -- >> *Lakshmi Gururaja Rao* >> SWE >> 217.778.7218 <+12177787218> >> [image: Lyft] <http://www.lyft.com/> >> -------------------------- Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com Custom big data solutions & training Flink, Solr, Hadoop, Cascading & Cassandra
