On Tue, Nov 06, 2018 at 07:44:50PM +0200, Nicos Maris wrote: > Ning can you provide another example except for rate limiting?
Our main use case and concern is rate limiting because without it we could potentially overwhelm downstream systems (Cassandra) when the job plays catch up or replay events from Kafka. The consequence of that is that exceptions will be thrown in the Cassandra sink causing the whole job to restart and end up in the same situation over and over. With that said, we do have other use cases such as doing canary deployment. We'd like to start processing events for a subset of users, then expand it to all if things look good. Without live configuration, we have to take a savepoint and restart the job, which will cause the job to play catch up at the beginning, potentially overwhelming downstream system without rate limiting. Hope the use cases described above clarifies. Thanks, -- Ning
