Thanks for the excellent proposal! Big +1 for introducing EFO as an incremental feature while retaining backward compatibility! This will make it easier for users to adopt.
Thanks for mentioning the reasons why one might not want to use EFO. Regarding "Streams with a single consumer would not benefit from the dedicated throughput (they already have the full quota)": Do you see the polling (non-EFO) mode as a permanent option going forward? Regarding "Registration/De-registration Configuration": The limit for "ListStreamConsumers" is 5 TPS per [1], which is even lower than that for "DescribeStream". That limit could cause significant issues during large scale job startup and the only solution was to switch to ListShards. Perhaps elaborate a bit more on limitations and reasoning for the different registration options? De-registration may never happen when task managers go into a bad state and are forcefully terminated. That won't cause an issue because a stale registration will be overridden/removed by a new job with the same name? Thanks, Thomas [1] https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html On Mon, Jun 22, 2020 at 2:42 AM Cranmer, Danny <cranm...@amazon.com.invalid> wrote: > Hello everyone, > This is a discussion thread for the FLIP [1] regarding Enhanced Fan Out > for AWS Kinesis Consumers. > > Enhanced Fan Out (EFO) allows AWS Kinesis Data Stream (KDS) consumers to > utilise a dedicated read throughput, rather than a shared quota. HTTP/2 > reduces latency and typically gives a 65% performance boost [2]. EFO is not > currently supported by the Flink Kinesis Consumer. Adding EFO support would > allow Flink applications to reap the benefits, widening Flink adoption. > Existing applications will be able to optionally perform a backwards > compatible library upgrade and configuration tweak to inherit the > performance benefits. > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-128%3A+Enhanced+Fan+Out+for+AWS+Kinesis+Consumers > [2] https://aws.amazon.com/blogs/aws/kds-enhanced-fanout/ > I look forward to your feedback, > Thanks, >
