Re: Kinesis getRecords read timeout and retry

2018-04-18 Thread Thomas Weise
--> On Wed, Apr 18, 2018 at 3:47 AM, Tzu-Li (Gordon) Tai wrote: > Hi Thomas, > > I see. If exposing access for these internal classes is a must to enable > further contributions, then I would agree to do so. > I think in the future, we should also keep a closer eye on parts of the > connector co

Re: Kinesis getRecords read timeout and retry

2018-04-18 Thread Tzu-Li (Gordon) Tai
Hi Thomas, I see. If exposing access for these internal classes is a must to enable further contributions, then I would agree to do so. I think in the future, we should also keep a closer eye on parts of the connector code which is highly subject to modifications on a per-environment basis and kee

Re: Kinesis getRecords read timeout and retry

2018-04-16 Thread Thomas Weise
Hi Gordon, This is indeed a discussion necessary to have! The purpose of previous PRs wasn't to provide solutions to the original identified issues, but rather to enable solve those through customization. What those customizations would be was also communicated, along with the intent to contribut

Re: Kinesis getRecords read timeout and retry

2018-04-15 Thread Tzu-Li (Gordon) Tai
Hi Thomas, Thanks for your PRs! I understand and fully agree with both points that you pointed out. What I'm still a bit torn with is the current proposed solutions for these issues (and other similar connector issues). This might actually call for a good opportunity to bring some thoughts up a

Re: Kinesis getRecords read timeout and retry

2018-04-02 Thread Thomas Weise
PR to provide the hooks: https://github.com/apache/flink/pull/5803 On Mon, Apr 2, 2018 at 6:14 PM, Thomas Weise wrote: > Hi, > > I’m working on implementing retry for getRecords in FlinkKinesisConsumer. > We occasionally get transient socket read timeouts. Instead of bubbling up > the exceptio

Kinesis getRecords read timeout and retry

2018-04-02 Thread Thomas Weise
Hi, I’m working on implementing retry for getRecords in FlinkKinesisConsumer. We occasionally get transient socket read timeouts. Instead of bubbling up the exception and forcing a topology reset to checkpoint, we want to retry getRecords. We also want to work with a lower socket read timeout than