Re: Consuming data from dynamoDB streams to flink

Thank you so much Fabian! Will update status in the JIRA. - Ying On Tue, Jul 31, 2018 at 1:37 AM, Fabian Hueske wrote: > Done! > > Thank you :-) > > 2018-07-31 6:41 GMT+02:00 Ying Xu : > > > Thanks Fabian and Thomas. > > > > Please assign FLINK-4582 to the following username: > > *yxu-apac

Re: Consuming data from dynamoDB streams to flink

Done! Thank you :-) 2018-07-31 6:41 GMT+02:00 Ying Xu : > Thanks Fabian and Thomas. > > Please assign FLINK-4582 to the following username: > *yxu-apache > * > > If needed I can get a ICLA or CCLA whichever is proper. >

Re: Consuming data from dynamoDB streams to flink

Thanks Fabian and Thomas. Please assign FLINK-4582 to the following username: *yxu-apache * If needed I can get a ICLA or CCLA whichever is proper. *Ying Xu* Software Engineer 510.368.1252 <+15103681252> [image: Lyft]

Re: Consuming data from dynamoDB streams to flink

The user is yxu-lyft, Ying had commented on that JIRA as well. https://issues.apache.org/jira/browse/FLINK-4582 On Mon, Jul 30, 2018 at 1:25 AM Fabian Hueske wrote: > Hi Ying, > > Thanks for considering to contribute the connector! > > In general, you don't need special permissions to contribu

Re: Consuming data from dynamoDB streams to flink

Hi Ying, Thanks for considering to contribute the connector! In general, you don't need special permissions to contribute to Flink. Anybody can open Jiras and PRs. You only need to be assigned to the Contributor role in Jira to be able to assign an issue to you. I can give you these permissions i

Re: Consuming data from dynamoDB streams to flink

Hello Flink dev: We have implemented the prototype design and the initial PoC worked pretty well. Currently, we plan to move ahead with this design in our internal production system. We are thinking of contributing this connector back to the flink community sometime soon. May I request to be gr

Re: Consuming data from dynamoDB streams to flink

Hi Gordon: Cool. Thanks for the thumb-up! We will include some test cases around the behavior of re-sharding. If needed we can double check the behavior with AWS, and see if additional changes are needed. Will keep you posted. - Ying On Wed, Jul 4, 2018 at 7:22 PM, Tzu-Li (Gordon) Tai wrote:

Re: Consuming data from dynamoDB streams to flink

Hi Ying, Sorry for the late reply here. >From the looks of the AmazonDynamoDBStreamsClient, yes it seems like this should simply work. Regarding the resharding behaviour I mentioned in the JIRA: I'm not sure if this is really a difference in behaviour. Internally, if DynamoDB streams is actually

Re: Consuming data from dynamoDB streams to flink

HI Gordon: We are starting to implement some of the primitives along this path. Please let us know if you have any suggestions. Thanks! On Fri, Jun 29, 2018 at 12:31 AM, Ying Xu wrote: > Hi Gordon: > > Really appreciate the reply. > > Yes our plan is to build the connector on top of the FlinkK

Re: Consuming data from dynamoDB streams to flink

Hi Gordon: Really appreciate the reply. Yes our plan is to build the connector on top of the FlinkKinesisConsumer. At the high level, FlinkKinesisConsumer mainly interacts with Kinesis through the AmazonKinesis client, more specifically through the following three function calls: - describeSt

Re: Consuming data from dynamoDB streams to flink

Hi! I think it would be definitely nice to have this feature. No actual previous work has been made on this issue, but AFAIK, we should be able to build this on top of the FlinkKinesisConsumer. Whether this should live within the Kinesis connector module or an independent module of its own is sti

Re: Consuming data from dynamoDB streams to flink

Thanks Fabian for the suggestion. *Ying Xu* Software Engineer 510.368.1252 <+15103681252> [image: Lyft] On Wed, Jun 27, 2018 at 2:01 AM, Fabian Hueske wrote: > Hi Ying, > > I'm not aware of any effort for this issue. > You could check with the assigned contributor in Jira

Re: Consuming data from dynamoDB streams to flink

Hi Ying, I'm not aware of any effort for this issue. You could check with the assigned contributor in Jira if there is some previous work. Best, Fabian 2018-06-26 9:46 GMT+02:00 Ying Xu : > Hello Flink dev: > > We have a number of use cases which involves pulling data from DynamoDB > streams in

Consuming data from dynamoDB streams to flink

Hello Flink dev: We have a number of use cases which involves pulling data from DynamoDB streams into Flink. Given that this issue is tracked by Flink-4582 . we would like to check if any prior work has been completed by the community. We are al