[jira] [Created] (FLINK-15476) Update StreamingFileSink documentation -- bulk encoded writer now supports customized checkpoint policy

Ying Xu created FLINK-15476: --- Summary: Update StreamingFileSink documentation -- bulk encoded writer now supports customized checkpoint policy Key: FLINK-15476 URL: https://issues.apache.org/jira/browse/FLINK-15476

[jira] [Created] (FLINK-15301) Flink Kinesis AsyncRecordEmitter needs to handle unchecked exception gracefully

Ying Xu created FLINK-15301: --- Summary: Flink Kinesis AsyncRecordEmitter needs to handle unchecked exception gracefully Key: FLINK-15301 URL: https://issues.apache.org/jira/browse/FLINK-15301 Project: Flink

[jira] [Created] (FLINK-14039) Flink Kinesis consumer: configurable per-shard consumption rate when running in adaptive mode

Ying Xu created FLINK-14039: --- Summary: Flink Kinesis consumer: configurable per-shard consumption rate when running in adaptive mode Key: FLINK-14039 URL: https://issues.apache.org/jira/browse/FLINK-14039

Re: Custom type serializer inside Flink state

gt; > CC @Tzu-Li (Gordon) Tai<mailto:tzuli...@apache.org> as he might provide > more information about this. > > [1] https://issues.apache.org/jira/browse/FLINK-11333 > > Best > Yun Tang > > > From: Ying Xu > Sent: Thursday, August

Custom type serializer inside Flink state

Hi Flink community: Our Flink application sends different types of protobuf messages on-the-wire. Since protobuf cannot be handled by Flink type serializer, we had to register custom Kyro serializer: env.getConfig().*registerTypeWithKryoSerializer*(MyProtoClass1.class, ProtobufSerializer.class);

Re: Rolling policy when using StreamingFileSink for bulk-encoded output

On Fri, Jun 28, 2019 at 9:51 AM Ying Xu wrote: > Hi Kostas: > > I'd like to. The account used to file the JIRA does not have contributor > access yet . I had contributed a few Flink JIRAs in the past, using a very > similar but different account. Now I would like to consolidate

Re: Rolling policy when using StreamingFileSink for bulk-encoded output

AM Kostas Kloudas wrote: > Hi Ying, > > That sounds great! > Looking forward to your PR! > > Btw don't you want to assign the issue to yourself if you are > planning to work on it? > > Kostas > > On Fri, Jun 28, 2019 at 9:54 AM Ying Xu wrote: > > >

Re: Rolling policy when using StreamingFileSink for bulk-encoded output

f things. > Could you open a JIRA so that we can keep track of the progress? > > Cheers, > Kostas > > On Tue, Jun 25, 2019 at 9:49 AM Ying Xu wrote: > > > HI Kostas: > > > > Thanks for the prompt reply. > > > > The file rolling policy mentioned previou

[jira] [Created] (FLINK-13027) StreamingFileSink bulk-encoded writer supports file rolling upon customized events

Ying Xu created FLINK-13027: --- Summary: StreamingFileSink bulk-encoded writer supports file rolling upon customized events Key: FLINK-13027 URL: https://issues.apache.org/jira/browse/FLINK-13027 Project

Re: Rolling policy when using StreamingFileSink for bulk-encoded output

amingFileSink has this > limitation. > > Cheers, > Kostas > > > On Mon, Jun 24, 2019 at 9:19 AM Ying Xu wrote: > > > Dear Flink community: > > > > We have a use case where StreamingFileSink > > < > > > https://ci.apache.org/proje

Rolling policy when using StreamingFileSink for bulk-encoded output

Dear Flink community: We have a use case where StreamingFileSink is used for persisting bulk-encoded data to AWS s3. In our case, the data sources consist of hybrid types of events, for which each type is

[jira] [Created] (FLINK-10358) Flink kinesis connector could throw NPE during getRecords() call

Ying Xu created FLINK-10358: --- Summary: Flink kinesis connector could throw NPE during getRecords() call Key: FLINK-10358 URL: https://issues.apache.org/jira/browse/FLINK-10358 Project: Flink

Re: Kinesis connector shard discovery mechanism

+1 on separating out the logic relevant to Kinesalite. Kinesalite is likely used in testing environment a lot. On Thu, Sep 13, 2018 at 1:52 AM, Евгений Юшин wrote: > Hi there > > Flink Kinesis consumer checks shards id for a particular pattern: > "^shardId-\\d{12}" > > https://github.com/apache

Re: Consuming data from dynamoDB streams to flink

Thank you so much Fabian! Will update status in the JIRA. - Ying On Tue, Jul 31, 2018 at 1:37 AM, Fabian Hueske wrote: > Done! > > Thank you :-) > > 2018-07-31 6:41 GMT+02:00 Ying Xu : > > > Thanks Fabian and Thomas. > > > > Please assign FLINK-4582 to

Re: Consuming data from dynamoDB streams to flink

Thanks Fabian and Thomas. Please assign FLINK-4582 to the following username: *yxu-apache <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=yxu-apache>* If needed I can get a ICLA or CCLA whichever is proper. *Ying Xu* Software Engineer 510.368.1252 <+15103681252>

Re: Consuming data from dynamoDB streams to flink

granted with a contributor role? Many thanks in advance. *Ying Xu* Software Engineer 510.368.1252 <+15103681252> [image: Lyft] <http://www.lyft.com/> On Fri, Jul 6, 2018 at 6:23 PM, Ying Xu wrote: > Hi Gordon: > > Cool. Thanks for the thumb-up! > > We will include

Re: Consuming data from dynamoDB streams to flink

secutive shard ids. As long as this is also the behaviour for DynamoDB > resharding, then we should be fine. > > Feel free to start with the implementation for this, I think design-wise > we're good to go. And thanks for working on this! > > Cheers, > Gordon > > On Wed,

Re: Consuming data from dynamoDB streams to flink

HI Gordon: We are starting to implement some of the primitives along this path. Please let us know if you have any suggestions. Thanks! On Fri, Jun 29, 2018 at 12:31 AM, Ying Xu wrote: > Hi Gordon: > > Really appreciate the reply. > > Yes our plan is to build the connecto

Re: Consuming data from dynamoDB streams to flink

onnector module or an > independent module of its own is still TBD. > If you want, I would be happy to look at any concrete design proposals you > have for this before you start the actual development efforts. > > Cheers, > Gordon > > On Thu, Jun 28, 2018 at 2:12 AM Ying X

Re: Consuming data from dynamoDB streams to flink

Thanks Fabian for the suggestion. *Ying Xu* Software Engineer 510.368.1252 <+15103681252> [image: Lyft] <http://www.lyft.com/> On Wed, Jun 27, 2018 at 2:01 AM, Fabian Hueske wrote: > Hi Ying, > > I'm not aware of any effort for this issue. > You could check wit

Consuming data from dynamoDB streams to flink

Hello Flink dev: We have a number of use cases which involves pulling data from DynamoDB streams into Flink. Given that this issue is tracked by Flink-4582 . we would like to check if any prior work has been completed by the community. We are al