Re: Sink buffering

2017-10-04 Thread nragon
Got it :) Thanks -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Sink buffering

2017-10-04 Thread Piotr Nowojski
Interval - Yes. TwoPhaseCommitSinkFunction - yes, but it depends how will you implement your “Transaction” class, it wouldn’t make a lot of sense, but you could store events inside the transaction “pojo”. Piotrek > On Oct 4, 2017, at 12:45 PM, nragon > wrote: > > checkpointing interval ~=

Re: Sink buffering

2017-10-04 Thread nragon
checkpointing interval ~= transactions are being committed on each Flink checkpoint So, if i set my checkpoint interval to 1ms, every 1ms there will be a commit, right? If I understoop correctly, TwoPhaseCommitSinkFunction stores transactions into it's state as for GenericWriteAheadSink it

Re: Sink buffering

2017-10-04 Thread Piotr Nowojski
What do you mean by "This always depends on checkpointing interval right?”? In TwoPhaseCommitSinkFunction, transactions are being committed on each Flink checkpoint. I guess same applies to GenericWriteAheadSink. The first one just commits/pre-commits the data on checkpoint, second rewrites them

Re: Sink buffering

2017-10-04 Thread nragon
Thanks for you opinion on this. TwoPhaseCommitSinkFunction would probably be the best solution overall. Using this with something like Phoenix or Tephra would probably work. This always depends on checkpointing interval right? -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.

Re: Sink buffering

2017-10-04 Thread Piotr Nowojski
> On Sep 29, 2017, at 6:21 PM, nragon > wrote: > > Hi, > > Just like mentioned at Berlin FF17, Pravega talk, can we simulate, somehow, > sink buffering(pravega transactions) and coordinate them with checkpoints? > My intension is to buffer records before sending them to

Re: Sink buffering

2017-10-03 Thread nragon
Anyone? :) -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Sink buffering

2017-09-29 Thread nragon
Hi, Just like mentioned at Berlin FF17, Pravega talk, can we simulate, somehow, sink buffering(pravega transactions) and coordinate them with checkpoints? My intension is to buffer records before sending them to hbase. Any opinions or tips? Thanks -- Sent from: http://apache-flink-user