Hi Ribat,
I haven't checked your PR but we introduced a new connector in flink 1.6
called StreamingFileSink that is supposed to replace BucketingSink long
term. I think it might solve a few problems of yours. Have you checked it
by chance?

Best,
Dawid

On Thu, 11 Oct 2018, 14:10 Rinat, <r.shari...@cleverdata.ru> wrote:

> Hi Piotr, during the migration to the latest Flink version, we’ve decided
> to try to contribute this functionality to the master branch.
>
> PR is available here https://github.com/apache/flink/pull/6824
> More details about hooking the state changes in BucketingSink are
> available in https://issues.apache.org/jira/browse/FLINK-9592
>
> Thx !
>
> On 14 Jun 2018, at 23:29, Rinat <r.shari...@cleverdata.ru> wrote:
>
> Hi Piotr, I’ve create an issue
> https://issues.apache.org/jira/browse/FLINK-9592
>
> The third proposal looks great, may I try to contribute this issue ?
>
> On 14 Jun 2018, at 12:29, Piotr Nowojski <pi...@data-artisans.com> wrote:
>
> Hi,
>
> Couple of things:
>
> 1. Please create a Jira ticket with this proposal, so we can move
> discussion from user mailing list.
>
> I haven’t thought it through, so take my comments with a grain of salt,
> however:
>
> 2. If we were to go with such callback, I would prefer to have one
> BucketStateChangeCallback, with methods like `onInProgressToPending(…)`,
> `onPendingToFinal`, `onPendingToCancelled(…)`, etc, in oppose to having one
> interface passed three times/four times for different purposes.
>
> 3. Other thing that I had in mind is that BucketingSink could be rewritten
> to extend TwoPhaseCommitSinkFunction. In that case, with
>
> public class BucketingSink2 extends TwoPhaseCommitSinkFunction<???>
>
> user could add his own hooks by overriding following methods
>
> BucketingSink2#beginTransaction,
> BucketingSink2#preCommit, BucketingSink2#commit, BucketingSink2#abort. For
> example:
>
> public class MyBucketingSink extends BucketingSink2 {
>   @Override
>   protected void  commit(??? txn) {
>     super.commit(txn);
>     // My hook on moving file from pending to commit state
>   };
> }
>
> Alternatively, we could implement before mentioned callbacks support in
> TwoPhaseCommitSinkFunction and provide such feature to
> Kafka/Pravega/BucketingSink at once.
>
> Piotrek
>
>
> Sincerely yours,
> *Rinat Sharipov*
> Software Engineer at 1DMP CORE Team
>
> email: r.shari...@cleverdata.ru <a.totma...@cleverdata.ru>
> mobile: +7 (925) 416-37-26
>
> CleverDATA
> make your data clever
>
>
> Sincerely yours,
> *Rinat Sharipov*
> Software Engineer at 1DMP CORE Team
>
> email: r.shari...@cleverdata.ru <a.totma...@cleverdata.ru>
> mobile: +7 (925) 416-37-26
>
> CleverDATA
> make your data clever
>
>

Reply via email to