By the way: I have disabled the test in the Hadoop 2.4.1 build, so my original problem is resolved.
I'm not convinced that posting on the user@ ml is the right approach. The last few questions to the user@ list asking for feedback have not really been answered. I believe that we have deprecated the sink for long enough, also we seem to have a common understanding of missing features we need to resolve first. Last but not least, we can always port the bucketing sink to another repository (apache bahir) and keep compatibility with later Flink versions there if a lot of users complain once it has been removed. On Tue, Mar 17, 2020 at 10:35 AM Kostas Kloudas <kklou...@gmail.com> wrote: > Thanks Robert for all this, > > I think that we should also post a thread in the user ML so that users > can also comment on the topic. > > What do you think? > > Kostas > > On Mon, Mar 16, 2020 at 12:27 PM Robert Metzger <rmetz...@apache.org> > wrote: > > > > Thank you all for your feedback. > > > > I will try to fix the test then (or disable it). > > > > Here's a ticket for dropping the BucketingSink: > > https://issues.apache.org/jira/browse/FLINK-16616 Please mark whatever > we > > consider necessary as a "depends on" ticket. > > @David / @Seth: Where are the tickets depending on FLIP-46 listed? Can > one > > of you add them to FLINK-16616 > > > > > > On Fri, Mar 13, 2020 at 11:32 AM Guowei Ma <guowei....@gmail.com> wrote: > > > > > +1 to drop it. > > > > > > To Jingsong : > > > we are planning to implement the orc StreamingFileSink in 1.11. > > > I think users also could reference the old BucktSink from the old > version. > > > > > > Best, > > > Guowei > > > > > > > > > Jingsong Li <jingsongl...@gmail.com> 于2020年3月13日周五 上午10:07写道： > > > > > > > Hi Robert, > > > > > > > > +1 to drop it but maybe not 1.11. > > > > > > > > ORC has not been supported on StreamingFileSink. I have seen lots of > > > users > > > > run ORC in the bucketing sink. > > > > > > > > Best, > > > > Jingsong Lee > > > > > > > > On Fri, Mar 13, 2020 at 1:11 AM Seth Wiesman <sjwies...@gmail.com> > > > wrote: > > > > > > > > > Sorry, I meant FLIP-46. > > > > > > > > > > Seth > > > > > > > > > > On Thu, Mar 12, 2020 at 11:52 AM Seth Wiesman <sjwies...@gmail.com > > > > > > wrote: > > > > > > > > > > > I agree with David, I think FLIP-49 needs to be prioritized for > 1.11 > > > if > > > > > we > > > > > > want to drop the bucketing sink. > > > > > > > > > > > > Seth > > > > > > > > > > > > On Thu, Mar 12, 2020 at 10:53 AM David Anderson < > da...@ververica.com > > > > > > > > > > wrote: > > > > > > > > > > > >> The BucketingSink is still somewhat widely used, I think in part > > > > because > > > > > >> of > > > > > >> shortcomings in the StreamingFileSink. > > > > > >> > > > > > >> I would hope that in tandem with removing the bucketing sink we > > > could > > > > > also > > > > > >> address some of these issues. I'm thinking in particular of > issues > > > > that > > > > > >> are > > > > > >> waiting on FLIP-46 [1]. > > > > > >> > > > > > >> Removing the bucketing sink will go down better, in my opinion, > if > > > > it's > > > > > >> coupled with progress on some of the open StreamingFileSink > tickets. > > > > > >> > > > > > >> Best, > > > > > >> David > > > > > >> > > > > > >> [1] > > > > > >> > > > > > >> > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-46%3A+Graceful+Shutdown+Handling+by+UDFs > > > > > >> > > > > > >> > > > > > >> On Thu, Mar 12, 2020 at 4:27 PM Zhijiang < > > > wangzhijiang...@aliyun.com > > > > > >> .invalid> > > > > > >> wrote: > > > > > >> > > > > > >> > Thanks for driving this discussion, Robert! > > > > > >> > > > > > > >> > This e2e test really fails frequently. +1 to drop bucketing > sink, > > > > it > > > > > is > > > > > >> > not worth paying more efforts since deprecated. > > > > > >> > > > > > > >> > Best, > > > > > >> > Zhijiang > > > > > >> > > > > > > >> > > > > > > >> > > ------------------------------------------------------------------ > > > > > >> > From:Jeff Zhang <zjf...@gmail.com> > > > > > >> > Send Time:2020 Mar. 12 (Thu.) 23:17 > > > > > >> > To:dev <dev@flink.apache.org> > > > > > >> > Subject:Re: [DISCUSS] Drop Bucketing Sink > > > > > >> > > > > > > >> > +1， dropping deprecated api is always necessary for a > sustainable > > > > > >> project. > > > > > >> > > > > > > >> > Kostas Kloudas <kklou...@gmail.com> 于2020年3月12日周四 下午11:06写道： > > > > > >> > > > > > > >> > > Hi Robert, > > > > > >> > > > > > > > >> > > +1 for dropping the BucketingSink. > > > > > >> > > In any case, it has not been maintained for quite some time. > > > > > >> > > > > > > > >> > > Cheers, > > > > > >> > > Kostas > > > > > >> > > > > > > > >> > > On Thu, Mar 12, 2020 at 3:41 PM Robert Metzger < > > > > rmetz...@apache.org > > > > > > > > > > > >> > > wrote: > > > > > >> > > > > > > > > >> > > > Hi all, > > > > > >> > > > > > > > > >> > > > I'm currently investigating a failing end to end test for > the > > > > > >> bucketing > > > > > >> > > > sink [1]. > > > > > >> > > > The bucketing sink has been deprecated in the 1.9 release > [2], > > > > > >> because > > > > > >> > we > > > > > >> > > > have the new StreamingFileSink [3] for quite a while. > > > > > >> > > > Before putting any effort into fixing the end to end test > for > > > > the > > > > > >> > sink, I > > > > > >> > > > wanted to propose dropping the bucketing sink from master > for > > > > the > > > > > >> > > upcoming > > > > > >> > > > 1.11 release. > > > > > >> > > > > > > > > >> > > > What do you think? > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > [1] https://issues.apache.org/jira/browse/FLINK-16227 > > > > > >> > > > [2] https://issues.apache.org/jira/browse/FLINK-13396 > > > > > >> > > > [3] https://issues.apache.org/jira/browse/FLINK-9749 > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > >> > -- > > > > > >> > Best Regards > > > > > >> > > > > > > >> > Jeff Zhang > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Best, Jingsong Lee > > > > > > > >
