By the way: I have disabled the test in the Hadoop 2.4.1 build, so my original problem is resolved.
I'm not convinced that posting on the user@ ml is the right approach. The last few questions to the user@ list asking for feedback have not really been answered. I believe that we have deprecated the sink for long enough, also we seem to have a common understanding of missing features we need to resolve first. Last but not least, we can always port the bucketing sink to another repository (apache bahir) and keep compatibility with later Flink versions there if a lot of users complain once it has been removed. On Tue, Mar 17, 2020 at 10:35 AM Kostas Kloudas <[email protected]> wrote: > Thanks Robert for all this, > > I think that we should also post a thread in the user ML so that users > can also comment on the topic. > > What do you think? > > Kostas > > On Mon, Mar 16, 2020 at 12:27 PM Robert Metzger <[email protected]> > wrote: > > > > Thank you all for your feedback. > > > > I will try to fix the test then (or disable it). > > > > Here's a ticket for dropping the BucketingSink: > > https://issues.apache.org/jira/browse/FLINK-16616 Please mark whatever > we > > consider necessary as a "depends on" ticket. > > @David / @Seth: Where are the tickets depending on FLIP-46 listed? Can > one > > of you add them to FLINK-16616 > > > > > > On Fri, Mar 13, 2020 at 11:32 AM Guowei Ma <[email protected]> wrote: > > > > > +1 to drop it. > > > > > > To Jingsong : > > > we are planning to implement the orc StreamingFileSink in 1.11. > > > I think users also could reference the old BucktSink from the old > version. > > > > > > Best, > > > Guowei > > > > > > > > > Jingsong Li <[email protected]> 于2020年3月13日周五 上午10:07写道： > > > > > > > Hi Robert, > > > > > > > > +1 to drop it but maybe not 1.11. > > > > > > > > ORC has not been supported on StreamingFileSink. I have seen lots of > > > users > > > > run ORC in the bucketing sink. > > > > > > > > Best, > > > > Jingsong Lee > > > > > > > > On Fri, Mar 13, 2020 at 1:11 AM Seth Wiesman <[email protected]> > > > wrote: > > > > > > > > > Sorry, I meant FLIP-46. > > > > > > > > > > Seth > > > > > > > > > > On Thu, Mar 12, 2020 at 11:52 AM Seth Wiesman <[email protected] > > > > > > wrote: > > > > > > > > > > > I agree with David, I think FLIP-49 needs to be prioritized for > 1.11 > > > if > > > > > we > > > > > > want to drop the bucketing sink. > > > > > > > > > > > > Seth > > > > > > > > > > > > On Thu, Mar 12, 2020 at 10:53 AM David Anderson < > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > >> The BucketingSink is still somewhat widely used, I think in part > > > > because > > > > > >> of > > > > > >> shortcomings in the StreamingFileSink. > > > > > >> > > > > > >> I would hope that in tandem with removing the bucketing sink we > > > could > > > > > also > > > > > >> address some of these issues. I'm thinking in particular of > issues > > > > that > > > > > >> are > > > > > >> waiting on FLIP-46 [1]. > > > > > >> > > > > > >> Removing the bucketing sink will go down better, in my opinion, > if > > > > it's > > > > > >> coupled with progress on some of the open StreamingFileSink > tickets. > > > > > >> > > > > > >> Best, > > > > > >> David > > > > > >> > > > > > >> [1] > > > > > >> > > > > > >> > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-46%3A+Graceful+Shutdown+Handling+by+UDFs > > > > > >> > > > > > >> > > > > > >> On Thu, Mar 12, 2020 at 4:27 PM Zhijiang < > > > [email protected] > > > > > >> .invalid> > > > > > >> wrote: > > > > > >> > > > > > >> > Thanks for driving this discussion, Robert! > > > > > >> > > > > > > >> > This e2e test really fails frequently. +1 to drop bucketing > sink, > > > > it > > > > > is > > > > > >> > not worth paying more efforts since deprecated. > > > > > >> > > > > > > >> > Best, > > > > > >> > Zhijiang > > > > > >> > > > > > > >> > > > > > > >> > > ------------------------------------------------------------------ > > > > > >> > From:Jeff Zhang <[email protected]> > > > > > >> > Send Time:2020 Mar. 12 (Thu.) 23:17 > > > > > >> > To:dev <[email protected]> > > > > > >> > Subject:Re: [DISCUSS] Drop Bucketing Sink > > > > > >> > > > > > > >> > +1， dropping deprecated api is always necessary for a > sustainable > > > > > >> project. > > > > > >> > > > > > > >> > Kostas Kloudas <[email protected]> 于2020年3月12日周四 下午11:06写道： > > > > > >> > > > > > > >> > > Hi Robert, > > > > > >> > > > > > > > >> > > +1 for dropping the BucketingSink. > > > > > >> > > In any case, it has not been maintained for quite some time. > > > > > >> > > > > > > > >> > > Cheers, > > > > > >> > > Kostas > > > > > >> > > > > > > > >> > > On Thu, Mar 12, 2020 at 3:41 PM Robert Metzger < > > > > [email protected] > > > > > > > > > > > >> > > wrote: > > > > > >> > > > > > > > > >> > > > Hi all, > > > > > >> > > > > > > > > >> > > > I'm currently investigating a failing end to end test for > the > > > > > >> bucketing > > > > > >> > > > sink [1]. > > > > > >> > > > The bucketing sink has been deprecated in the 1.9 release > [2], > > > > > >> because > > > > > >> > we > > > > > >> > > > have the new StreamingFileSink [3] for quite a while. > > > > > >> > > > Before putting any effort into fixing the end to end test > for > > > > the > > > > > >> > sink, I > > > > > >> > > > wanted to propose dropping the bucketing sink from master > for > > > > the > > > > > >> > > upcoming > > > > > >> > > > 1.11 release. > > > > > >> > > > > > > > > >> > > > What do you think? > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > [1] https://issues.apache.org/jira/browse/FLINK-16227 > > > > > >> > > > [2] https://issues.apache.org/jira/browse/FLINK-13396 > > > > > >> > > > [3] https://issues.apache.org/jira/browse/FLINK-9749 > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > >> > -- > > > > > >> > Best Regards > > > > > >> > > > > > > >> > Jeff Zhang > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Best, Jingsong Lee > > > > > > > >
