[DISCUSS] Hadoop 3, dropping support for Hadoop 2.x

Hi all, I've been assisting with some experiments to see how we might want to migrate Druid to support Hadoop 3.x, and more importantly, see if maybe we can finally be free of some of the dependency issues it has been causing for as long as I can remember working with Druid. Hadoop 3 introduced s

Re: [DISCUSS] Hadoop 3, dropping support for Hadoop 2.x

Hey Clint, I think it's definitely a step in the right direction. One thing I would suggest, since the are several deployments using Hadoop (either for deep storage and/or for ingestion), is to let the wider community know in advance that Hadoop 2.x support is going to be dropped in favor of 3.x (s

Re: Enabling dependabot in our github repository

Here's a running list of PRs opened by the dependabot: https://github.com/apache/druid/pulls?q=is%3Apr+author%3Aapp%2Fdependabot On Mon, Jun 7, 2021 at 12:22 PM Gian Merlino wrote: > There's been some extra discussion this PR: > https://github.com/apache/druid/pull/11079 > > I just +1'ed it, but

Re: [DISCUSS] Hadoop 3, dropping support for Hadoop 2.x

Also how about officially supporting minio? I know that support for s3 exists but it will be good to officially support minio as well as the deep storage. * Rajiv From: Clint Wylie Date: Tuesday, June 8, 2021 at 1:08 AM To: dev@druid.apache.org Subject: [DISCUSS] Hadoop 3, dropping suppor

Re: [DISCUSS] Hadoop 3, dropping support for Hadoop 2.x

Clint, thank you for starting this thread. I love the idea of dropping support for Hadoop 2.x. The shaded jars will definitely help us upgrade our rusty dependencies. Another problem with hadoop is that the hadoop ingestion lives in the Druid core today, not in a separate extension. Longer term, we

Re: Enabling dependabot in our github repository

I agree that PRs should not be committed immediately and unconditionally when the dependabot finds them. But if we defer, there is a concern that good PRs will be forgotten. How about making a particular person (say the release manager) or triggering event (say voting on an RC) responsible for c

Re: [E] [DISCUSS] Hadoop 3, dropping support for Hadoop 2.x

Unfortunately, the migration off of hadoop3 is a hard one (maybe not for Druid, but certainly for big organizations running large hadoop2 workloads). If druid migrated to hadoop3 after 0.22, that would probably prevent me from taking any new versions of Druid for at least the remainder of the year

Re: [E] [DISCUSS] Hadoop 3, dropping support for Hadoop 2.x

Just to follow up on this, our main problem with hadoop3 right now has been instability in HDFS, to the extent that we put on hold any plans to deploy it to our production systems. I would claim Hadoop3 isn't mature enough yet to consider migrating Druid to it. WIll

Re: [VOTE] Release Apache Druid 0.21.1 [RC2]

+1 (binding) src - verified signature/checksum - LICENSE/NOTICE present - ran RAT check - ran unit tests - built binary and ran ingestion tutorial and a few queries bin - verified signature/checksum - LICENSE/NOTICE present - ran ingestion tutorial and a few queries docker - built docker image fr

Re: [E] [DISCUSS] Hadoop 3, dropping support for Hadoop 2.x

Considering Druid takes advantage of lots of external components to work, I think we should upgrade Druid in a little bit conservitive way. Dropping support of hadoop2 is not a good idea. The upgrading of the ZooKeeper client in Druid also prevents me from adopting 0.22 for a longer time. Although

[RESULT] [VOTE] Release Apache Druid 0.21.1 [RC2]

Thanks to everyone who participated in the vote! The vote has passed with 3 binding +1s and 1 non-binding +1. Clint Wylie: +1 (binding) Frank Chen: +1 (non-binding) Jihoon Son: +1 (binding) Jon Wei: +1 (binding)

Re: [VOTE] Release Apache Druid 0.21.1 [RC2]

This vote has passed, the final results can be seen in this thread: https://lists.apache.org/thread.html/r3c7db826cdf9025efa2f3906e4f0d2ff69b66ae4e3513212a0bad2e3%40%3Cdev.druid.apache.org%3E On Tue, Jun 8, 2021 at 5:38 PM Jonathan Wei wrote: > +1 (binding) > src > - verified signature/checksum

Re: [E] [DISCUSS] Hadoop 3, dropping support for Hadoop 2.x

@itai, I think pending the outcome of this discussion that it makes sense to have a wider community thread to announce any decisions we make here, thanks for bringing that up. @rajiv, Minio support seems unrelated to this discussion. It seems like a reasonable request, but I recommend starting ano