Gradle Consistent Versions makes it such that it’s not allowed to have two different Spark versions in the module. I might be missing some context here, but why should Iceberg ever depend on two different versions of Spark at a time?
-Matt Cheah From: Ryan Blue <rb...@netflix.com.INVALID> Reply-To: "dev@iceberg.apache.org" <dev@iceberg.apache.org>, "rb...@netflix.com" <rb...@netflix.com> Date: Tuesday, March 3, 2020 at 8:32 AM To: Iceberg Dev List <dev@iceberg.apache.org> Subject: Re: [Discuss] Merge spark-3 branch into master Thanks for bringing this up, Saisai. I tried to do this a couple of months ago, but ran into a problem with dependency locks. I couldn't get two different versions of Spark packages in the build with baseline, but maybe I was missing something. If you can get it working, I think it's a great idea to get this into master. Otherwise, I was thinking about proposing an 0.8.0 release in the next month or so based on Spark 2.4. Then we could merge the branch into master and do another release for Spark 3.0 when it's ready. rb On Tue, Mar 3, 2020 at 6:07 AM Saisai Shao <sai.sai.s...@gmail.com<mailto:sai.sai.s...@gmail.com>> wrote: Hi team, I was thinking of merging spark-3 branch into master, also per the discussion before we could make spark-2 and spark-3 coexisted into 2 different sub-modules. With this, one build could generate both spark-2 and spark-3 runtime jars, user could pick either at preference. One concern is that they share lots of common code in read/write path, this will increase the maintenance overhead to keep consistency of two copies. So I'd like to hear your thoughts, any suggestions on it? Thanks Saisai -- Ryan Blue Software Engineer Netflix
