Oops - I actually should have added v0.13.0 (i.e. to match whatever I did in the profile).
On Tue, Oct 28, 2014 at 10:05 PM, Patrick Wendell <pwend...@gmail.com> wrote: > Cheng - to make it recognize the new HiveShim for 0.12 I had to click > on spark-hive under "packages" in the left pane, then go to "Open > Module Settings" - then explicitly add the v0.12.0/src/main/scala > folder to the sources by navigating to it and then <ctrl>+click to add > it as a source. Did you have to do this? > > On Tue, Oct 28, 2014 at 9:57 PM, Patrick Wendell <pwend...@gmail.com> wrote: >> I just started a totally fresh IntelliJ project importing from our >> root pom. I used all the default options and I added "hadoop-2.4, >> hive, hive-0.13.1" profiles. I was able to run spark core tests from >> within IntelliJ. Didn't try anything beyond that, but FWIW this >> worked. >> >> - Patrick >> >> On Tue, Oct 28, 2014 at 9:54 PM, Cheng Lian <lian.cs....@gmail.com> wrote: >>> You may first open the root pom.xml file in IDEA, and then go for menu View >>> / Tool Windows / Maven Projects, then choose desired Maven profile >>> combination under the "Profiles" node (e.g. I usually use hadoop-2.4 + hive >>> + hive-0.12.0). IDEA will ask you to re-import the Maven projects, confirm, >>> then it should be OK. >>> >>> I can debug within IDEA with this approach. However, you have to clean the >>> whole project before debugging Spark within IDEA if you compiled the project >>> outside IDEA. Haven't got time to investigate this annoying issue. >>> >>> Also, you can remove sub projects unrelated to your tasks to accelerate >>> compilation and/or avoid other IDEA build issues (e.g. Avro related Spark >>> streaming build failure in IDEA). >>> >>> >>> On 10/29/14 12:42 PM, Stephen Boesch wrote: >>> >>> I am interested specifically in how to build (and hopefully run/debug..) >>> under Intellij. Your posts sound like command line maven - which has always >>> been working already. >>> >>> Do you have instructions for building in IJ? >>> >>> 2014-10-28 21:38 GMT-07:00 Cheng Lian <lian.cs....@gmail.com>: >>>> >>>> Yes, these two combinations work for me. >>>> >>>> >>>> On 10/29/14 12:32 PM, Zhan Zhang wrote: >>>>> >>>>> -Phive is to enable hive-0.13.1 and "-Phive -Phive-0.12.0" is to enable >>>>> hive-0.12.0. Note that the thrift-server is not supported yet in >>>>> hive-0.13, >>>>> but expected to go to upstream soon (Spark-3720). >>>>> >>>>> Thanks. >>>>> >>>>> Zhan Zhang >>>>> >>>>> >>>>> On Oct 28, 2014, at 9:09 PM, Stephen Boesch <java...@gmail.com> wrote: >>>>> >>>>>> Thanks Patrick for the heads up. >>>>>> >>>>>> I have not been successful to discover a combination of profiles (i.e. >>>>>> enabling hive or hive-0.12.0 or hive-13.0) that works in Intellij with >>>>>> maven. Anyone who knows how to handle this - a quick note here would be >>>>>> appreciated. >>>>>> >>>>>> >>>>>> >>>>>> 2014-10-28 20:20 GMT-07:00 Patrick Wendell <pwend...@gmail.com>: >>>>>> >>>>>>> Hey Stephen, >>>>>>> >>>>>>> In some cases in the maven build we now have pluggable source >>>>>>> directories based on profiles using the maven build helper plug-in. >>>>>>> This is necessary to support cross building against different Hive >>>>>>> versions, and there will be additional instances of this due to >>>>>>> supporting scala 2.11 and 2.10. >>>>>>> >>>>>>> In these cases, you may need to add source locations explicitly to >>>>>>> intellij if you want the entire project to compile there. >>>>>>> >>>>>>> Unfortunately as long as we support cross-building like this, it will >>>>>>> be an issue. Intellij's maven support does not correctly detect our >>>>>>> use of the maven-build-plugin to add source directories. >>>>>>> >>>>>>> We should come up with a good set of instructions on how to import the >>>>>>> pom files + add the few extra source directories. Off hand I am not >>>>>>> sure exactly what the correct sequence is. >>>>>>> >>>>>>> - Patrick >>>>>>> >>>>>>> On Tue, Oct 28, 2014 at 7:57 PM, Stephen Boesch <java...@gmail.com> >>>>>>> wrote: >>>>>>>> >>>>>>>> Hi Matei, >>>>>>>> Until my latest pull from upstream/master it had not been necessary >>>>>>>> to >>>>>>>> add the hive profile: is it now?? >>>>>>>> >>>>>>>> I am not using sbt gen-idea. The way to open in intellij has been to >>>>>>>> Open >>>>>>>> the parent directory. IJ recognizes it as a maven project. >>>>>>>> >>>>>>>> There are several steps to do surgery on the yarn-parent / yarn >>>>>>>> projects >>>>>>> >>>>>>> , >>>>>>>> >>>>>>>> then do a full rebuild. That was working until one week ago. >>>>>>>> Intellij/maven is presently broken in two ways: this hive shim >>>>>>>> (which >>>>>>> >>>>>>> may >>>>>>>> >>>>>>>> yet hopefully be a small/simple fix - let us see) and (2) the >>>>>>>> "NoClassDefFoundError >>>>>>>> on ThreadFactoryBuilder" from my prior emails -and which is quite a >>>>>>> >>>>>>> serious >>>>>>>> >>>>>>>> problem . >>>>>>>> >>>>>>>> 2014-10-28 19:46 GMT-07:00 Matei Zaharia <matei.zaha...@gmail.com>: >>>>>>>> >>>>>>>>> Hi Stephen, >>>>>>>>> >>>>>>>>> How did you generate your Maven workspace? You need to make sure the >>>>>>> >>>>>>> Hive >>>>>>>>> >>>>>>>>> profile is enabled for it. For example sbt/sbt -Phive gen-idea. >>>>>>>>> >>>>>>>>> Matei >>>>>>>>> >>>>>>>>>> On Oct 28, 2014, at 7:42 PM, Stephen Boesch <java...@gmail.com> >>>>>>> >>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> I have run on the command line via maven and it is fine: >>>>>>>>>> >>>>>>>>>> mvn -Dscalastyle.failOnViolation=false -DskipTests -Pyarn >>>>>>> >>>>>>> -Phadoop-2.3 >>>>>>>>>> >>>>>>>>>> compile package install >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> But with the latest code Intellij builds do not work. Following is >>>>>>> >>>>>>> one of >>>>>>>>>> >>>>>>>>>> 26 similar errors: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Error:(173, 38) not found: value HiveShim >>>>>>>>>> >>>>>>>>> Option(tableParameters.get(HiveShim.getStatsSetupConstTotalSize)) >>>>>>>>>> >>>>>>>>>> ^ >>>>>>>>> >>>>>>>>> >>>>> >>>> >>> >>> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
