So, do we have a short-term fix until Hive 0.14 comes out? Perhaps adding the hive-exec jar to the spark-project repo? It doesn¹t look like there¹s a release date schedule for 0.14.
On 7/28/14, 10:50, "Cheng Lian" <lian.cs....@gmail.com> wrote: >Exactly, forgot to mention Hulu team also made changes to cope with those >incompatibility issues, but they said that¹s relatively easy once the >re-packaging work is done. > > >On Tue, Jul 29, 2014 at 1:20 AM, Patrick Wendell <pwend...@gmail.com> >wrote: > >> I've heard from Cloudera that there were hive internal changes between >> 0.12 and 0.13 that required code re-writing. Over time it might be >> possible for us to integrate with hive using API's that are more >> stable (this is the domain of Michael/Cheng/Yin more than me!). It >> would be interesting to see what the Hulu folks did. >> >> - Patrick >> >> On Mon, Jul 28, 2014 at 10:16 AM, Cheng Lian <lian.cs....@gmail.com> >> wrote: >> > AFAIK, according a recent talk, Hulu team in China has built Spark SQL >> > against Hive 0.13 (or 0.13.1?) successfully. Basically they also >> > re-packaged Hive 0.13 as what the Spark team did. The slides of the >>talk >> > hasn't been released yet though. >> > >> > >> > On Tue, Jul 29, 2014 at 1:01 AM, Ted Yu <yuzhih...@gmail.com> wrote: >> > >> >> Owen helped me find this: >> >> https://issues.apache.org/jira/browse/HIVE-7423 >> >> >> >> I guess this means that for Hive 0.14, Spark should be able to >>directly >> >> pull in hive-exec-core.jar >> >> >> >> Cheers >> >> >> >> >> >> On Mon, Jul 28, 2014 at 9:55 AM, Patrick Wendell <pwend...@gmail.com> >> >> wrote: >> >> >> >> > It would be great if the hive team can fix that issue. If not, >>we'll >> >> > have to continue forking our own version of Hive to change the way >>it >> >> > publishes artifacts. >> >> > >> >> > - Patrick >> >> > >> >> > On Mon, Jul 28, 2014 at 9:34 AM, Ted Yu <yuzhih...@gmail.com> >>wrote: >> >> > > Talked with Owen offline. He confirmed that as of 0.13, >>hive-exec is >> >> > still >> >> > > uber jar. >> >> > > >> >> > > Right now I am facing the following error building against Hive >> 0.13.1 >> >> : >> >> > > >> >> > > [ERROR] Failed to execute goal on project spark-hive_2.10: Could >>not >> >> > > resolve dependencies for project >> >> > > org.apache.spark:spark-hive_2.10:jar:1.1.0-SNAPSHOT: The >>following >> >> > > artifacts could not be resolved: >> >> > > org.spark-project.hive:hive-metastore:jar:0.13.1, >> >> > > org.spark-project.hive:hive-exec:jar:0.13.1, >> >> > > org.spark-project.hive:hive-serde:jar:0.13.1: Failure to find >> >> > > org.spark-project.hive:hive-metastore:jar:0.13.1 in >> >> > > http://repo.maven.apache.org/maven2 was cached in the local >> >> repository, >> >> > > resolution will not be reattempted until the update interval of >> >> > maven-repo >> >> > > has elapsed or updates are forced -> [Help 1] >> >> > > >> >> > > Some hint would be appreciated. >> >> > > >> >> > > Cheers >> >> > > >> >> > > >> >> > > On Mon, Jul 28, 2014 at 9:15 AM, Sean Owen <so...@cloudera.com> >> wrote: >> >> > > >> >> > >> Yes, it is published. As of previous versions, at least, >>hive-exec >> >> > >> included all of its dependencies *in its artifact*, making it >> unusable >> >> > >> as-is because it contained copies of dependencies that clash >>with >> >> > >> versions present in other artifacts, and can't be managed with >> Maven >> >> > >> mechanisms. >> >> > >> >> >> > >> I am not sure why hive-exec was not published normally, with >>just >> its >> >> > >> own classes. That's why it was copied, into an artifact with >>just >> >> > >> hive-exec code. >> >> > >> >> >> > >> You could do the same thing for hive-exec 0.13.1. >> >> > >> Or maybe someone knows that it's published more 'normally' now. >> >> > >> I don't think hive-metastore is related to this question? >> >> > >> >> >> > >> I am no expert on the Hive artifacts, just remembering what the >> issue >> >> > >> was initially in case it helps you get to a similar solution. >> >> > >> >> >> > >> On Mon, Jul 28, 2014 at 4:47 PM, Ted Yu <yuzhih...@gmail.com> >> wrote: >> >> > >> > hive-exec (as of 0.13.1) is published here: >> >> > >> > >> >> > >> >> >> > >> >> >> >>http://search.maven.org/#artifactdetails%7Corg.apache.hive%7Chive-exec%7C >>0.13.1%7Cjar >> >> > >> > >> >> > >> > Should a JIRA be opened so that dependency on hive-metastore >>can >> be >> >> > >> > replaced by dependency on hive-exec ? >> >> > >> > >> >> > >> > Cheers >> >> > >> > >> >> > >> > >> >> > >> > On Mon, Jul 28, 2014 at 8:26 AM, Sean Owen >><so...@cloudera.com> >> >> > wrote: >> >> > >> > >> >> > >> >> The reason for org.spark-project.hive is that Spark relies on >> >> > >> >> hive-exec, but the Hive project does not publish this >>artifact >> by >> >> > >> >> itself, only with all its dependencies as an uber jar. Maybe >> that's >> >> > >> >> been improved. If so, you need to point at the new hive-exec >>and >> >> > >> >> perhaps sort out its dependencies manually in your build. >> >> > >> >> >> >> > >> >> On Mon, Jul 28, 2014 at 4:01 PM, Ted Yu <yuzhih...@gmail.com> >> >> wrote: >> >> > >> >> > I found 0.13.1 artifacts in maven: >> >> > >> >> > >> >> > >> >> >> >> > >> >> >> > >> >> >> >>http://search.maven.org/#artifactdetails%7Corg.apache.hive%7Chive-metasto >>re%7C0.13.1%7Cjar >> >> > >> >> > >> >> > >> >> > However, Spark uses groupId of org.spark-project.hive, not >> >> > >> >> org.apache.hive >> >> > >> >> > >> >> > >> >> > Can someone tell me how it is supposed to work ? >> >> > >> >> > >> >> > >> >> > Cheers >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > On Mon, Jul 28, 2014 at 7:44 AM, Steve Nunez < >> >> > snu...@hortonworks.com> >> >> > >> >> wrote: >> >> > >> >> > >> >> > >> >> >> I saw a note earlier, perhaps on the user list, that at >>least >> >> one >> >> > >> >> person is >> >> > >> >> >> using Hive 0.13. Anyone got a working build configuration >>for >> >> this >> >> > >> >> version >> >> > >> >> >> of Hive? >> >> > >> >> >> >> >> > >> >> >> Regards, >> >> > >> >> >> - Steve >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> -- >> >> > >> >> >> CONFIDENTIALITY NOTICE >> >> > >> >> >> NOTICE: This message is intended for the use of the >> individual >> >> or >> >> > >> >> entity to >> >> > >> >> >> which it is addressed and may contain information that is >> >> > >> confidential, >> >> > >> >> >> privileged and exempt from disclosure under applicable >>law. >> If >> >> the >> >> > >> >> reader >> >> > >> >> >> of this message is not the intended recipient, you are >>hereby >> >> > >> notified >> >> > >> >> that >> >> > >> >> >> any printing, copying, dissemination, distribution, >> disclosure >> >> or >> >> > >> >> >> forwarding of this communication is strictly prohibited. >>If >> you >> >> > have >> >> > >> >> >> received this communication in error, please contact the >> sender >> >> > >> >> immediately >> >> > >> >> >> and delete it from your system. Thank You. >> >> > >> >> >> >> >> > >> >> >> >> > >> >> >> > >> >> >> -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
