I've heard from Cloudera that there were hive internal changes between 0.12 and 0.13 that required code re-writing. Over time it might be possible for us to integrate with hive using API's that are more stable (this is the domain of Michael/Cheng/Yin more than me!). It would be interesting to see what the Hulu folks did.
- Patrick On Mon, Jul 28, 2014 at 10:16 AM, Cheng Lian <lian.cs....@gmail.com> wrote: > AFAIK, according a recent talk, Hulu team in China has built Spark SQL > against Hive 0.13 (or 0.13.1?) successfully. Basically they also > re-packaged Hive 0.13 as what the Spark team did. The slides of the talk > hasn't been released yet though. > > > On Tue, Jul 29, 2014 at 1:01 AM, Ted Yu <yuzhih...@gmail.com> wrote: > >> Owen helped me find this: >> https://issues.apache.org/jira/browse/HIVE-7423 >> >> I guess this means that for Hive 0.14, Spark should be able to directly >> pull in hive-exec-core.jar >> >> Cheers >> >> >> On Mon, Jul 28, 2014 at 9:55 AM, Patrick Wendell <pwend...@gmail.com> >> wrote: >> >> > It would be great if the hive team can fix that issue. If not, we'll >> > have to continue forking our own version of Hive to change the way it >> > publishes artifacts. >> > >> > - Patrick >> > >> > On Mon, Jul 28, 2014 at 9:34 AM, Ted Yu <yuzhih...@gmail.com> wrote: >> > > Talked with Owen offline. He confirmed that as of 0.13, hive-exec is >> > still >> > > uber jar. >> > > >> > > Right now I am facing the following error building against Hive 0.13.1 >> : >> > > >> > > [ERROR] Failed to execute goal on project spark-hive_2.10: Could not >> > > resolve dependencies for project >> > > org.apache.spark:spark-hive_2.10:jar:1.1.0-SNAPSHOT: The following >> > > artifacts could not be resolved: >> > > org.spark-project.hive:hive-metastore:jar:0.13.1, >> > > org.spark-project.hive:hive-exec:jar:0.13.1, >> > > org.spark-project.hive:hive-serde:jar:0.13.1: Failure to find >> > > org.spark-project.hive:hive-metastore:jar:0.13.1 in >> > > http://repo.maven.apache.org/maven2 was cached in the local >> repository, >> > > resolution will not be reattempted until the update interval of >> > maven-repo >> > > has elapsed or updates are forced -> [Help 1] >> > > >> > > Some hint would be appreciated. >> > > >> > > Cheers >> > > >> > > >> > > On Mon, Jul 28, 2014 at 9:15 AM, Sean Owen <so...@cloudera.com> wrote: >> > > >> > >> Yes, it is published. As of previous versions, at least, hive-exec >> > >> included all of its dependencies *in its artifact*, making it unusable >> > >> as-is because it contained copies of dependencies that clash with >> > >> versions present in other artifacts, and can't be managed with Maven >> > >> mechanisms. >> > >> >> > >> I am not sure why hive-exec was not published normally, with just its >> > >> own classes. That's why it was copied, into an artifact with just >> > >> hive-exec code. >> > >> >> > >> You could do the same thing for hive-exec 0.13.1. >> > >> Or maybe someone knows that it's published more 'normally' now. >> > >> I don't think hive-metastore is related to this question? >> > >> >> > >> I am no expert on the Hive artifacts, just remembering what the issue >> > >> was initially in case it helps you get to a similar solution. >> > >> >> > >> On Mon, Jul 28, 2014 at 4:47 PM, Ted Yu <yuzhih...@gmail.com> wrote: >> > >> > hive-exec (as of 0.13.1) is published here: >> > >> > >> > >> >> > >> http://search.maven.org/#artifactdetails%7Corg.apache.hive%7Chive-exec%7C0.13.1%7Cjar >> > >> > >> > >> > Should a JIRA be opened so that dependency on hive-metastore can be >> > >> > replaced by dependency on hive-exec ? >> > >> > >> > >> > Cheers >> > >> > >> > >> > >> > >> > On Mon, Jul 28, 2014 at 8:26 AM, Sean Owen <so...@cloudera.com> >> > wrote: >> > >> > >> > >> >> The reason for org.spark-project.hive is that Spark relies on >> > >> >> hive-exec, but the Hive project does not publish this artifact by >> > >> >> itself, only with all its dependencies as an uber jar. Maybe that's >> > >> >> been improved. If so, you need to point at the new hive-exec and >> > >> >> perhaps sort out its dependencies manually in your build. >> > >> >> >> > >> >> On Mon, Jul 28, 2014 at 4:01 PM, Ted Yu <yuzhih...@gmail.com> >> wrote: >> > >> >> > I found 0.13.1 artifacts in maven: >> > >> >> > >> > >> >> >> > >> >> > >> http://search.maven.org/#artifactdetails%7Corg.apache.hive%7Chive-metastore%7C0.13.1%7Cjar >> > >> >> > >> > >> >> > However, Spark uses groupId of org.spark-project.hive, not >> > >> >> org.apache.hive >> > >> >> > >> > >> >> > Can someone tell me how it is supposed to work ? >> > >> >> > >> > >> >> > Cheers >> > >> >> > >> > >> >> > >> > >> >> > On Mon, Jul 28, 2014 at 7:44 AM, Steve Nunez < >> > snu...@hortonworks.com> >> > >> >> wrote: >> > >> >> > >> > >> >> >> I saw a note earlier, perhaps on the user list, that at least >> one >> > >> >> person is >> > >> >> >> using Hive 0.13. Anyone got a working build configuration for >> this >> > >> >> version >> > >> >> >> of Hive? >> > >> >> >> >> > >> >> >> Regards, >> > >> >> >> - Steve >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> -- >> > >> >> >> CONFIDENTIALITY NOTICE >> > >> >> >> NOTICE: This message is intended for the use of the individual >> or >> > >> >> entity to >> > >> >> >> which it is addressed and may contain information that is >> > >> confidential, >> > >> >> >> privileged and exempt from disclosure under applicable law. If >> the >> > >> >> reader >> > >> >> >> of this message is not the intended recipient, you are hereby >> > >> notified >> > >> >> that >> > >> >> >> any printing, copying, dissemination, distribution, disclosure >> or >> > >> >> >> forwarding of this communication is strictly prohibited. If you >> > have >> > >> >> >> received this communication in error, please contact the sender >> > >> >> immediately >> > >> >> >> and delete it from your system. Thank You. >> > >> >> >> >> > >> >> >> > >> >> > >>
