Thanks for replying namit.. It is motivating to receive a mail from the authors of Hive :).
I filed the jira based on the discussion.. https://issues.apache.org/jira/browse/HIVE-1938 I will try to update my idea asap. Thanks Bharath,V 4th year Undergrad,IIIT Hyderabad. w: http://research.iiit.ac.in/~bharath.v On Tue, Feb 1, 2011 at 11:46 AM, Namit Jain <nj...@fb.com> wrote: > Bharath, > > This would be great. > > Why don¹t you write up something about how you are planning to proceed ? > File a new jira and load some design notes/spec. there. > We can definitely sync up. from there. > > > This feature would be very useful to the community - We, at facebook, > Would definitely like to use it. > > > Thanks, > -namit > > > On 1/31/11 9:50 PM, "bharath vissapragada" > <bharathvissapragada1...@gmail.com> wrote: > >>Hi Ning,Anja, >> >>I am doing my Masters thesis on this topic . I have implemented all >>SQL features like joins , selects etc on top of Hadoop (before knowing >>about Hive) and we have derived some basic cost-models for join >>re-ordering which seem to be working fine on some basic scales of TPCH >>datasets .. Later I came to know about Hive and I am trying to >>implement the same in Hive . >> >>Right now I am in the process of understanding Hive's source and I am >>almost done with "ql" package. I think it would be great if you guys >>can help us in this regard .. I am a bit confused about the >>implementation of joins and once i'm done with that , I can modify the >>"joinReorder" of Optimizer package by using the cost-formulae and >>metadata. It would be a great opportunity to work with you guys at fb >>and contribute to Hive.. >> >>Thanks >>Bharath,V >>4th year Undergrad,IIIT Hyderabad. >>w: http://research.iiit.ac.in/~bharath.v >> >>On Tue, Feb 1, 2011 at 9:22 AM, Ning Zhang <nzh...@fb.com> wrote: >>> Hi Anja, >>> >>> As you noticed Hive only have limited supports for cost-baesd >>>optimization. One of the reasons is that Hive used to have very small >>>number of optional execution plans to choose from. One exception is >>>mapjoin vs common joins. Liying Tang had some work on his last intern to >>>convert common joins to mapjoin in a rule-based fashion. One of his >>>future works is to automatically convert common join to mapjoins based >>>on stats. There are also ongoing work on indexes on Hive. With the >>>support of indexes, CBO will be much needed. >>> >>> In order for a decent CBO to work, we need stats and cost models. There >>>are some work in stats. Table/partition level stats has already been >>>supported. There is a JIRA open for column level stats (HIVE-1362). Cost >>>model is much more complex in Hadoop environment and closely dependent >>>on the mapjoin/index implementations. Given al these in place, we can >>>then talk about plan enumeration etc. >>> >>> So yes, we are interested in CBO, but it is a large area and many >>>missing pieces need to be filled in Hive. If you have particular >>>interest in some area, you can propose your ideas in >>>hive-...@hive.apache.org mailing list or even apply for an intern at FB >>>if you would like to work closely with us. >>> >>> Thanks, >>> Ning >>> >>> On Jan 31, 2011, at 2:04 PM, Anja Gruenheid wrote: >>> >>>> Hi! >>>> >>>> I'm a graduate student from Georgia Tech and I'm working with Hive for >>>>a research project. I am interested in query optimization and the Hive >>>>MetaStore in that context. Working through the documentation and code, >>>>I noticed that the implementation right now is using a rule-based >>>>optimization system. Therefore, I was wondering whether cost-based >>>>query optimization will be a future task in the development of Hive and >>>>if it would be possible for me to cooperate with the developers of Hive >>>>to advance the project in general. >>>> >>>> Best regards, >>>> Anja Gruenheid >>> >>> > >
