[
https://issues.apache.org/jira/browse/HIVE-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933208#action_12933208
]
Arun C Murthy commented on HIVE-1107:
-------------------------------------
{quote}
Adaptive query optimization is indeed a noble goal. Oozie seems to think at the
level of workflow rather than dataflow, so as you say, it may not be an
appropriate layer for performing these optimizations. I'm not sure if it
detracts from the ability of Hive or Pig to perform adaptive query optimization
though, either.
Anyways, thanks for the discussion. We're certainly thinking through these
issues as well.
{quote}
Yep, this is a fun discussion, thanks to you too.
A simple example:
Hive/Pig might want to change the #reduces of job3 in the pipeline after
looking at the output of job1 and job2 - this is not necessarily changing the
DAG itself, but changing the components of the DAG.
As you point out, Oozie is at the level of workflow, not dataflow, and thus
might be cumbersome to deal with for these purposes. Sure, you could support
this Oozie, but I'm not sure it is the right thing to do.
----
In some way, my sense is that we need a common DAG-execution library for Pig
and Hive, not a DAG-execution framework. Thoughts?
> Generic parallel execution framework for Hive (and Pig, and ...)
> ----------------------------------------------------------------
>
> Key: HIVE-1107
> URL: https://issues.apache.org/jira/browse/HIVE-1107
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Reporter: Carl Steinbach
>
> Pig and Hive each have their own libraries for handling plan execution. As we
> prepare to invest more time improving Hive's plan execution mechanism we
> should also start to consider ways of building a generic plan execution
> mechanism that is capable of supporting the needs of Hive and Pig, as well as
> other Hadoop data flow programming environments.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
