Currently, hive does not launch map-reduce jobs for the following queries: select * from <T> where <condition on partition columns> (limit <n>)?
This behavior is not configurable, and cannot be altered. HIVE-2925 wants to extend this behavior. The goal is not to spawn map-reduce jobs for the following queries: Select <expr> from <T> where <any condition> (limit <n>)? It is currently controlled by one parameter: hive.aggressive.fetch.task.conversion, based on which it is decided, whether to spawn map-reduce jobs or not for the queries of the above type. Note that this can be beneficial for certain types of queries, since it is avoiding the expensive step of spawning map-reduce. However, it can be pretty expensive for certain types of queries: selecting a very large number of rows, the query having a very selective filter (which is satisfied by a very number of rows, and therefore involves scanning a very large table) etc. The user does not have any control on this. Note that it cannot be done by hooks, since the pre-semantic hooks does not have enough information: type of the query, inputs etc. and it is too late to do anything in the post-semantic hook (the query plan has already been altered). I would like to propose the following configuration parameters to control this behavior. hive.fetch.task.conversion: true, false, auto If the value is true, then all queries with only selects and filters will be converted If the value is false, then no query will be converted If the value is auto (which should be the default behavior), there should be additional parameters to control the semantics. hive.fetch.task.auto.limit.threshold ---> integer value X1 hive.fetch.task.auto.inputsize.threshold ---> integer value X2 If either the query has a limit lower than X1, or the input size is smaller than X2, the queries containing only filters and selects will be converted to not use map-reudce jobs. Comments… -namit
