[
https://issues.apache.org/jira/browse/HIVE-3997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577893#comment-13577893
]
Gopal V commented on HIVE-3997:
-------------------------------
Benchmarks are not promising
SCALE=1 data-set
|| run || client-hash || dist-cache ||
| 1 | 35798 | 32707 |
| 2 | 34911 | 35772 |
SCALE=10 data-set
|| run || client-hash || dist-cache ||
| 1 | 60646 | 80649 |
| 2 | 61591 | 76510 |
The good news is that 10x the amount of data results in time being < 2x.
But the bad news is that the SCALE=10 data has 60 splits in a cluster with 32
map tasks concurrently running, which means that the map-side processing is
slowing down the over-all the operation, having to do it twice on each task.
Spinning up 3 more nodes to bring the cluster count upto 6 to test this out
again.
> Use distributed cache to cache/localize dimension table & filter it in map
> task setup
> -------------------------------------------------------------------------------------
>
> Key: HIVE-3997
> URL: https://issues.apache.org/jira/browse/HIVE-3997
> Project: Hive
> Issue Type: Improvement
> Reporter: Gopal V
> Assignee: Gopal V
>
> The hive clients are not always co-located with the hadoop/hdfs cluster.
> This means that the dimension table filtering, when done on the client side
> becomes very slow. Not only that, the conversion of the small tables into
> hashtables has to be done every single time a query is run with different
> filters on the big table.
> That entire hashtable has to be part of the job, which involves even more
> HDFS writes from the far client side.
> Using the distributed cache also has the advantage that the localized files
> can be kept between jobs instead of firing off an HDFS read for every query.
> Moving the operator pipeline for the hash generation into the map task itself
> has perhaps a few cons.
> The map task might OOM due to this change, but it will take longer to recover
> until all the map attempts fail, instead of being conditional on the client.
> The client has no idea how much memory the hashtable needs and has to rely on
> the disk sizes (compressed sizes, perhaps) to determine if it needs to fall
> back onto a reduce-join instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
