Abdullah Yousufi created HIVE-14165:
---------------------------------------
Summary: Enable faster S3 Split Computation by listing files in
blocks
Key: HIVE-14165
URL: https://issues.apache.org/jira/browse/HIVE-14165
Project: Hive
Issue Type: Improvement
Affects Versions: 2.1.0
Reporter: Abdullah Yousufi
Assignee: Abdullah Yousufi
During split computation when a large of files are required to be listed from
S3 then instead of executing 1 API call per file, one can optimize by listing
1000 files in each API call. Thereby reducing the amount of time required for
listing files.
Qubole has this optimization in place as detailed here:
https://www.qubole.com/blog/product/optimizing-hadoop-for-s3-part-1/?nabe=5695374637924352:0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
