[
https://issues.apache.org/jira/browse/HIVE-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13921377#comment-13921377
]
Hive QA commented on HIVE-6455:
-------------------------------
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12632762/HIVE-6455.11.patch
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 5356 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge4
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_dyn_part
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dyn_part_max_per_node
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input1
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input2
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input3
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input6
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input7
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input9
org.apache.hadoop.hive.ql.parse.TestParse.testParse_sample2
org.apache.hadoop.hive.ql.parse.TestParse.testParse_union
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1632/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1632/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12632762
> Scalable dynamic partitioning and bucketing optimization
> --------------------------------------------------------
>
> Key: HIVE-6455
> URL: https://issues.apache.org/jira/browse/HIVE-6455
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.13.0
> Reporter: Prasanth J
> Assignee: Prasanth J
> Labels: optimization
> Attachments: HIVE-6455.1.patch, HIVE-6455.1.patch,
> HIVE-6455.10.patch, HIVE-6455.10.patch, HIVE-6455.11.patch,
> HIVE-6455.2.patch, HIVE-6455.3.patch, HIVE-6455.4.patch, HIVE-6455.4.patch,
> HIVE-6455.5.patch, HIVE-6455.6.patch, HIVE-6455.7.patch, HIVE-6455.8.patch,
> HIVE-6455.9.patch, HIVE-6455.9.patch
>
>
> The current implementation of dynamic partition works by keeping at least one
> record writer open per dynamic partition directory. In case of bucketing
> there can be multispray file writers which further adds up to the number of
> open record writers. The record writers of column oriented file format (like
> ORC, RCFile etc.) keeps some sort of in-memory buffers (value buffer or
> compression buffers) open all the time to buffer up the rows and compress
> them before flushing it to disk. Since these buffers are maintained per
> column basis the amount of constant memory that will required at runtime
> increases as the number of partitions and number of columns per partition
> increases. This often leads to OutOfMemory (OOM) exception in mappers or
> reducers depending on the number of open record writers. Users often tune the
> JVM heapsize (runtime memory) to get over such OOM issues.
> With this optimization, the dynamic partition columns and bucketing columns
> (in case of bucketed tables) are sorted before being fed to the reducers.
> Since the partitioning and bucketing columns are sorted, each reducers can
> keep only one record writer open at any time thereby reducing the memory
> pressure on the reducers. This optimization is highly scalable as the number
> of partition and number of columns per partition increases at the cost of
> sorting the columns.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
