[
https://issues.apache.org/jira/browse/HIVE-28955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis resolved HIVE-28955.
----------------------------------------
Fix Version/s: 4.1.0
Resolution: Fixed
Fixed in
https://github.com/apache/hive/commit/d895f0d487b41ba54aee6726034673a16f98b725
> Tests using HiveConfForTest and Tez engine consume large amounts of disk space
> ------------------------------------------------------------------------------
>
> Key: HIVE-28955
> URL: https://issues.apache.org/jira/browse/HIVE-28955
> Project: Hive
> Issue Type: Test
> Reporter: Stamatis Zampetakis
> Assignee: Stamatis Zampetakis
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.0
>
>
> Some tests that use HiveConfForTest and Tez as the underlying engine consume
> large amounts of disk space (in the order of GBs) when run.
> One vivid example is TestTxnCommandsWithSplitUpdateAndVectorization which
> creates files and directories under ql/target directory summing up to ~5GB of
> data.
> {code:bash}
> mvn clean test -Dtest=TestTxnCommandsWithSplitUpdateAndVectorization
> cd target
> du -csh *TestTxn*
> {code}
> {noformat}
> 87M
> org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization-1747139827364
> 87M
> org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization-1747139840720
> 87M
> org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization-1747139843358
> 87M
> ...
> 87M
> org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization-1747140109446
> 4.8G total
> {noformat}
> All of the above contain a localized hive-exec.jar that ends up in a
> different directory since each instance of HiveConfForTest [passes a unique
> value for
> "hive.user.install.directory"|https://github.com/apache/hive/blob/8324ae424a69c32fe8fd32d15f11f31ea635198a/common/src/test/org/apache/hadoop/hive/conf/HiveConfForTest.java#L53].
> The problem becomes worse taking into account that many other tests are
> affected which leads to lots of wasted disk space.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
