[
https://issues.apache.org/jira/browse/HIVE-26213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingxuan Fu updated HIVE-26213:
-------------------------------
Issue Type: Improvement (was: Bug)
> "hive.limit.pushdown.memory.usage" better not be equal to 1.0, otherwise it
> will raise an error
> -----------------------------------------------------------------------------------------------
>
> Key: HIVE-26213
> URL: https://issues.apache.org/jira/browse/HIVE-26213
> Project: Hive
> Issue Type: Improvement
> Affects Versions: 3.1.2
> Environment: Hive 3.1.2
> os.name=Linux
> os.arch=amd64
> os.version=5.4.0-72-generic
> java.version=1.8.0_162
> java.vendor=Oracle Corporation
> Reporter: Jingxuan Fu
> Assignee: Jingxuan Fu
> Priority: Major
>
> In hive-default.xml.template
> {code:java}
> <property>
> <name>hive.limit.pushdown.memory.usage</name>
> <value>0.1</value>
> <description>
> Expects value between 0.0f and 1.0f.
> The fraction of available memory to be used for buffering rows in
> Reducesink operator for limit pushdown optimization.
> </description>
> </property>{code}
> Based on the description of hive-default.xml.template,
> hive.limit.pushdown.memory.usage expects a value between 0.0 and 1.0, setting
> hive.limit.pushdown.memory.usage to 1.0 means that it expects the available
> memory of all buffered lines for the limit pushdown optimization, and
> successfully start hiveserver2.
> Then, call the java api to write a program to establish a jdbc connection as
> a client to access hive, using JDBCDemo as an example.
> {code:java}
> import demo.utils.JDBCUtils;
> public class JDBCDemo{
> public static void main(String[] args) throws Exception
> { JDBCUtils.init(); JDBCUtils.createDatabase();
> JDBCUtils.showDatabases(); JDBCUtils.createTable();
> JDBCUtils.showTables(); JDBCUtils.descTable(); JDBCUtils.loadData();
> JDBCUtils.selectData(); JDBCUtils.countData(); JDBCUtils.dropDatabase();
> JDBCUtils.dropTable(); JDBCUtils.destory(); }
> }
> {code}
> After running the client program, both the client and the hiveserver throw
> exceptions.
> {code:java}
> 2022-05-09 19:05:36: Starting HiveServer2
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Hive Session ID = 67a6db8d-f957-4d5d-ac18-28403adab7f3
> Hive Session ID = f9f8772c-5765-4c3e-bcff-ca605c667be7
> OK
> OK
> OK
> OK
> OK
> OK
> OK
> Loading data to table default.emp
> OK
> FAILED: SemanticException Invalid memory usage value 1.0 for
> hive.limit.pushdown.memory.usage{code}
> {code:java}
> liky@ljq1:~/hive_jdbc_test$ ./startJDBC_0.sh
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/home/liky/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/liky/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Running: drop database if exists hive_jdbc_test
> Running: create database hive_jdbc_test
> Running: show databases
> default
> hive_jdbc_test
> Running: drop table if exists emp
> Running: create table emp(
> empno int,
> ename string,
> job string,
> mgr int,
> hiredate string,
> sal double,
> comm double,
> deptno int
> )
> row format delimited fields terminated by '\t'
> Running: show tables
> emp
> Running: desc emp
> empno int
> ename string
> job string
> mgr int
> hiredate string
> sal double
> comm double
> deptno int
> Running: load data local inpath '/home/liky/hiveJDBCTestData/data.txt'
> overwrite into table emp
> Running: select * from emp
> Exception in thread "main" org.apache.hive.service.cli.HiveSQLException:
> Error while compiling statement: FAILED: SemanticException Invalid memory
> usage value 1.0 for hive.limit.pushdown.memory.usage
> at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:380)
> at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:366)
> at
> org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:354)
> at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:293)
> at
> org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:509)
> at demo.utils.JDBCUtils.selectData(JDBCUtils.java:98)
> at demo.test.JDBCDemo.main(JDBCDemo.java:19){code}
> Setting hive.limit.pushdown.memory.usage to 0.0 has no exception.
> So, setting hive.limit.pushdown.memory.usage to 1.0 is not desirable,
> *hive-default.xml.template is not clear enough for the description of the
> boundary of the value, it is better to use the interval to indicate the value
> that is [0.0,1.0).*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
