Jingxuan Fu created HIVE-26213:
----------------------------------
Summary: "hive.limit.pushdown.memory.usage" better not be equal to
1.0, otherwise it will raise an error
Key: HIVE-26213
URL: https://issues.apache.org/jira/browse/HIVE-26213
Project: Hive
Issue Type: Bug
Affects Versions: 3.1.2
Environment: Hive 3.1.2
os.name=Linux
os.arch=amd64
os.version=5.4.0-72-generic
java.version=1.8.0_162
java.vendor=Oracle Corporation
Reporter: Jingxuan Fu
Assignee: Jingxuan Fu
In hive-default.xml.template
{code:java}
<property> <name>hive.limit.pushdown.memory.usage</name> <value>0.1</value>
<description> Expects value between 0.0f and 1.0f. The fraction of available
memory to be used for buffering rows in Reducesink operator for limit pushdown
optimization. </description> </property>
{code}
Based on the description of hive-default.xml.template,
hive.limit.pushdown.memory.usage expects a value between 0.0 and 1.0, setting
hive.limit.pushdown.memory.usage to 1.0 means that it expects the available
memory of all buffered lines for the limit pushdown optimization, and
successfully start hiveserver2.
Then, call the java api to write a program to establish a jdbc connection as a
client to access hive, using JDBCDemo as an example.
{code:java}
import demo.utils.JDBCUtils; public class JDBCDemo{ public static void
main(String[] args) throws Exception { JDBCUtils.init();
JDBCUtils.createDatabase(); JDBCUtils.showDatabases();
JDBCUtils.createTable(); JDBCUtils.showTables(); JDBCUtils.descTable();
JDBCUtils.loadData(); JDBCUtils.selectData(); JDBCUtils.countData();
JDBCUtils.dropDatabase(); JDBCUtils.dropTable(); JDBCUtils.destory(); } }
{code}
After running the client program, both the client and the hiveserver throw
exceptions.
{code:java}
2022-05-09 19:05:36: Starting HiveServer2 SLF4J: Class path contains multiple
SLF4J bindings. SLF4J: Found binding in
[jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation. SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory] Hive Session ID =
67a6db8d-f957-4d5d-ac18-28403adab7f3 Hive Session ID =
f9f8772c-5765-4c3e-bcff-ca605c667be7 OK OK OK OK OK OK OK Loading data to table
default.emp OK FAILED: SemanticException Invalid memory usage value 1.0 for
hive.limit.pushdown.memory.usage{code}
{code:java}
liky@ljq1:~/hive_jdbc_test$ ./startJDBC_0.sh SLF4J: Class path contains
multiple SLF4J bindings. SLF4J: Found binding in
[jar:file:/home/liky/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/home/liky/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation. SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory] Running: drop database if exists
hive_jdbc_test Running: create database hive_jdbc_test Running: show databases
default hive_jdbc_test Running: drop table if exists emp Running: create table
emp( empno int, ename string, job string, mgr int, hiredate string, sal double,
comm double, deptno int ) row format delimited fields terminated by '\t'
Running: show tables emp Running: desc emp empno int ename string job string
mgr int hiredate string sal double comm double deptno int Running: load data
local inpath '/home/liky/hiveJDBCTestData/data.txt' overwrite into table emp
Running: select * from emp Exception in thread "main"
org.apache.hive.service.cli.HiveSQLException: Error while compiling statement:
FAILED: SemanticException Invalid memory usage value 1.0 for
hive.limit.pushdown.memory.usage at
org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:380) at
org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:366) at
org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:354) at
org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:293) at
org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:509) at
demo.utils.JDBCUtils.selectData(JDBCUtils.java:98) at
demo.test.JDBCDemo.main(JDBCDemo.java:19){code}
Setting hive.limit.pushdown.memory.usage to 0.0 has no exception.
So, setting hive.limit.pushdown.memory.usage to 1.0 is not desirable,
*hive-default.xml.template is not clear enough for the description of the
boundary of the value, it is better to use the interval to indicate the value
that is [0.0,1.0).*
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
