[
https://issues.apache.org/jira/browse/HIVE-26213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingxuan Fu updated HIVE-26213:
-------------------------------
Description:
In hive-default.xml.template
{code:java}
<property>
<name>hive.limit.pushdown.memory.usage</name>
<value>0.1</value>
<description>
Expects value between 0.0f and 1.0f.
The fraction of available memory to be used for buffering rows in
Reducesink operator for limit pushdown optimization.
</description>
</property>{code}
Based on the description of hive-default.xml.template,
hive.limit.pushdown.memory.usage expects a value between 0.0 and 1.0, setting
hive.limit.pushdown.memory.usage to 1.0 means that it expects the available
memory of all buffered lines for the limit pushdown optimization, and
successfully start hiveserver2.
Then, call the java api to write a program to establish a jdbc connection as a
client to access hive, using JDBCDemo as an example.
{code:java}
import demo.utils.JDBCUtils;
public class JDBCDemo{
public static void main(String[] args) throws Exception
{ JDBCUtils.init(); JDBCUtils.createDatabase();
JDBCUtils.showDatabases(); JDBCUtils.createTable(); JDBCUtils.showTables();
JDBCUtils.descTable(); JDBCUtils.loadData(); JDBCUtils.selectData();
JDBCUtils.countData(); JDBCUtils.dropDatabase(); JDBCUtils.dropTable();
JDBCUtils.destory(); }
}
{code}
After running the client program, both the client and the hiveserver throw
exceptions.
{code:java}
2022-05-09 19:05:36: Starting HiveServer2
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = 67a6db8d-f957-4d5d-ac18-28403adab7f3
Hive Session ID = f9f8772c-5765-4c3e-bcff-ca605c667be7
OK
OK
OK
OK
OK
OK
OK
Loading data to table default.emp
OK
FAILED: SemanticException Invalid memory usage value 1.0 for
hive.limit.pushdown.memory.usage{code}
{code:java}
liky@ljq1:~/hive_jdbc_test$ ./startJDBC_0.sh
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/home/liky/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/home/liky/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Running: drop database if exists hive_jdbc_test
Running: create database hive_jdbc_test
Running: show databases
default
hive_jdbc_test
Running: drop table if exists emp
Running: create table emp(
empno int,
ename string,
job string,
mgr int,
hiredate string,
sal double,
comm double,
deptno int
)
row format delimited fields terminated by '\t'
Running: show tables
emp
Running: desc emp
empno int
ename string
job string
mgr int
hiredate string
sal double
comm double
deptno int
Running: load data local inpath '/home/liky/hiveJDBCTestData/data.txt'
overwrite into table emp
Running: select * from emp
Exception in thread "main" org.apache.hive.service.cli.HiveSQLException: Error
while compiling statement: FAILED: SemanticException Invalid memory usage value
1.0 for hive.limit.pushdown.memory.usage
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:380)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:366)
at
org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:354)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:293)
at org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:509)
at demo.utils.JDBCUtils.selectData(JDBCUtils.java:98)
at demo.test.JDBCDemo.main(JDBCDemo.java:19){code}
Setting hive.limit.pushdown.memory.usage to 0.0 has no exception.
So, setting hive.limit.pushdown.memory.usage to 1.0 is not desirable,
*hive-default.xml.template is not clear enough for the description of the
boundary of the value, it is better to use the interval to indicate the value
that is [0.0,1.0).*
was:
In hive-default.xml.template
{code:java}
<property>
<name>hive.limit.pushdown.memory.usage</name>
<value>0.1</value>
<description>
Expects value between 0.0f and 1.0f.
The fraction of available memory to be used for buffering rows in
Reducesink operator for limit pushdown optimization.
</description>
</property>{code}
Based on the description of hive-default.xml.template,
hive.limit.pushdown.memory.usage expects a value between 0.0 and 1.0, setting
hive.limit.pushdown.memory.usage to 1.0 means that it expects the available
memory of all buffered lines for the limit pushdown optimization, and
successfully start hiveserver2.
Then, call the java api to write a program to establish a jdbc connection as a
client to access hive, using JDBCDemo as an example.
<property>
<name>hive.limit.pushdown.memory.usage</name>
<value>0.1</value>
<description>
Expects value between 0.0f and 1.0f.
The fraction of available memory to be used for buffering rows in
Reducesink operator for limit pushdown optimization.
</description>
</property>
Based on the description of hive-default.xml.template,
hive.limit.pushdown.memory.usage expects a value between 0.0 and 1.0, setting
hive.limit.pushdown.memory.usage to 1.0 means that it expects the available
memory of all buffered lines for the limit pushdown optimization, and
successfully start hiveserver2.
Then, call the java api to write a program to establish a jdbc connection as a
client to access hive, using JDBCDemo as an example.
import demo.utils.JDBCUtils;
public class JDBCDemo{
public static void main(String[] args) throws Exception {
JDBCUtils.init();
JDBCUtils.createDatabase();
JDBCUtils.showDatabases();
JDBCUtils.createTable();
JDBCUtils.showTables();
JDBCUtils.descTable();
JDBCUtils.loadData();
JDBCUtils.selectData();
JDBCUtils.countData();
JDBCUtils.dropDatabase();
JDBCUtils.dropTable();
JDBCUtils.destory();
}
}
After running the client program, both the client and the hiveserver throw
exceptions.
{code:java}
2022-05-09 19:05:36: Starting HiveServer2
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = 67a6db8d-f957-4d5d-ac18-28403adab7f3
Hive Session ID = f9f8772c-5765-4c3e-bcff-ca605c667be7
OK
OK
OK
OK
OK
OK
OK
Loading data to table default.emp
OK
FAILED: SemanticException Invalid memory usage value 1.0 for
hive.limit.pushdown.memory.usage{code}
{code:java}
liky@ljq1:~/hive_jdbc_test$ ./startJDBC_0.sh
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/home/liky/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/home/liky/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Running: drop database if exists hive_jdbc_test
Running: create database hive_jdbc_test
Running: show databases
default
hive_jdbc_test
Running: drop table if exists emp
Running: create table emp(
empno int,
ename string,
job string,
mgr int,
hiredate string,
sal double,
comm double,
deptno int
)
row format delimited fields terminated by '\t'
Running: show tables
emp
Running: desc emp
empno int
ename string
job string
mgr int
hiredate string
sal double
comm double
deptno int
Running: load data local inpath '/home/liky/hiveJDBCTestData/data.txt'
overwrite into table emp
Running: select * from emp
Exception in thread "main" org.apache.hive.service.cli.HiveSQLException: Error
while compiling statement: FAILED: SemanticException Invalid memory usage value
1.0 for hive.limit.pushdown.memory.usage
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:380)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:366)
at
org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:354)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:293)
at org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:509)
at demo.utils.JDBCUtils.selectData(JDBCUtils.java:98)
at demo.test.JDBCDemo.main(JDBCDemo.java:19){code}
Setting hive.limit.pushdown.memory.usage to 0.0 has no exception.
So, setting hive.limit.pushdown.memory.usage to 1.0 is not desirable,
*hive-default.xml.template is not clear enough for the description of the
boundary of the value, it is better to use the interval to indicate the value
that is [0.0,1.0).*
> "hive.limit.pushdown.memory.usage" better not be equal to 1.0, otherwise it
> will raise an error
> -----------------------------------------------------------------------------------------------
>
> Key: HIVE-26213
> URL: https://issues.apache.org/jira/browse/HIVE-26213
> Project: Hive
> Issue Type: Bug
> Affects Versions: 3.1.2
> Environment: Hive 3.1.2
> os.name=Linux
> os.arch=amd64
> os.version=5.4.0-72-generic
> java.version=1.8.0_162
> java.vendor=Oracle Corporation
> Reporter: Jingxuan Fu
> Assignee: Jingxuan Fu
> Priority: Major
>
> In hive-default.xml.template
> {code:java}
> <property>
> <name>hive.limit.pushdown.memory.usage</name>
> <value>0.1</value>
> <description>
> Expects value between 0.0f and 1.0f.
> The fraction of available memory to be used for buffering rows in
> Reducesink operator for limit pushdown optimization.
> </description>
> </property>{code}
> Based on the description of hive-default.xml.template,
> hive.limit.pushdown.memory.usage expects a value between 0.0 and 1.0, setting
> hive.limit.pushdown.memory.usage to 1.0 means that it expects the available
> memory of all buffered lines for the limit pushdown optimization, and
> successfully start hiveserver2.
> Then, call the java api to write a program to establish a jdbc connection as
> a client to access hive, using JDBCDemo as an example.
> {code:java}
> import demo.utils.JDBCUtils;
> public class JDBCDemo{
> public static void main(String[] args) throws Exception
> { JDBCUtils.init(); JDBCUtils.createDatabase();
> JDBCUtils.showDatabases(); JDBCUtils.createTable();
> JDBCUtils.showTables(); JDBCUtils.descTable(); JDBCUtils.loadData();
> JDBCUtils.selectData(); JDBCUtils.countData(); JDBCUtils.dropDatabase();
> JDBCUtils.dropTable(); JDBCUtils.destory(); }
> }
> {code}
> After running the client program, both the client and the hiveserver throw
> exceptions.
> {code:java}
> 2022-05-09 19:05:36: Starting HiveServer2
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Hive Session ID = 67a6db8d-f957-4d5d-ac18-28403adab7f3
> Hive Session ID = f9f8772c-5765-4c3e-bcff-ca605c667be7
> OK
> OK
> OK
> OK
> OK
> OK
> OK
> Loading data to table default.emp
> OK
> FAILED: SemanticException Invalid memory usage value 1.0 for
> hive.limit.pushdown.memory.usage{code}
> {code:java}
> liky@ljq1:~/hive_jdbc_test$ ./startJDBC_0.sh
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/home/liky/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/liky/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Running: drop database if exists hive_jdbc_test
> Running: create database hive_jdbc_test
> Running: show databases
> default
> hive_jdbc_test
> Running: drop table if exists emp
> Running: create table emp(
> empno int,
> ename string,
> job string,
> mgr int,
> hiredate string,
> sal double,
> comm double,
> deptno int
> )
> row format delimited fields terminated by '\t'
> Running: show tables
> emp
> Running: desc emp
> empno int
> ename string
> job string
> mgr int
> hiredate string
> sal double
> comm double
> deptno int
> Running: load data local inpath '/home/liky/hiveJDBCTestData/data.txt'
> overwrite into table emp
> Running: select * from emp
> Exception in thread "main" org.apache.hive.service.cli.HiveSQLException:
> Error while compiling statement: FAILED: SemanticException Invalid memory
> usage value 1.0 for hive.limit.pushdown.memory.usage
> at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:380)
> at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:366)
> at
> org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:354)
> at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:293)
> at
> org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:509)
> at demo.utils.JDBCUtils.selectData(JDBCUtils.java:98)
> at demo.test.JDBCDemo.main(JDBCDemo.java:19){code}
> Setting hive.limit.pushdown.memory.usage to 0.0 has no exception.
> So, setting hive.limit.pushdown.memory.usage to 1.0 is not desirable,
> *hive-default.xml.template is not clear enough for the description of the
> boundary of the value, it is better to use the interval to indicate the value
> that is [0.0,1.0).*
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
