[
https://issues.apache.org/jira/browse/HIVE-24994?focusedWorklogId=579363&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-579363
]
ASF GitHub Bot logged work on HIVE-24994:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 08/Apr/21 17:05
Start Date: 08/Apr/21 17:05
Worklog Time Spent: 10m
Work Description: rajkrrsingh opened a new pull request #2162:
URL: https://github.com/apache/hive/pull/2162
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA:
https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g.,
'[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a
faster review.
-->
### What changes were proposed in this pull request?
enabling the batching for direct sql if the db type is Postgres
### Why are the changes needed?
without batching the direct sql is hitting the postgres upper limit where it
can not expect the no of parts in 'IN CLAUSE' more than 32767.
### Does this PR introduce _any_ user-facing change?
NO
### How was this patch tested?
Locally
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 579363)
Remaining Estimate: 0h
Time Spent: 10m
> get_aggr_stats_for call fail with "Tried to send an out-of-range integer"
> -------------------------------------------------------------------------
>
> Key: HIVE-24994
> URL: https://issues.apache.org/jira/browse/HIVE-24994
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Reporter: Rajkumar Singh
> Assignee: Rajkumar Singh
> Priority: Major
> Fix For: 4.0.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> aggrColStatsForPartitions call fail with the Postgres LIMIT if the no of
> partitions passed in the direct sql goes beyond the 32767
> {code:java}
> postgresql.util.PSQLException: An I/O error occurred while sending to the
> backend.
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:337)
> ~[postgresql-42.2.8.jar:42.2.8]
> at
> org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:446)
> ~[postgresql-42.2.8.jar:42.2.8]
> at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:370)
> ~[postgresql-42.2.8.jar:42.2.8]
> at
> org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:149)
> ~[postgresql-42.2.8.jar:42.2.8]
> at
> org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:108)
> ~[postgresql-42.2.8.jar:42.2.8]
> at
> com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
> ~[HikariCP-2.6.1.jar:?]
> at
> com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
> [HikariCP-2.6.1.jar:?]
> at
> org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeQuery(ParamLoggingPreparedStatement.java:375)
> [datanucleus-rdbms-4.1.19.jar:?]
> at
> org.datanucleus.store.rdbms.SQLController.executeStatementQuery(SQLController.java:552)
> [datanucleus-rdbms-4.1.19.jar:?]
> at
> org.datanucleus.store.rdbms.query.SQLQuery.performExecute(SQLQuery.java:645)
> [datanucleus-rdbms-4.1.19.jar:?]
> at
> org.datanucleus.store.query.Query.executeQuery(Query.java:1855)
> [datanucleus-core-4.1.17.jar:?]
> at
> org.datanucleus.store.rdbms.query.SQLQuery.executeWithArray(SQLQuery.java:807)
> [datanucleus-rdbms-4.1.19.jar:?]
> at
> org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:368)
> [datanucleus-api-jdo-4.2.4.jar:?]
> at
> org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:267)
> [datanucleus-api-jdo-4.2.4.jar:?]
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:2058)
> [hive-exec-3.1.0.3.1.5.6019-4.jar:3.1.0.3.1.5.6019-4]
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:2050)
> [hive-exec-3.1.0.3.1.5.6019-4.jar:3.1.0.3.1.5.6019-4]
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.access$1500(MetaStoreDirectSql.java:110)
> [hive-exec-3.1.0.3.1.5.6019-4.jar:3.1.0.3.1.5.6019-4]
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql$15$1.run(MetaStoreDirectSql.java:1530)
> [hive-exec-3.1.0.3.1.5.6019-4.jar:3.1.0.3.1.5.6019-4]
> at
> org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:73)
> [hive-exec-3.1.0.3.1.5.6019-4.jar:3.1.0.3.1.5.6019-4]
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql$15.run(MetaStoreDirectSql.java:1521)
> [hive-exec-3.1.0.3.1.5.6019-4.jar:3.1.0.3.1.5.6019-4]
> at
> org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:73)
> [hive-exec-3.1.0.3.1.5.6019-4.jar:3.1.0.3.1.5.6019-4]
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.partsFoundForPartitions(MetaStoreDirectSql.java:1518)
> [hive-exec-3.1.0.3.1.5.6019-4.jar:3.1.0.3.1.5.6019-4]
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.aggrColStatsForPartitions(MetaStoreDirectSql.java:1489)
> [hive-exec-3.1.0.3.1.5.6019-4.jar:3.1.0.3.1.5.6019-4]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$20.getSqlResult(ObjectStore.java:8966)
> [hive-exec-3.1.0.3.1.5.6019-4.jar:3.1.0.3.1.5.6019-4]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$20.getSqlResult(ObjectStore.java:8962)
> [hive-exec-3.1.0.3.1.5.6019-4.jar:3.1.0.3.1.5.6019-4]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3757)
> [hive-exec-3.1.0.3.1.5.6019-4.jar:3.1.0.3.1.5.6019-4]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.get_aggr_stats_for(ObjectStore.java:8981)
> [hive-exec-3.1.0.3.1.5.6019-4.jar:3.1.0.3.1.5.6019-4]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.get_aggr_stats_for(ObjectStore.java:8951)
> [hive-exec-3.1.0.3.1.5.6019-4.jar:3.1.0.3.1.5.6019-4]
> at sun.reflect.GeneratedMethodAccessor65.invoke(Unknown
> Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_261]
> at java.lang.reflect.Method.invoke(Method.java:498)
> ~[?:1.8.0_261]
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> [hive-exec-3.1.0.3.1.5.6019-4.jar:3.1.0.3.1.5.6019-4]
> at com.sun.proxy.$Proxy37.get_aggr_stats_for(Unknown Source)
> [?:?]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_aggr_stats_for(HiveMetaStore.java:8254)
> [hive-exec-3.1.0.3.1.5.6019-4.jar:3.1.0.3.1.5.6019-4]
> at sun.reflect.GeneratedMethodAccessor64
> Caused by: java.io.IOException: Tried to send an out-of-range
> integer as a 2-byte value: 83666
> at org.postgresql.core.PGStream.sendInteger2(PGStream.java:266)
> ~[postgresql-42.2.8.jar:42.2.8]
> at
> org.postgresql.core.v3.QueryExecutorImpl.sendParse(QueryExecutorImpl.java:1492)
> ~[postgresql-42.2.8.jar:42.2.8]
> at
> org.postgresql.core.v3.QueryExecutorImpl.sendOneQuery(QueryExecutorImpl.java:1815)
> ~[postgresql-42.2.8.jar:42.2.8]
> at
> org.postgresql.core.v3.QueryExecutorImpl.sendQuery(QueryExecutorImpl.java:1378)
> ~[postgresql-42.2.8.jar:42.2.8]
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:302)
> ~[postgresql-42.2.8.jar:42.2.8]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
