[jira] [Created] (HIVE-25382) "No space left on device" error when running show database
Feroze Daud created HIVE-25382:
--
Summary: "No space left on device" error when running show database
Key: HIVE-25382
URL: https://issues.apache.org/jira/browse/HIVE-25382
Project: Hive
Issue Type: Bug
Components: Clients
Affects Versions: 2.3.2
Reporter: Feroze Daud
We are using Hive provided by Amazon EMR.
When executing "show database" command, we are getting a "No space left on
device" exception.
{noformat}
Error running query: show databases like 'my_database' ->
java.sql.SQLException: [Amazon][HiveJDBCDriver](500051) ERROR processing
query/statement. Error Code: 1, SQL state:
org.apache.hive.service.cli.HiveSQLException: Error while processing statement:
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. java.io.IOException: No space left on
device
at
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:257)
at
org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91)
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:348)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:363)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on
device
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:265)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:74)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:108)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:414)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:74)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:108)
at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:251)
at org.apache.hadoop.io.IOUtils.closeStream(IOUtils.java:268)
at
org.apache.hadoop.hive.ql.exec.DDLTask.showDatabases(DDLTask.java:2515)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:439)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2183)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1839)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1526)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1232)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:255)
... 11 more
Caused by: java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:326)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:263)
... 31 more
, Query: show databases like 'my_database'. Query: show databases like
'my_database' Parameters: [] {noformat}
As mentioned we use Amazon EMR, and it comes with it's own Amazon HiveJdbc
client.
I am not sure if this error is happening on the client or the server. I looked
at the client and dont see any disk space issues.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-25383) Make TestMarkPartitionRemote more stable
Zhihua Deng created HIVE-25383:
--
Summary: Make TestMarkPartitionRemote more stable
Key: HIVE-25383
URL: https://issues.apache.org/jira/browse/HIVE-25383
Project: Hive
Issue Type: Test
Components: Standalone Metastore
Reporter: Zhihua Deng
Sometimes the TestMarkPartitionRemote failed by
{noformat}
org.apache.hadoop.hive.metastore.api.MetaException: Exception determining
external table location:Default location is not available for table:
file:/path/to/tableat
org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transformCreateTable(MetastoreDefaultTransformer.java:660)
~[classes/:?]at
org.apache.hadoop.hive.metastore.HMSHandler.create_table_core(HMSHandler.java:2325)
~[classes/:?]at
org.apache.hadoop.hive.metastore.HMSHandler.create_table_req(HMSHandler.java:2578)
[classes/:?]{noformat}
[http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-2441/15/tests]
[http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-2473/3/tests]
The cause is that the table path is existed before the test executed,
TableLocationStrategy with prohibit does not allow alternate locations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-25384) Bump ORC to 1.6.9
Dongjoon Hyun created HIVE-25384: Summary: Bump ORC to 1.6.9 Key: HIVE-25384 URL: https://issues.apache.org/jira/browse/HIVE-25384 Project: Hive Issue Type: Bug Reporter: Dongjoon Hyun ORC-804 affects ORC 1.6.0 ~ 1.6.8. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-25385) Prevent Hive Server 2 process failures when InterruptedException encountered
Matt McCline created HIVE-25385: --- Summary: Prevent Hive Server 2 process failures when InterruptedException encountered Key: HIVE-25385 URL: https://issues.apache.org/jira/browse/HIVE-25385 Project: Hive Issue Type: Bug Reporter: Matt McCline Assignee: Matt McCline To prevent Hive Server 2 process failure, wrap InterruptedException with another Exception like MetaException, HiveSQLException, etc. Otherwise, InterruptedException rises to Thread.run and kills the process. Example of problem stack trace: java.lang.reflect.UndeclaredThrowableExceptionjava.lang.reflect.UndeclaredThrowableException at com.sun.proxy.$Proxy44.heartbeat(Unknown Source) at sun.reflect.GeneratedMethodAccessor127.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2990) at com.sun.proxy.$Proxy44.heartbeat(Unknown Source) at org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.heartbeat(DbTxnManager.java:622) at org.apache.hadoop.hive.ql.lockmgr.DbTxnManager$Heartbeater.lambda$run$0(DbTxnManager.java:999) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.hadoop.hive.ql.lockmgr.DbTxnManager$Heartbeater.run(DbTxnManager.java:998) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)Caused by: java.lang.InterruptedException: sleep interrupted at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:259) ... 19 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-25386) hive-storage-api should not have guava compile dependency
Dongjoon Hyun created HIVE-25386: Summary: hive-storage-api should not have guava compile dependency Key: HIVE-25386 URL: https://issues.apache.org/jira/browse/HIVE-25386 Project: Hive Issue Type: Bug Components: storage-api Affects Versions: 4.0.0 Reporter: Dongjoon Hyun https://mvnrepository.com/artifact/org.apache.hive/hive-storage-api/2.8.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [VOTE] Should we release Hive Storage API 2.8.0-rc0 ?
Hi, All. It turns out that 2.8.0 has a regression in terms of the dependency although it passed all tests. At Apache Hive 2.2.0, HIVE-13906 removed the Guava compile dependency. However, recently, HIVE-24542 converts the test dependency to the compile dependency. https://github.com/apache/hive/pull/1784/files#diff-e5e45b8481f3d87f4fa561d45b22fa20bb5084e5240d825717e6bde511b733baL122 I noticed this breaking dependency change during Apache ORC integration stage. https://github.com/apache/orc/pull/773 For now, I filed a JIRA issue and made a PR. Please comment your opinion on the Hive and ORC PRs. - HIVE-25386: hive-storage-api should not have guava compile dependency - https://github.com/apache/hive/pull/2531 Bests, Dongjoon. On 2021/07/23 16:40:39, "Owen O'Malley" wrote: > I can do the publishing. Thank you all! > > .. Owen > > On Fri, Jul 23, 2021 at 10:16 AM Panos Garefalakis > wrote: > > > Thank you all for your help! With 3 binding votes this release passed! > > I am going to send a vote result follow up but will need some PMC help > > creating the new Storage API version (on Jira) and releasing (upload > > artifacts)! > > > > Cheers, > > Panagiotis > > > > On Fri, Jul 23, 2021 at 2:21 AM Chao Sun wrote: > > > > > Thanks Owen! I just verified the checksum and gpg signature and they both > > > look good, so +1 too. > > > > > > Panos: please fix the "Fix version" of the JIRA when you get a chance. > > > Thanks. > > > > > > Best, > > > Chao > > > > > > > > > On Thu, Jul 22, 2021 at 2:29 PM Owen O'Malley > > > wrote: > > > > > >> Chao, > > >>Panos key doesn't seem to have propagated to the Apache servers. It > > >> referenced here: > > >> > > >> https://people.apache.org/keys/committer/ as "pgaref > > >> 7DFAB216AB7D96B3B2072184DC11DE4D00F8FA1D" > > >> > > >> The key itself can be found here: > > >> > > >> > > >> > > https://keyserver.ubuntu.com/pks/lookup?search=pgaref&fingerprint=on&op=index > > >> > > >> > > >> On Wed, Jul 21, 2021 at 8:47 PM Chao Sun wrote: > > >> > > >> > I built the source from the branch and ran the tests, which all > > passed. > > >> > However I was not able to find the public GPG key. Panos: could you > > >> point > > >> > me to the location? > > >> > > > >> > Also seems we should create a new version 2.8.0 in the JIRA page: > > >> > > > >> > > > >> > > https://issues.apache.org/jira/projects/HIVE?selectedItem=com.atlassian.jira.jira-projects-plugin:release-page > > >> > and update "Fix version" of > > >> > https://issues.apache.org/jira/browse/HIVE-24458 > > >> > . > > >> > > > >> > Chao > > >> > > > >> > On Wed, Jul 21, 2021 at 9:10 AM Szehon Ho > > >> wrote: > > >> > > > >> > > +1 (binding) > > >> > > > > >> > > * Built module > > >> > > * Ran tests > > >> > > * Checked artifact checksum and signature > > >> > > > > >> > > Thanks > > >> > > Szehon > > >> > > > > >> > > On Tue, Jul 20, 2021 at 2:11 PM Owen O'Malley < > > owen.omal...@gmail.com > > >> > > > >> > > wrote: > > >> > > > > >> > > > I think we should go ahead and release storage-api 2.8.0 and catch > > >> it > > >> > on > > >> > > > the next cycle. HIVE-25190 is a long standing bug that rarely > > >> affects > > >> > > > users. (We have had a user at LinkedIn hit it, which is why I > > fixed > > >> > it.) > > >> > > > I'll sign up to make the 2.8.1 (and 2.7.3) bug fix releases > > >> afterwards. > > >> > > > > > >> > > > .. Owen > > >> > > > > > >> > > > On Tue, Jul 20, 2021 at 8:53 PM Chao Sun > > >> wrote: > > >> > > > > > >> > > > > Going to check the release and vote here too. Since HIVE-25190 > > is > > >> > > already > > >> > > > > merged, instead of waiting for another release, should we start > > >> > another > > >> > > > RC1 > > >> > > > > with that included? > > >> > > > > > > >> > > > > Chao > > >> > > > > > > >> > > > > On Tue, Jul 20, 2021 at 1:30 PM Dongjoon Hyun < > > >> dongj...@apache.org> > > >> > > > wrote: > > >> > > > > > > >> > > > > > +1 > > >> > > > > > > > >> > > > > > * Build and tested locally. > > >> > > > > > > > >> > > > > > Thanks, > > >> > > > > > Dongjoon. > > >> > > > > > > > >> > > > > > On 2021/07/19 23:15:46, "Owen O'Malley" < > > owen.omal...@gmail.com > > >> > > > >> > > > wrote: > > >> > > > > > > +1 (binding): > > >> > > > > > > * Built and tested > > >> > > > > > > * Built hive main branch using it > > >> > > > > > > * Verified signatures and checksums > > >> > > > > > > > > >> > > > > > > It is too bad that we didn't get HIVE-25190 into it, but > > that > > >> can > > >> > > > wait > > >> > > > > > for > > >> > > > > > > 2.8.1. > > >> > > > > > > > > >> > > > > > > .. Owen > > >> > > > > > > > > >> > > > > > > On Mon, Jun 28, 2021 at 9:44 PM Pavan Lanka > > >> > > > >> > > > > > > >> > > > > > > wrote: > > >> > > > > > > > > >> > > > > > > > +1 (non-binding) > > >> > > > > > > > > > >> > > > > > > > I have done the following: > > >> > > > > > > > * Built and Tested storage-release-2.8.0-rc0 using > > OpenJDK8 > > >> > > > > > > > * Built and Tested ORC
