Issue in Insert Overwrite directory operation

Hi All, I see a weird issue when trying to do a "INSERT OVERWRITE DIRECTORY" operation. The query seems to work when I limit the data set but fails with the following exception if the data set is larger: Failed with exception Unable to move source hdfs://namenode/user/grp_admin/external_test1/out

Re: Disable Hive autogather optimization

| >> NULL | >> | InputFormat: | >> org.apache.hadoop.mapred.TextInputFormat| >> NULL | >> | OutputFormat: | >> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat

Re: Disable Hive autogather optimization

o listen to your idea and > motivation so that we can better design Hive stats collection. Thanks! > > Best > Pengcheng > > > On Thu, Apr 28, 2016 at 4:12 PM, Udit Mehta wrote: > >> Any insights on this? >> >> On Tue, Apr 26, 2016 at 7:32 PM, Udit Mehta wr

Re: Disable Hive autogather optimization

Any insights on this? On Tue, Apr 26, 2016 at 7:32 PM, Udit Mehta wrote: > Update: Realized this works if we create a fresh table with this config > already disabled but does not work if there is already a table created when > this config was enabled. We now need to figure out how t

Re: Disable Hive autogather optimization

6:16 PM, Udit Mehta wrote: > Hive version we are using is 1.2.1. > > On Tue, Apr 26, 2016 at 6:01 PM, Udit Mehta wrote: > >> Hi, >> >> We need to disable the Hive autogather stats optimization by disabling " >> *hive.stats.autogather*" but for some

Re: Disable Hive autogather optimization

Hive version we are using is 1.2.1. On Tue, Apr 26, 2016 at 6:01 PM, Udit Mehta wrote: > Hi, > > We need to disable the Hive autogather stats optimization by disabling " > *hive.stats.autogather*" but for some reason, the config change doesnt > seem to go through. We mo

Disable Hive autogather optimization

Hi, We need to disable the Hive autogather stats optimization by disabling " *hive.stats.autogather*" but for some reason, the config change doesnt seem to go through. We modified this config in the hive-site.xml and restarted the Hive metastore. We also made this change explicitly in the job but

Best Hive Authorization Model for Shared data

Hi all, I wanted to understand what authorization model is most suitable for a production environment where most of the data is shared between multiple teams and users. I know this is would depend more on the use case but I cant seem to figure out the best model for our use: We have data that is

Re: Hive Metastore Bottleneck

te: >> >>> Can you elaborate on where you see the bottleneck? A general overview >>> of your access path would be useful. For instance if you'r accessing Hive >>> metastore via HiveServer2 or from webhcat using embedded cli or something >>> else. >

Re: Hive Metastore Bottleneck

database (MySQL)? > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadeh

Hive Metastore Bottleneck

Hi all, We are currently running Hive in production and staging with the metastore connecting to a MySql database in the backend. The traffic in production accessing the metastore is more than staging which is expected. We have had a sudden increase in traffic which has led to the metastore operat

Re: Hive on spark table caching

ed by Peridale Technology > Ltd, its subsidiaries or their employees, unless expressly so stated. It is > the responsibility of the recipient to ensure that this email is virus > free, therefore neither Peridale Ltd, its subsidiaries nor their employees > accept any responsibility. > &g

Hive on spark table caching

Hi, I have started using Hive on Spark recently and am exploring the benefits it offers. I was wondering if Hive on Spark has capabilities to cache table like Spark SQL. Or does it do any form of implicit caching in the long running job which it starts after running the first query? Thanks, Udit

Re: Hive version with Spark

As per this link : https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started, you need to build Spark without Hive. On Wed, Nov 18, 2015 at 8:50 AM, Sofia wrote: > Hello > > After various failed tries to use my Hive (1.2.1) with my Spark (Spark > 1.4.1 built for Hadoop 2.

Building Spark to use for Hive on Spark

Hi, I am planning to test out the Hive on Spark functionality provided by the newer versions of Hive. I wanted to know why is it necessary to remove the Hive jars from the Spark build as mentioned on this this page.

Re: Hive Server2 Monitoring

Yes thats right. We dont use Cloudera Manager so cannot use that. Increasing the PermGen and Heap space is what I did as well but I still have no insight into what causes it to increase. On Wed, Nov 11, 2015 at 11:23 AM, Personal wrote: > We use Cloudera Manager, but I’m guessing since you’re as

Hive Server2 Monitoring

Hi, I was planning to use the Hive Server2 in production and was wondering how others monitor the Hive Server2 usage. I saw some recent errors related to the PermGen and Heap Space so maybe that could be a start. It would also be useful to detect some unusual activity like a sudden increase in con

Best way to deal with incompatible column type changes

Hi, I have a Hive external table with a lot of partitions where the underlying data is in JSON. I use this popular serde to read and write in JSON format. So I have a data stream where sometimes there are changes to the JSON structure. For eg, a key mi

Re: Change hive column size

Also this might be relevant now: https://issues.apache.org/jira/browse/HIVE-9815 On Thu, May 28, 2015 at 10:41 PM, Udit Mehta wrote: > Hi Steve, > > I do see that it applies to hive 0.5. But I am facing a similar issue > where my column of complex type is not able to hold a nested s

Re: Change hive column size

confluence/display/Hive/LanguageManual+Types > > Thanks, > > Steve > > Sent from my iPad > > On May 28, 2015, at 9:08 PM, Udit Mehta wrote: > > Hi, > > Per this ticket: https://issues.apache.org/jira/browse/HIVE-1364 , the > max column size in hive is limi

Change hive column size

Hi, Per this ticket: https://issues.apache.org/jira/browse/HIVE-1364 , the max column size in hive is limited to 4000 chars. But i do read that there is a way to increase it via *mysql* which is our database for the metastore. Can anyone point me as to how I can do this? Our columns have deeply n

Storage Based Authorization

Hi, I have enabled storage based authorization in the hive metastore by adding the following configs to hive-site: > > hive.security.authorization.enabled > true > > > > hive.security.authorization.manager > > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedA

Re: Hive drop table error

As a secondary thought, is it possible to remove the table from mysql if its not possible to remove it from hive. What all entries in the mysql tables would I need to remove? On Tue, Apr 7, 2015 at 10:52 AM, Udit Mehta wrote: > Hi, > > I was able to create a highly nested table in hiv

Hive drop table error

Hi, I was able to create a highly nested table in hive but for some reason now I am unable to drop it or describe it. I get an IllegalArgumentException Error and dont know how to delete the table now. Does anyone have any ideas on how I can do this? The table has more than a 100 fields. Thanks, U

hyphen in hive struct field

Hi, I have a hive table query: create external table test (field1 struct< `inner-table` : string>); I believe hyphens are disallowed but to overcome that i read that we can use ``(ticks) around them. But even this seems to fail. Is there a way around this or hypens are not allowed in nested hi

Hive Json Serde

I am using hive from HDP 2.2 and need to create a Hive table to query multilevel Json data in HDFS of the following format: { "timestamp": "1424100629409", "head": { "time": "2015-02-16T15:30:29.409Z", "place": { "url": null, "country": "US" }