can you check if the user that the metastore is running as has right to
write to the table dir?
On 10/26/16, 12:23 PM, "aft" <aft...@gmail.com> wrote:
>On Thu, Oct 27, 2016 at 12:00 AM, Eugene Koifman
><ekoif...@hortonworks.com> wrote:
>> could you provide output of ³SHOW COMPACTIONS¹ command
>
>I have dropped the earlier table ...but i created a simple table
>
>create table a (x int , y int) clustered by (x) into 2 buckets stored
>as orc tblproperties("transactional"="ture");
>
>inserted couple of values. then tried to compact it, both major and minor.
>
>but the same thing happens. The compaction jobs are mapred jobs, which
>also shows up at mapred job history page. The FileNotFound exception
>is shown there.
>
>0: jdbc:hive2://hadoop1.openstacksetup.com:21> show compactions;
>+-----------+----------+------------+--------+------------+---------------
>-----------------+----------------+--+
>| dbname | tabname | partname | type | state |
>workerid | starttime |
>+-----------+----------+------------+--------+------------+---------------
>-----------------+----------------+--+
>| Database | Table | Partition | Type | State | Worker
> | Start Time |
>| default | a | NULL | MINOR | failed |
>hadoop3.openstacksetup.com-38 | 1477476121000 |
>| default | a | NULL | MINOR | failed |
>hadoop3.openstacksetup.com-34 | 1477483153000 |
>| default | a | NULL | MINOR | failed |
>hadoop3.openstacksetup.com-38 | 1477483391000 |
>| default | a | NULL | MINOR | attempted | NULL
> | 0 |
>| default | a | NULL | MINOR | attempted | NULL
> | 0 |
>+-----------+----------+------------+--------+------------+---------------
>-----------------+----------------+--+
>6 rows selected (0.065 seconds)
>0: jdbc:hive2://hadoop1.openstacksetup.com:21>
>
>>
>> On 10/25/16, 11:21 PM, "aft" <aft...@gmail.com> wrote:
>>
>>>Well the auto compactor fails every time....and it has been going on
>>>for couple of days. As "day" is a partition, I have to assume, it
>>>happens with all of them.
>>>
>>>Both manual/auto compactor fails.
>>>
>>>On Wed, Oct 26, 2016 at 12:52 AM, Eugene Koifman
>>><ekoif...@hortonworks.com> wrote:
>>>> does this happen for 1 specific partition or all of them?
>>>>
>>>> On 10/25/16, 12:47 AM, "aft" <aft...@gmail.com> wrote:
>>>>
>>>>>Hi,
>>>>>
>>>>>Table created with this :
>>>>>
>>>>> $hive>create table syslog_staged (id string, facility string,
>>>>>sender string, severity string, tstamp string, service string, msg
>>>>>string) partitioned by (hostname string, year string, month string,
>>>>>day string) clustered by (id) into 20 buckets stored as orc
>>>>>tblproperties("transactional"="true");
>>>>>
>>>>>the table is populated with Apache nifi's PutHiveStreaming...
>>>>>
>>>>> $hive>alter table syslog_staged partition
>>>>>(hostname="cloudserver19", year="2016", month="10", day="24") compact
>>>>>'major';
>>>>>
>>>>>Now it turns out compaction fails for some reason.....(from job
>>>>>history)
>>>>>
>>>>> No of maps and reduces are 0 job_1476884195505_0031
>>>>> Job commit failed: java.io.FileNotFoundException: File
>>>>>hdfs://hadoop1.openstacksetup.com:8020/apps/hive/warehouse/log.db/sysl
>>>>>og
>>>>>_s
>>>>>taged/hostname=cloudserver19/year=2016/month=10/day=24/_tmp_27c40005-6
>>>>>58
>>>>>e-
>>>>>48c1-90f7-2acaa124e2fa
>>>>>does not exist.
>>>>> at
>>>>>org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(Distri
>>>>>bu
>>>>>te
>>>>>dFileSystem.java:904)
>>>>> at
>>>>>org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFil
>>>>>eS
>>>>>ys
>>>>>tem.java:113)
>>>>> at
>>>>>org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFile
>>>>>Sy
>>>>>st
>>>>>em.java:966)
>>>>> at
>>>>>org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFile
>>>>>Sy
>>>>>st
>>>>>em.java:962)
>>>>> at
>>>>>org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkReso
>>>>>lv
>>>>>er
>>>>>.java:81)
>>>>> at
>>>>>org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFil
>>>>>eS
>>>>>ys
>>>>>tem.java:962)
>>>>> at
>>>>>org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorOutputCom
>>>>>mi
>>>>>tt
>>>>>er.commitJob(CompactorMR.java:776)
>>>>> at
>>>>>org.apache.hadoop.mapred.OutputCommitter.commitJob(OutputCommitter.jav
>>>>>a:
>>>>>29
>>>>>1)
>>>>> at
>>>>>org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventP
>>>>>ro
>>>>>ce
>>>>>ssor.handleJobCommit(CommitterEventHandler.java:285)
>>>>> at
>>>>>org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventP
>>>>>ro
>>>>>ce
>>>>>ssor.run(CommitterEventHandler.java:237)
>>>>> at
>>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
>>>>>av
>>>>>a:
>>>>>1142)
>>>>> at
>>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
>>>>>ja
>>>>>va
>>>>>:617)
>>>>>
>>>>>from hive metastore log :
>>>>>
>>>>> 2016-10-24 16:33:35,503 WARN [Thread-14]: compactor.Initiator
>>>>>(Initiator.java:run(132)) - Will not initiate compaction for
>>>>>log.syslog_staged.hostname=cloudserver19/year=2016/month=10/day=24
>>>>>since last hive.compactor.initiator.failed.compacts.threshold attempts
>>>>>to compact it failed.
>>>>>
>>>>>
>>>>>Hive version:
>>>>>1.2.1000
>>>>>
>>>>
>>>
>>
>
