Well the auto compactor fails every time....and it has been going on
for couple of days. As "day" is a partition, I have to assume, it
happens with all of them.
Both manual/auto compactor fails.
On Wed, Oct 26, 2016 at 12:52 AM, Eugene Koifman
<ekoif...@hortonworks.com> wrote:
> does this happen for 1 specific partition or all of them?
>
> On 10/25/16, 12:47 AM, "aft" <aft...@gmail.com> wrote:
>
>>Hi,
>>
>>Table created with this :
>>
>> $hive>create table syslog_staged (id string, facility string,
>>sender string, severity string, tstamp string, service string, msg
>>string) partitioned by (hostname string, year string, month string,
>>day string) clustered by (id) into 20 buckets stored as orc
>>tblproperties("transactional"="true");
>>
>>the table is populated with Apache nifi's PutHiveStreaming...
>>
>> $hive>alter table syslog_staged partition
>>(hostname="cloudserver19", year="2016", month="10", day="24") compact
>>'major';
>>
>>Now it turns out compaction fails for some reason.....(from job history)
>>
>> No of maps and reduces are 0 job_1476884195505_0031
>> Job commit failed: java.io.FileNotFoundException: File
>>hdfs://hadoop1.openstacksetup.com:8020/apps/hive/warehouse/log.db/syslog_s
>>taged/hostname=cloudserver19/year=2016/month=10/day=24/_tmp_27c40005-658e-
>>48c1-90f7-2acaa124e2fa
>>does not exist.
>> at
>>org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(Distribute
>>dFileSystem.java:904)
>> at
>>org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSys
>>tem.java:113)
>> at
>>org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSyst
>>em.java:966)
>> at
>>org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSyst
>>em.java:962)
>> at
>>org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver
>>.java:81)
>> at
>>org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSys
>>tem.java:962)
>> at
>>org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorOutputCommitt
>>er.commitJob(CompactorMR.java:776)
>> at
>>org.apache.hadoop.mapred.OutputCommitter.commitJob(OutputCommitter.java:29
>>1)
>> at
>>org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProce
>>ssor.handleJobCommit(CommitterEventHandler.java:285)
>> at
>>org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProce
>>ssor.run(CommitterEventHandler.java:237)
>> at
>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
>>1142)
>> at
>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
>>:617)
>>
>>from hive metastore log :
>>
>> 2016-10-24 16:33:35,503 WARN [Thread-14]: compactor.Initiator
>>(Initiator.java:run(132)) - Will not initiate compaction for
>>log.syslog_staged.hostname=cloudserver19/year=2016/month=10/day=24
>>since last hive.compactor.initiator.failed.compacts.threshold attempts
>>to compact it failed.
>>
>>
>>Hive version:
>>1.2.1000
>>
>
