could you provide output of ³SHOW COMPACTIONS¹ command
On 10/25/16, 11:21 PM, "aft" <aft...@gmail.com> wrote:
>Well the auto compactor fails every time....and it has been going on
>for couple of days. As "day" is a partition, I have to assume, it
>happens with all of them.
>
>Both manual/auto compactor fails.
>
>On Wed, Oct 26, 2016 at 12:52 AM, Eugene Koifman
><ekoif...@hortonworks.com> wrote:
>> does this happen for 1 specific partition or all of them?
>>
>> On 10/25/16, 12:47 AM, "aft" <aft...@gmail.com> wrote:
>>
>>>Hi,
>>>
>>>Table created with this :
>>>
>>> $hive>create table syslog_staged (id string, facility string,
>>>sender string, severity string, tstamp string, service string, msg
>>>string) partitioned by (hostname string, year string, month string,
>>>day string) clustered by (id) into 20 buckets stored as orc
>>>tblproperties("transactional"="true");
>>>
>>>the table is populated with Apache nifi's PutHiveStreaming...
>>>
>>> $hive>alter table syslog_staged partition
>>>(hostname="cloudserver19", year="2016", month="10", day="24") compact
>>>'major';
>>>
>>>Now it turns out compaction fails for some reason.....(from job history)
>>>
>>> No of maps and reduces are 0 job_1476884195505_0031
>>> Job commit failed: java.io.FileNotFoundException: File
>>>hdfs://hadoop1.openstacksetup.com:8020/apps/hive/warehouse/log.db/syslog
>>>_s
>>>taged/hostname=cloudserver19/year=2016/month=10/day=24/_tmp_27c40005-658
>>>e-
>>>48c1-90f7-2acaa124e2fa
>>>does not exist.
>>> at
>>>org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(Distribu
>>>te
>>>dFileSystem.java:904)
>>> at
>>>org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileS
>>>ys
>>>tem.java:113)
>>> at
>>>org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSy
>>>st
>>>em.java:966)
>>> at
>>>org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSy
>>>st
>>>em.java:962)
>>> at
>>>org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolv
>>>er
>>>.java:81)
>>> at
>>>org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileS
>>>ys
>>>tem.java:962)
>>> at
>>>org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorOutputCommi
>>>tt
>>>er.commitJob(CompactorMR.java:776)
>>> at
>>>org.apache.hadoop.mapred.OutputCommitter.commitJob(OutputCommitter.java:
>>>29
>>>1)
>>> at
>>>org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventPro
>>>ce
>>>ssor.handleJobCommit(CommitterEventHandler.java:285)
>>> at
>>>org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventPro
>>>ce
>>>ssor.run(CommitterEventHandler.java:237)
>>> at
>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>>a:
>>>1142)
>>> at
>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>va
>>>:617)
>>>
>>>from hive metastore log :
>>>
>>> 2016-10-24 16:33:35,503 WARN [Thread-14]: compactor.Initiator
>>>(Initiator.java:run(132)) - Will not initiate compaction for
>>>log.syslog_staged.hostname=cloudserver19/year=2016/month=10/day=24
>>>since last hive.compactor.initiator.failed.compacts.threshold attempts
>>>to compact it failed.
>>>
>>>
>>>Hive version:
>>>1.2.1000
>>>
>>
>
