On Thu, Oct 27, 2016 at 12:00 AM, Eugene Koifman
<ekoif...@hortonworks.com> wrote:
> could you provide output of ³SHOW COMPACTIONS¹ command
I have dropped the earlier table ...but i created a simple table
create table a (x int , y int) clustered by (x) into 2 buckets stored
as orc tblproperties("transactional"="ture");
inserted couple of values. then tried to compact it, both major and minor.
but the same thing happens. The compaction jobs are mapred jobs, which
also shows up at mapred job history page. The FileNotFound exception
is shown there.
0: jdbc:hive2://hadoop1.openstacksetup.com:21> show compactions;
+-----------+----------+------------+--------+------------+--------------------------------+----------------+--+
| dbname | tabname | partname | type | state |
workerid | starttime |
+-----------+----------+------------+--------+------------+--------------------------------+----------------+--+
| Database | Table | Partition | Type | State | Worker
| Start Time |
| default | a | NULL | MINOR | failed |
hadoop3.openstacksetup.com-38 | 1477476121000 |
| default | a | NULL | MINOR | failed |
hadoop3.openstacksetup.com-34 | 1477483153000 |
| default | a | NULL | MINOR | failed |
hadoop3.openstacksetup.com-38 | 1477483391000 |
| default | a | NULL | MINOR | attempted | NULL
| 0 |
| default | a | NULL | MINOR | attempted | NULL
| 0 |
+-----------+----------+------------+--------+------------+--------------------------------+----------------+--+
6 rows selected (0.065 seconds)
0: jdbc:hive2://hadoop1.openstacksetup.com:21>
>
> On 10/25/16, 11:21 PM, "aft" <aft...@gmail.com> wrote:
>
>>Well the auto compactor fails every time....and it has been going on
>>for couple of days. As "day" is a partition, I have to assume, it
>>happens with all of them.
>>
>>Both manual/auto compactor fails.
>>
>>On Wed, Oct 26, 2016 at 12:52 AM, Eugene Koifman
>><ekoif...@hortonworks.com> wrote:
>>> does this happen for 1 specific partition or all of them?
>>>
>>> On 10/25/16, 12:47 AM, "aft" <aft...@gmail.com> wrote:
>>>
>>>>Hi,
>>>>
>>>>Table created with this :
>>>>
>>>> $hive>create table syslog_staged (id string, facility string,
>>>>sender string, severity string, tstamp string, service string, msg
>>>>string) partitioned by (hostname string, year string, month string,
>>>>day string) clustered by (id) into 20 buckets stored as orc
>>>>tblproperties("transactional"="true");
>>>>
>>>>the table is populated with Apache nifi's PutHiveStreaming...
>>>>
>>>> $hive>alter table syslog_staged partition
>>>>(hostname="cloudserver19", year="2016", month="10", day="24") compact
>>>>'major';
>>>>
>>>>Now it turns out compaction fails for some reason.....(from job history)
>>>>
>>>> No of maps and reduces are 0 job_1476884195505_0031
>>>> Job commit failed: java.io.FileNotFoundException: File
>>>>hdfs://hadoop1.openstacksetup.com:8020/apps/hive/warehouse/log.db/syslog
>>>>_s
>>>>taged/hostname=cloudserver19/year=2016/month=10/day=24/_tmp_27c40005-658
>>>>e-
>>>>48c1-90f7-2acaa124e2fa
>>>>does not exist.
>>>> at
>>>>org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(Distribu
>>>>te
>>>>dFileSystem.java:904)
>>>> at
>>>>org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileS
>>>>ys
>>>>tem.java:113)
>>>> at
>>>>org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSy
>>>>st
>>>>em.java:966)
>>>> at
>>>>org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSy
>>>>st
>>>>em.java:962)
>>>> at
>>>>org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolv
>>>>er
>>>>.java:81)
>>>> at
>>>>org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileS
>>>>ys
>>>>tem.java:962)
>>>> at
>>>>org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorOutputCommi
>>>>tt
>>>>er.commitJob(CompactorMR.java:776)
>>>> at
>>>>org.apache.hadoop.mapred.OutputCommitter.commitJob(OutputCommitter.java:
>>>>29
>>>>1)
>>>> at
>>>>org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventPro
>>>>ce
>>>>ssor.handleJobCommit(CommitterEventHandler.java:285)
>>>> at
>>>>org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventPro
>>>>ce
>>>>ssor.run(CommitterEventHandler.java:237)
>>>> at
>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>>>a:
>>>>1142)
>>>> at
>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>>va
>>>>:617)
>>>>
>>>>from hive metastore log :
>>>>
>>>> 2016-10-24 16:33:35,503 WARN [Thread-14]: compactor.Initiator
>>>>(Initiator.java:run(132)) - Will not initiate compaction for
>>>>log.syslog_staged.hostname=cloudserver19/year=2016/month=10/day=24
>>>>since last hive.compactor.initiator.failed.compacts.threshold attempts
>>>>to compact it failed.
>>>>
>>>>
>>>>Hive version:
>>>>1.2.1000
>>>>
>>>
>>
>
