On Mon, Nov 7, 2016 at 1:46 PM, Eugene Koifman <ekoif...@hortonworks.com>
wrote:
> can you check if the user that the metastore is running as has right to
> write to the table dir?
>
> On 10/26/16, 12:23 PM, "aft" <aft...@gmail.com> wrote:
>
> >On Thu, Oct 27, 2016 at 12:00 AM, Eugene Koifman
> ><ekoif...@hortonworks.com> wrote:
> >> could you provide output of ³SHOW COMPACTIONS¹ command
> >
> >I have dropped the earlier table ...but i created a simple table
> >
> >create table a (x int , y int) clustered by (x) into 2 buckets stored
> >as orc tblproperties("transactional"="ture");
> >
> >inserted couple of values. then tried to compact it, both major and minor.
> >
> >but the same thing happens. The compaction jobs are mapred jobs, which
> >also shows up at mapred job history page. The FileNotFound exception
> >is shown there.
> >
> >0: jdbc:hive2://hadoop1.openstacksetup.com:21> show compactions;
> >+-----------+----------+------------+--------+------------+
> ---------------
> >-----------------+----------------+--+
> >| dbname | tabname | partname | type | state |
> >workerid | starttime |
> >+-----------+----------+------------+--------+------------+
> ---------------
> >-----------------+----------------+--+
> >| Database | Table | Partition | Type | State | Worker
> > | Start Time |
> >| default | a | NULL | MINOR | failed |
> >hadoop3.openstacksetup.com-38 | 1477476121000 |
> >| default | a | NULL | MINOR | failed |
> >hadoop3.openstacksetup.com-34 | 1477483153000 |
> >| default | a | NULL | MINOR | failed |
> >hadoop3.openstacksetup.com-38 | 1477483391000 |
> >| default | a | NULL | MINOR | attempted | NULL
> > | 0 |
> >| default | a | NULL | MINOR | attempted | NULL
> > | 0 |
> >+-----------+----------+------------+--------+------------+
> ---------------
> >-----------------+----------------+--+
> >6 rows selected (0.065 seconds)
> >0: jdbc:hive2://hadoop1.openstacksetup.com:21>
> >
> >>
> >> On 10/25/16, 11:21 PM, "aft" <aft...@gmail.com> wrote:
> >>
> >>>Well the auto compactor fails every time....and it has been going on
> >>>for couple of days. As "day" is a partition, I have to assume, it
> >>>happens with all of them.
> >>>
> >>>Both manual/auto compactor fails.
> >>>
> >>>On Wed, Oct 26, 2016 at 12:52 AM, Eugene Koifman
> >>><ekoif...@hortonworks.com> wrote:
> >>>> does this happen for 1 specific partition or all of them?
> >>>>
> >>>> On 10/25/16, 12:47 AM, "aft" <aft...@gmail.com> wrote:
> >>>>
> >>>>>Hi,
> >>>>>
> >>>>>Table created with this :
> >>>>>
> >>>>> $hive>create table syslog_staged (id string, facility string,
> >>>>>sender string, severity string, tstamp string, service string, msg
> >>>>>string) partitioned by (hostname string, year string, month string,
> >>>>>day string) clustered by (id) into 20 buckets stored as orc
> >>>>>tblproperties("transactional"="true");
> >>>>>
> >>>>>the table is populated with Apache nifi's PutHiveStreaming...
> >>>>>
> >>>>> $hive>alter table syslog_staged partition
> >>>>>(hostname="cloudserver19", year="2016", month="10", day="24") compact
> >>>>>'major';
> >>>>>
> >>>>>Now it turns out compaction fails for some reason.....(from job
> >>>>>history)
> >>>>>
> >>>>> No of maps and reduces are 0 job_1476884195505_0031
> >>>>> Job commit failed: java.io.FileNotFoundException: File
> >>>>>hdfs://hadoop1.openstacksetup.com:8020/apps/
> hive/warehouse/log.db/sysl
> >>>>>og
> >>>>>_s
> >>>>>taged/hostname=cloudserver19/year=2016/month=
> 10/day=24/_tmp_27c40005-6
> >>>>>58
> >>>>>e-
> >>>>>48c1-90f7-2acaa124e2fa
> >>>>>does not exist.
> >>>>> at
> >>>>>org.apache.hadoop.hdfs.DistributedFileSystem.
> listStatusInternal(Distri
> >>>>>bu
> >>>>>te
> >>>>>dFileSystem.java:904)
> >>>>> at
> >>>>>org.apache.hadoop.hdfs.DistributedFileSystem.access$
> 600(DistributedFil
> >>>>>eS
> >>>>>ys
> >>>>>tem.java:113)
> >>>>> at
> >>>>>org.apache.hadoop.hdfs.DistributedFileSystem$21.
> doCall(DistributedFile
> >>>>>Sy
> >>>>>st
> >>>>>em.java:966)
> >>>>> at
> >>>>>org.apache.hadoop.hdfs.DistributedFileSystem$21.
> doCall(DistributedFile
> >>>>>Sy
> >>>>>st
> >>>>>em.java:962)
> >>>>> at
> >>>>>org.apache.hadoop.fs.FileSystemLinkResolver.
> resolve(FileSystemLinkReso
> >>>>>lv
> >>>>>er
> >>>>>.java:81)
> >>>>> at
> >>>>>org.apache.hadoop.hdfs.DistributedFileSystem.
> listStatus(DistributedFil
> >>>>>eS
> >>>>>ys
> >>>>>tem.java:962)
> >>>>> at
> >>>>>org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$
> CompactorOutputCom
> >>>>>mi
> >>>>>tt
> >>>>>er.commitJob(CompactorMR.java:776)
> >>>>> at
> >>>>>org.apache.hadoop.mapred.OutputCommitter.commitJob(
> OutputCommitter.jav
> >>>>>a:
> >>>>>29
> >>>>>1)
> >>>>> at
> >>>>>org.apache.hadoop.mapreduce.v2.app.commit.
> CommitterEventHandler$EventP
> >>>>>ro
> >>>>>ce
> >>>>>ssor.handleJobCommit(CommitterEventHandler.java:285)
> >>>>> at
> >>>>>org.apache.hadoop.mapreduce.v2.app.commit.
> CommitterEventHandler$EventP
> >>>>>ro
> >>>>>ce
> >>>>>ssor.run(CommitterEventHandler.java:237)
> >>>>> at
> >>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.j
> >>>>>av
> >>>>>a:
> >>>>>1142)
> >>>>> at
> >>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.
> >>>>>ja
> >>>>>va
> >>>>>:617)
> >>>>>
> >>>>>from hive metastore log :
> >>>>>
> >>>>> 2016-10-24 16:33:35,503 WARN [Thread-14]: compactor.Initiator
> >>>>>(Initiator.java:run(132)) - Will not initiate compaction for
> >>>>>log.syslog_staged.hostname=cloudserver19/year=2016/month=10/day=24
> >>>>>since last hive.compactor.initiator.failed.compacts.threshold
> attempts
> >>>>>to compact it failed.
> >>>>>
> >>>>>
> >>>>>Hive version:
> >>>>>1.2.1000
> >>>>>
> >>>>
> >>>
> >>
> >
>
>
Something in your snippet
as orc tblproperties("transactional"="ture");
ture vs true
