One more thing. I'm using Apache Storm to stream data in Hive. And when I turned off Storm topology compactions started to work properly.
On Tue, Jul 26, 2016 at 6:28 PM, Igor Kuzmenko <f1she...@gmail.com> wrote: > I'm using Hive 1.2.1 transactional table. Inserting data in it via Hive > Streaming API. After some time i expect compaction to start but it didn't > happen: > > Here's part of log, which shows that compactor initiator thread doesn't > see any delta files: > *2016-07-26 18:06:52,459 INFO [Thread-8]: compactor.Initiator > (Initiator.java:run(89)) - Checking to see if we should compact > default.data_aaa.dt=20160726* > *2016-07-26 18:06:52,496 DEBUG [Thread-8]: io.AcidUtils > (AcidUtils.java:getAcidState(432)) - in directory > hdfs://sorm-master01.msk.mts.ru:8020/apps/hive/warehouse/data_aaa/dt=20160726 > base = null deltas = 0* > > *2016-07-26 18:06:52,496 DEBUG [Thread-8]: compactor.Initiator > (Initiator.java:determineCompactionType(271)) - delta size: 0 base size: 0 > threshold: 0.1 will major compact: false* > > But in that directory there's actually 23 files: > > hadoop fs -ls /apps/hive/warehouse/data_aaa/dt=20160726 > Found 23 items > -rw-r--r-- 3 storm hdfs 4 2016-07-26 17:20 > /apps/hive/warehouse/data_aaa/dt=20160726/_orc_acid_version > drwxrwxrwx - storm hdfs 0 2016-07-26 17:22 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71741256_71741355 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:23 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71762456_71762555 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:25 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71787756_71787855 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:26 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71795756_71795855 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:27 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71804656_71804755 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:29 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71828856_71828955 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:30 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71846656_71846755 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:32 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71850756_71850855 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:33 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71867356_71867455 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:34 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71891556_71891655 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:36 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71904856_71904955 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:37 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71907256_71907355 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:39 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71918756_71918855 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:40 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71947556_71947655 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:41 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71960656_71960755 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:43 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71963156_71963255 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:44 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71964556_71964655 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:46 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71987156_71987255 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:47 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_72015756_72015855 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:48 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_72021356_72021455 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:50 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_72048756_72048855 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:50 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_72070856_72070955 > > > Full log here <http://pastebin.com/gHwvgRUV>. > > What could go wrong? >
