[
https://issues.apache.org/jira/browse/HIVE-20852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365272#comment-17365272
]
David G. edited comment on HIVE-20852 at 6/18/21, 6:43 AM:
-----------------------------------------------------------
We faced the same issue with the nifi processor PutHive3streaming and after
analysed code and hive metastore tables we understood that the stransctions
were not written into the metastore which prevent hive compaction scanner to
found our partitions. The workaround that we used is the following :
We set the table property 'NO_AUTO_COMPACTION'='true' to force Hive compaction
job to ignore our tables
We Use nifi processor PutHiveQL to run scheduled compaction queries like that
regarding :
alter table xxx PARTITION yyy compact 'major';
or
alter table xxx PARTITION yyy compact 'minor';
was (Author: dgu):
We faced the same issue with the nifi processor PutHive3streaming and after
analysed code and hive metastore tables we understood that the stransctions
were not written into the mestatore. The workaround that we used is the
following :
We set the table property 'NO_AUTO_COMPACTION'='true' to force Hive compaction
job to ignore our tables
We Use nifi processor PutHiveQL to run scheduled compaction queries like that
regarding :
alter table xxx PARTITION yyy compact 'major';
or
alter table xxx PARTITION yyy compact 'minor';
> Compaction Initiator ignores datas inserted by Stream Data Ingest
> -----------------------------------------------------------------
>
> Key: HIVE-20852
> URL: https://issues.apache.org/jira/browse/HIVE-20852
> Project: Hive
> Issue Type: Bug
> Components: API, Transactions
> Affects Versions: 3.1.0
> Reporter: Kei Miyauchi
> Priority: Major
>
> HI,
> Before compaction, Initiator decides whether a table/partition is
> potentialCompaction by querying COMPLETE_TXN_COMPONENT.
> But I found the transactions which committed by Stream Data Ingest is not
> stored on COMPLETE_TXN_COMPONENT. This is because the statement "insert into
> COMPLETED_TXN_COMPONENTS (ctc_txnid, ctc_database, ctc_table, ctc_partition,
> ctc_writeid, ctc_update_delete) select tc_txnid, tc_database, tc_table,
> tc_partition, tc_writeid, 'N' from TXN_COMPONENTS where tc_txnid = (id)"
> fails.
> I found INSERT statement to TXN_COMPONENTS isn't fired. select subquery above
> returns 0 row.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
