Hi All,
I am facing this issue on HDP setup on which COMPACTION is required only
once for transactional tables to fetch records with Spark SQL.
On the other hand, Apache setup doesn't required compaction even once.
May be something got triggered on meta-store after compaction, Spark SQL
start recognizing delta files.
Let know me if needed other details to get root cause.
Try this,
*See complete scenario :*
hive> create table default.foo(id int) clustered by (id) into 2 buckets
STORED AS ORC TBLPROPERTIES ('transactional'='true');
hive> insert into default.foo values(10);
scala> sqlContext.table("default.foo").count // Gives 0, which is wrong
because data is still in delta files
Now run major compaction:
hive> ALTER TABLE default.foo COMPACT 'MAJOR';
scala> sqlContext.table("default.foo").count // Gives 1
hive> insert into foo values(20);
scala> sqlContext.table("default.foo").count* // Gives 2 , no compaction
required.*
Regards
Sanjiv Singh
Mob : +091 9990-447-339
