That is an interesting point Alan. Does this imply that Hive on Spark (Hive 2 encourages Spark or TEZ) is going to have an issue with transactional tables?
Besides this begs the question that we still run Hive on Spark 1.3.1 engine. We don't seem to make Hive work on Spark engine with a newer version of Spark. Cheers, Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com On 14 March 2016 at 16:48, Alan Gates <alanfga...@gmail.com> wrote: > I don’t know why you’re seeing Hive on Spark sometimes work with > transactional tables and sometimes not. But note that in general it > doesn’t work. The Spark runtime in Hive does not send heartbeats to the > transaction/lock manager so it will timeout any job that takes longer than > the heartbeat interval (5 min by default). > > Alan. > > > On Mar 12, 2016, at 00:24, @Sanjiv Singh <sanjiv.is...@gmail.com> wrote: > > > > Hi All, > > > > I am facing this issue on HDP setup on which COMPACTION is required only > once for transactional tables to fetch records with Spark SQL. > > On the other hand, Apache setup doesn't required compaction even once. > > > > May be something got triggered on meta-store after compaction, Spark SQL > start recognizing delta files. > > > > Let know me if needed other details to get root cause. > > > > Try this, > > > > See complete scenario : > > > > hive> create table default.foo(id int) clustered by (id) into 2 buckets > STORED AS ORC TBLPROPERTIES ('transactional'='true'); > > hive> insert into default.foo values(10); > > > > scala> sqlContext.table("default.foo").count // Gives 0, which is wrong > because data is still in delta files > > > > Now run major compaction: > > > > hive> ALTER TABLE default.foo COMPACT 'MAJOR'; > > > > scala> sqlContext.table("default.foo").count // Gives 1 > > > > hive> insert into foo values(20); > > > > scala> sqlContext.table("default.foo").count // Gives 2 , no compaction > required. > > > > > > > > > > Regards > > Sanjiv Singh > > Mob : +091 9990-447-339 > >
