Re: Auto Refresh Hive Table Metadata

2018-08-10 Thread Gopal Vijayaraghavan
> By the way, if you want near-real-time tables with Hive, maybe you should > have a look at this project from Uber: https://uber.github.io/hudi/ > I don't know how mature it is yet, but I think it aims at solving that kind > of challenge. Depending on your hive setup, you don't need a differe

Enabling Snappy compression on Parquet

2018-08-10 Thread Patrick Duin
Hi, I got some hive tables in Parquet format and I am trying to find out how best to enable compression. Done a bit of searching and the information is a bit scattered but I found I can use this hive property to enable compression.It needs to be set before doing an insert. set parquet.compressio

Re: Auto Refresh Hive Table Metadata

2018-08-10 Thread Furcy Pin
Hi Chintan, Yes, this sounds weird... "REFRESH TABLES" is the kind of statement required by SQL engines such as Impala, Presto or Spark-SQL that cache metadata from the Metastore, but vanilla Hive usually don't cache it and query the metastore every time (unless some new feature was added recentl