Re: Gather Partition Locations

2019-11-17 Thread mb
Vivek Shrivastava wrote: > If you have access to HCatalog, it also has jdbc connection that would > allow you to get faster response. ah ok. sounds awesome as well! I will check. thanks marko -- Marko Bauhardt Software Engineer www.datameer.com Phone: +49 345 279 5030 Datameer GmbH Magdeburg

Re: Gather Partition Locations

2019-11-16 Thread Vivek Shrivastava
If you have access to HCatalog, it also has jdbc connection that would allow you to get faster response. On Tue, Nov 12, 2019 at 6:53 AM Elliot West wrote: > Hello, > > We faced a similar problem. Additionally, we had job clients were > difficult to integrate directly with the Thirft API, but ne

Re: Gather Partition Locations

2019-11-12 Thread Elliot West
Hello, We faced a similar problem. Additionally, we had job clients were difficult to integrate directly with the Thirft API, but needed to resolve file locations via the metastore. To handle this, we build a cut down service with a REST API that fronts the Hive metastore. The API is optimised for

Re: Gather Partition Locations

2019-11-12 Thread mb
Gopal Vijayaraghavan wrote: > That was the reason Hive shipped with metatool, though it remains fairly > obscure outside of the devs. > > hive --service metatool -executeJDOQL "select database.name + '.' + tableName > from org.apache.hadoop.hive.metastore.model.MTable" > > You need to join MPa

Re: Gather Partition Locations

2019-11-12 Thread mb
Ashutosh Bapat wrote: > There are multiple ways > 1. Query the HiveMetaStore directly. do you mean via thrift client? or directly native jdbc? But i think this is in an enterprise env not possible, when i'm not on the same machine where hive server is running. i believe the mysql or postgres ser

re: Gather Partition Locations

2019-11-11 Thread Gopal Vijayaraghavan
Hi, > I have a question about how to get the location for a bunch of partitions. ... > But in an enterprise environment I'm pretty sure this approach would not be > the best because the RDS (mysql or derby) is maybe not reachable or > I don't have the permission to it. That was the reason Hive sh

Re: Gather Partition Locations

2019-11-11 Thread Ashutosh Bapat
There are multiple ways 1. Query the HiveMetaStore directly. 2. Use sys.* tables or better even information_schema to get the same. On Fri, Nov 8, 2019 at 9:14 PM wrote: > Hi, > I have a question about how to get the location for a bunch of partitions. > My answer is: using the hive query `DESCR