Re: Wildcard and Automatic partitioning

2017-08-03 Thread Nirav Patel
Thanks for config for recursive read. However it is not working. Do the file have to have to be in particular format? I am using LOCATION '/user/mycomp/customers/228/departments/partition/'; my subdirectories are like 469990542/dept.tsv, 469990543/dept.tsv. Also I am still not sure why hive ne

Re: Wildcard and Automatic partitioning

2017-08-03 Thread Sergey Shelukhin
How would Hive determine partition keys for partitioning from arbitrary directory structure? There has to be some format, and there already is. Also columns for keys need to be added to the table, with types. For reading it without partitions, Hive already supports mapred.input.dir.recursive, wh

Re: Wildcard and Automatic partitioning

2017-08-03 Thread Nirav Patel
What is the point if I have to rename hdfs directories to hive format (key=value/key=value etc.). Why it can't just be normal directory like everyone has. That entire directory can be considered as a key ("period" in my example) and hive add all values as partitions. On Thu, Aug 3, 2017 at 11:25

Re: Wildcard and Automatic partitioning

2017-08-03 Thread Sergey Shelukhin
The typical, although not technically intended for the purpose, approach is to use msck to “repair” the table and create the partitions. The partitions have to be in the standard Hive format (key=value/key=value etc.) and the table must be created with the corresponding partition keys. It may ac

Wildcard and Automatic partitioning

2017-08-03 Thread Nirav Patel
Hi, is there a way in hive when I create an external table I can specify wild card in LOCATION and have hive automatically identify partitions. I have opened HIVE-17236 for wildcard support. same time I also have issue of specifying partitions. I