Prasanna, AFAIK spark does not handle folders without partition column names in them and there is no way to get spark to do it. I think the reason for this is that parquet file hierarchies had this info and historically spark deals more with those.
On Mon, Nov 28, 2016 at 9:48 AM, Prasanna Santhanam <[email protected]> wrote: > I've been toying around with Spark SQL lately and trying to move some > workloads from Hive. In the hive world the partitions below are recovered > on an ALTER TABLE RECOVER PARTITIONS > > *Path:* > s3://bucket-company/path/2016/03/11 > s3://bucket-company/path/2016/03/12 > s3://bucket-company/path/2016/03/13 > > Where as Spark ignores these unless the partition information is of the > format below > > s3://bucket-company/path/year=2016/month=03/day=11 > s3://bucket-company/path/year=2016/month=03/day=12 > s3://bucket-company/path/year=2016/month=03/day=13 > > The code for this is in ddl.scala. > <https://github.com/apache/spark/blob/ddd02f50bb7458410d65427321efc75da5e65224/sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala#L589> > If my DDL already expresses the partition information why does Spark > ignore the partition and enforce this separator? > > *DDL:* > CREATE EXTERNAL TABLE test_tbl > ( > column1 STRING, > column2 STRUCT <... > > ) > PARTITIONED BY (year STRING, month STRING, day STRING) > LOCATION s3://bucket-company/path > > Thanks, > > > > > -- Bharath (ಭರತ್)
