Hi folks, How can I force spark sql to recursively get data stored in parquet format from subdirectories ? In Hive, I could achieve this by setting few Hive configs.
set hive.input.dir.recursive=true; set hive.mapred.supports.subdirectories=true; set hive.supports.subdirectories=true; set mapred.input.dir.recursive=true; I tried to set these configs through spark sql queries but I get 0 records all the times compared to hive which get me the expected results. I also put these confs in hive-site.xml file but nothing changed. How can I handle this issue ? Spark Version : 2.1.0 I used Hive 2.1.1 on emr-5.3.1 Regards, Ahmed Kamal MTS in Data Science Email: ahmed.abdelfa...@careem.com<mailto:ahmed.abdelfa...@careem.com>
