Thanks much Zhan Zhang. I will open a JIRA saying orc files created using
hiveContext.sql can't be read by dataframe reader.
Regards,
Umesh
On Oct 4, 2015 10:14, "Zhan Zhang" wrote:
> HI Umesh,
>
> It depends on how you create and read the orc file, although everything
> happens in side of spark
Hi Zang any idea why is this happening? I can load ORC files created by
Hive table but I cant load ORC files created by Spark itself. It looks like
bug.
On Wed, Sep 30, 2015 at 12:03 PM, Umesh Kacha wrote:
> Hi Zang thanks much please find the code below
>
> Working code loading data from a path
Hi Zang thanks much please find the code below
Working code loading data from a path created by Hive table using hive
console outside of spark :
DataFrame df =
hiveContext.read().format("orc").load("/hdfs/path/to/hive/table/partition")
Not working code inside spark hive tables created using hive
Hi I can read/load orc data created by hive table in a dataframe why is it
throwing Malformed ORC exception when I try to load data created by
hiveContext.sql into dataframe?
On Sep 30, 2015 2:37 AM, "Hortonworks" wrote:
> You can try to use data frame for both read and write
>
> Thanks
>
> Zhan
You can try to use data frame for both read and write
Thanks
Zhan Zhang
Sent from my iPhone
> On Sep 29, 2015, at 1:56 PM, Umesh Kacha wrote:
>
> Hi Zang, thanks for the response. Table is created using Spark
> hiveContext.sql and data inserted into table also using hiveContext.sql.
> Inse
Hi Zang, thanks for the response. Table is created using Spark
hiveContext.sql and data inserted into table also using hiveContext.sql.
Insert into partition table. When I try to load orc data into dataframe I
am loading particular partition data stored in path say
/user/xyz/Hive/xyz.db/sparktable/
How was the table is generated, by hive or by spark?
If you generate table using have but read it by data frame, it may have some
comparability issue.
Thanks
Zhan Zhang
Sent from my iPhone
> On Sep 29, 2015, at 1:47 PM, unk1102 wrote:
>
> Hi I have a spark job which creates hive tables in
