Thanks Michael so the following code written using Spark 1.5.1 should be
able to recognise by Hive table right
dataFrame.write().mode(SaveMode.Append).partitionBy("
entity","date").format("orc").save("baseTable");
Hive console:
Create external table bla bla
stored as ORC
Location '/user/xyz/baseT
I believe this is fixed in Spark 1.5.1 as long as the table is only using
types that hive understands and is not partitioned. The problem with
partitioned tables it that hive does not support dynamic discovery unless
you manually run the repair command.
On Tue, Oct 6, 2015 at 9:33 AM, Umesh Kacha
Hi Ted thanks I know I solved that by using dataframe for both reading and
writing. I am running into different problem now if spark can read hive orc
files why can't hive read orc files created by Spark?
On Oct 6, 2015 9:28 PM, "Ted Yu" wrote:
> See this thread:
> http://search-hadoop.com/m/q3RT
See this thread:
http://search-hadoop.com/m/q3RTtwwjNxXvPEe1
A brief search in Spark JIRAs didn't find anything opened on this subject.
On Tue, Oct 6, 2015 at 8:51 AM, unk1102 wrote:
> Hi I have a spark job which creates ORC files in partitions using the
> following code
>
>
> dataFrame.write()
