Re: Saving data using tempTable versus save() method

thanks Robin. This is data from Hive (source) to Hive (Target) via Spark. The database in Hive is called oraclehadoop (mainly used to import data from Oracle in the first place) I am very sceptical of these methods in Spark pertaining to store data in Hive database. I all probability they just st

Re: Saving data using tempTable versus save() method

if you are able to trace the underlying oracle session you can see whether a commit has been called or not. > On 21 Jun 2016, at 09:57, Robin East wrote: > > I’m not sure - I don’t know what those APIs do under the hood. It simply rang > a bell with something I have fallen foul of in the pa

Re: Saving data using tempTable versus save() method

I’m not sure - I don’t know what those APIs do under the hood. It simply rang a bell with something I have fallen foul of in the past (not with Spark though) - have wasted many hours forgetting to commit and then scratching my head as why my data is not persisting. > On 21 Jun 2016, at 09:20

Re: Saving data using tempTable versus save() method

that is a very interesting point. I am not sure. how can I do that with sorted.save("oraclehadoop.sales2") like .. commit? thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: Saving data using tempTable versus save() method

random thought - do you need an explicit commit with the 2nd method? > On 20 Jun 2016, at 21:35, Mich Talebzadeh wrote: > > Hi, > > I have a DF based on a table and sorted and shown below > > This is fine and when I register as tempTable I can populate the underlying > table sales 2 in Hiv

Saving data using tempTable versus save() method

Hi, I have a DF based on a table and sorted and shown below This is fine and when I register as tempTable I can populate the underlying table sales 2 in Hive. That sales2 is an ORC table val s = HiveContext.table("sales_staging") val sorted = s.sort("prod_id","cust_id","time_id","channel_id",