I’m not sure - I don’t know what those APIs do under the hood. It simply rang a
bell with something I have fallen foul of in the past (not with Spark though) -
have wasted many hours forgetting to commit and then scratching my head as why
my data is not persisting.
> On 21 Jun 2016, at 09:20, Mich Talebzadeh <mich.talebza...@gmail.com> wrote:
>
> that is a very interesting point. I am not sure. how can I do that with
>
> sorted.save("oraclehadoop.sales2")
>
> like .. commit?
>
> thanks
>
> Dr Mich Talebzadeh
>
> LinkedIn
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>
>
> http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
>
>
> On 21 June 2016 at 08:56, Robin East <robin.e...@xense.co.uk
> <mailto:robin.e...@xense.co.uk>> wrote:
> random thought - do you need an explicit commit with the 2nd method?
>
>
>
>
>> On 20 Jun 2016, at 21:35, Mich Talebzadeh <mich.talebza...@gmail.com
>> <mailto:mich.talebza...@gmail.com>> wrote:
>>
>> Hi,
>>
>> I have a DF based on a table and sorted and shown below
>>
>> This is fine and when I register as tempTable I can populate the underlying
>> table sales 2 in Hive. That sales2 is an ORC table
>>
>> val s = HiveContext.table("sales_staging")
>> val sorted = s.sort("prod_id","cust_id","time_id","channel_id","promo_id")
>> sorted.registerTempTable("tmp")
>> sqltext = """
>> INSERT INTO TABLE oraclehadoop.sales2
>> SELECT
>> PROD_ID
>> , CUST_ID
>> , TIME_ID
>> , CHANNEL_ID
>> , PROMO_ID
>> , QUANTITY_SOLD
>> , AMOUNT_SOLD
>> FROM tmp
>> """
>> HiveContext.sql(sqltext)
>> HiveContext.sql("select count(1) from oraclehadoop.sales2").show
>> HiveContext.sql("truncate table oraclehadoop.sales2")
>>
>> sorted.save("oraclehadoop.sales2")
>> HiveContext.sql("select count(1) from oraclehadoop.sales2").show
>>
>> When I truncate the Hive table and use sorted.save("oraclehadoop.sales2")
>>
>> It does not save any data
>>
>> Started at
>> [20/06/2016 21:21:57.57]
>> +------+
>> | _c0|
>> +------+
>> |918843| // This works
>> +------+
>> [Stage 7:============================================> (3 + 1)
>> / 4]SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>> SLF4J: Defaulting to no-operation (NOP) logger implementation
>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder
>> <http://www.slf4j.org/codes.html#StaticLoggerBinder> for further details.
>> +---+
>> |_c0|
>> +---+
>> | 0| // This does not
>> +---+
>> Finished at
>> [20/06/2016 21:22:30.30]
>>
>> Any ideas if anyone has seen this before?
>>
>>
>> The issue is saving data. Saving through tempTable works but the other one
>> does not work.
>>
>>
>> Thanks
>>
>> Dr Mich Talebzadeh
>>
>> LinkedIn
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>
>>
>> http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
>>
>
>
