, also make sure your mode is append rather than overwrite.
>
> Ed
>
> --
> *From:* Siavash Namvar
> *Sent:* Wednesday, August 12, 2020 4:09:07 PM
> *To:* Sean Owen
> *Cc:* User
> *Subject:* Re: How can I use pyspark to upsert one row without re
use pyspark to upsert one row without replacing entire
table
Thanks Sean,
Do you have any URL or reference to help me how to upsert in Spark? I need to
update Sybase db
On Wed, Aug 12, 2020 at 11:06 AM Sean Owen
mailto:sro...@gmail.com>> wrote:
It's not so much Spark but the
The delta docs have examples of upserting:
https://docs.delta.io/0.4.0/delta-update.html#upsert-into-a-table-using-merge
> On Aug 12, 2020, at 08:31, Siavash Namvar wrote:
>
>
> Thanks Sean,
>
> Do you have any URL or reference to help me how to upsert in Spark? I need to
> update Sybase db
Thanks Sean,
Do you have any URL or reference to help me how to upsert in Spark? I need
to update Sybase db
On Wed, Aug 12, 2020 at 11:06 AM Sean Owen wrote:
> It's not so much Spark but the data format, whether it supports
> upserts. Parquet, CSV, JSON, etc would not.
> That is what Delta, Hud
It's not so much Spark but the data format, whether it supports
upserts. Parquet, CSV, JSON, etc would not.
That is what Delta, Hudi et al are for, and yes you can upsert them in Spark.
On Wed, Aug 12, 2020 at 9:57 AM Siavash Namvar wrote:
>
> Hi,
>
> I have a use case, and read data from a db ta
