Alternatively, you should be able to write to a new table and use trigger
or some other mechanism to update the particular row. I don't have any
experience with this myself but just looking at this documentation:
https://docs.cloud.databricks.com/docs/latest/databricks_guide/index.html#03%20Data%20
in that case one alternative would be to save the new table on hdfs and
then using some simple ETL load it into a staging table in MySQL and
update the original table from staging table
The whole thing can be done in a shell script.
HTH
Dr Mich Talebzadeh
LinkedIn *
https://www.linkedin.com
I read the table via spark SQL , and perform some ML activity on the data
, and the resultant will be to update some specific columns with the ML
improvised result,
hence i do not have a option to do the whole operation in MySQL,
Thanks,
Sujeet
On Thu, Aug 11, 2016 at 3:29 PM, Mich Talebzadeh
Ok it is clearer now.
You are using Spark as the query tool on an RDBMS table? Read table via
JDBC, write back updating certain records.
I have not done this myself but I suspect the issue would be if Spark write
will commit the transaction and maintains ACID compliance. (locking the
rows etc).
1 ) using mysql DB
2 ) will be inserting/update/overwrite to the same table
3 ) i want to update a specific column in a record, the data is read via
Spark SQL,
on the below table which is read via sparkSQL, i would like to update the
NumOfSamples column .
consider DF as the dataFrame which holds
Hi,
1. what is the underlying DB, say Hive etc
2. Is table transactional or you are going to do insert/overwrite to the
same table
3. can you do all this in the database itself assuming it is an RDBMS
4. Can you provide the sql or pseudo code for such an update
HTH
Dr Mich Taleb
Hi,
Is it possible to update certain columnr records in DB from spark,
for example i have 10 rows with 3 columns which are read from Spark SQL,
i want to update specific column entries and write back to DB, but since
RDD"s are immutable i believe this would be difficult, is there a
workaroun
