Hi Dimitri,
you can do the following:
1. create an initial dataframe from an empty csv
2. use "union" to insert new rows
Do not forget that Spark cannot replace a DBMS. Spark is mainly be used
for analytics.
If you need select/insert/delete/update capabilities, perhaps you should
look at a DBMS.
Another alternative, in case you need "append only" semantics, is to use
streaming or structured streaming.
regards,
Apostolos
On 30/06/2018 05:46 μμ, dimitris plakas wrote:
I am new to Pyspark and want to initialize a new empty dataframe with
sqlContext() with two columns ("Column1", "Column2"), and i want to
append rows dynamically in a for loop.
Is there any way to achieve this?
Thank you in advance.
--
Apostolos N. Papadopoulos, Associate Professor
Department of Informatics
Aristotle University of Thessaloniki
Thessaloniki, GREECE
tel: ++0030312310991918
email: papad...@csd.auth.gr
twitter: @papadopoulos_ap
web: http://delab.csd.auth.gr/~apostol
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
