Hi all,
I have a Dataframe with 1000 records. I want to split them into 100
each and post to rest API.
If it was RDD, I could use something like this
myRDD.foreachRDD {
rdd =>
rdd.foreachPartition {
partition => {
This will ensure that code is executed on executors and not on driver.
Is there any similar approach that we can take for Dataframes? I see
examples on stackoverflow with collect() which will bring whole data
to driver.
Thanks and Regards
Noorul
---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]
