Re: Spark DataFrame <--> Arrow Roundtrip

2018-02-06 Thread Li Jin
Hi Michael, Please see my comments inline. Keep in mind these are all pretty internal APIs so they might change in the future. On Mon, Feb 5, 2018 at 11:30 AM, Michael Shtelma wrote: > Hi all, > > I would like to make some changes (updates) to the data stored in > Spark data frames, which I ge

Spark DataFrame <--> Arrow Roundtrip

2018-02-05 Thread Michael Shtelma
Hi all, I would like to make some changes (updates) to the data stored in Spark data frames, which I get as a result of different queries. Afterwards, I would like to operate with these changed data frames as with normal data frames in Spark, e.g. use them for further transformations. I would lik