There isn't a direct conversion to/from Spark, I made
https://issues.apache.org/jira/browse/SPARK-29040 a while ago for
conversion to Spark from an Arrow table. If possible, make a comment there
for your use case which might help get support for it.
Bryan
On Mon, Aug 31, 2020, 9:12 PM Micah Kornf
Hi Radu,
I'm not a spark expert, but I haven't seen any documentation on direct
conversion. You might be better off asking the user@spark or dev@spark
mailing lists.
Thanks,
Micah
On Wed, Aug 26, 2020 at 1:46 PM Radu Teodorescu
wrote:
> Hi,
> I noticed that arrow is mentioned as an optional i
Hi,
I noticed that arrow is mentioned as an optional intermediary format for
converting between pandas DFs and spark DFs. Is there a way to explicitly
convert an pyarrow Table to a spark DataFrame and the other way around.
Absent that, going pysprak->pandas->pyarrow and back works but it’s obviou
