Re: Access binary representation of Dataframe

2024-11-05 Thread Sem
I think you can use Spark Connect's way and represent DataFrame as a LogicalPlan that can be serialized to the protobuf message and back. On Tue, 2024-11-05 at 16:46 +0100, vincent gromakowski wrote: > Hi spark users, > I am looking for a way to access the binary representation of a > Dataframe to

Access binary representation of Dataframe

2024-11-05 Thread vincent gromakowski
Hi spark users, I am looking for a way to access the binary representation of a Dataframe to implement my own checkpointing without involving serdes. Any idea on how to do that? Vincent