I think you can use Spark Connect's way and represent DataFrame as a LogicalPlan that can be serialized to the protobuf message and back.
On Tue, 2024-11-05 at 16:46 +0100, vincent gromakowski wrote: > Hi spark users, > I am looking for a way to access the binary representation of a > Dataframe to implement my own checkpointing without involving serdes. > Any idea on how to do that? > > Vincent --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
