Hi Jeetendra, I am not sure if I understand your question correctly.
Arrow is an in-memory columnar data format, and Spark has its own in-memory data format for DataFrame, which is invisible to end users. So the Spark user has no control over the underlying in-memory layout. If you really want to convert a DataFrame into Arrow format, maybe you can save the results of a Spark job to some external store (e.g. in ORC format), and then load it back to memory in Arrow format (if this is what you want). Best, Liya Fan On Thu, Dec 5, 2019 at 5:53 PM Jeetendra Kumar Jaiswal <jeetendra.jais...@impetus.co.in.invalid> wrote: > Hi Dev Team, > > Can someone please let me know how to convert spark data frame to Arrow > format. I am coding in Java. > > Java documentation of Arrow just has function API information. It is > little hard to develop without proper documentation. > > Is there a way to directly convert spark dataframe to Arrow format > dataframes. > > Thanks, > Jeetendra > > ________________________________ > > > > > > > NOTE: This message may contain information that is confidential, > proprietary, privileged or otherwise protected by law. The message is > intended solely for the named addressee. If received in error, please > destroy and notify the sender. Any use of this email is prohibited when > received in error. Impetus does not represent, warrant and/or guarantee, > that the integrity of this communication has been maintained nor that the > communication is free of errors, virus, interception or interference. >
