Hello, At the moment for the upcoming release of Spark 2.3, Arrow support is limited to using PySpark DataFrame.toPandas(), so reading Parquet data from Spark does not use Arrow.
-Bryan On Wed, Aug 30, 2017 at 2:47 AM, big data <[email protected]> wrote: > I want to use Arrow as a middle layer between spark and parquet data in > HDFS, but I don't find any docs about how to load parquet data to arrow > in memory, and how spark read arrow data format. Does anyone provide > some examples or manuals to describe it? > > thanks. > >
