Hello Spark developers,
I'm from the Apache Arrow project. We've discussed Java version support [1],
and crucially, whether to continue supporting Java 8 or not. As Spark is a big
user of Arrow in Java, I was curious what Spark's policy here was.
If Spark intends to stay on Java 8, for instance
d in open JDK until 2026, I'm not sure if we're
>>>> going to see enough folks moving to JRE17 by the Spark 4 release unless we
>>>> have a strong benefit from dropping 11 support I'd be inclined to keep it.
>>>>
>>>> On Tue, Jun 6, 2023
Hello all,
We've been working with PySpark and Pandas, and have found that to
convert a dataset using N bytes of memory to Pandas, we need to have
2N bytes free, even with the Arrow optimization enabled. The
fundamental reason is ARROW-3789[1]: Arrow does not free the Arrow
table until conversion
