Hi,
I'm puzzling over the following problem: when I cache a small sample of a
big dataframe, the small dataframe is recomputed when selecting a column
(but not if show() or count() is invoked).
Why is that so and how can I avoid recomputation of the small sample
dataframe?
More details:
- I have a big dataframe "df" of ~190million rows and ~10 columns, obtained
via 3 different joins; I cache it and invoke count() to make sure it really
is in memory and confirm in web UI
- val sdf = df.sample(false, 1e-6); sdf.cache(); sdf.count() // 170 lines;
cached is also confirmed in webUI, size in memory is 150kB
*- sdf.select("colname").show() // this triggers a complete recomputation
of sdf with 3 joins!*
- show(), count() or take() do not trigger the recomputation of the 3
joins, but select(), collect() or withColumn() do.
I have --executor-memory 30G --driver-memory 10g, so memory is not a
problem. I'm using Spark 1.4.0. Could anybody shed some light on this or
where I can find more info?
Many thanks,
Kristina
