Yes, actually even if you don’t set it to true, on-disk data is compressed.
(This setting only affects serialized data in memory).
Matei
On Jun 11, 2014, at 2:56 PM, Surendranauth Hiraman
wrote:
> Hi,
>
> Will spark.rdd.compress=true enable compression when using DISK_ONLY
> persistence?
>
Hi,
Will spark.rdd.compress=true enable compression when using DISK_ONLY
persistence?
SUREN HIRAMAN, VP TECHNOLOGY
Velos
Accelerating Machine Learning
440 NINTH AVENUE, 11TH FLOOR
NEW YORK, NY 10001
O: (917) 525-2466 ext. 105
F: 646.349.4063
E: suren.hiraman@v elos.io
W: www.velos.io
