[
https://issues.apache.org/jira/browse/SPARK-19266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15827315#comment-15827315
]
Marcelo Vanzin commented on SPARK-19266:
----------------------------------------
Hmm... seems {{partiallySerializedValues.finishWritingToStream}} will actually
keep writing encrypted data when it's called (it uses a
{{RedirectableOutputStream}} which just replaces the byte array stream with a
file stream, keeping the existing encryption and compression state). But I
can't find where that spilled data is read, and I'm not sure whether that's
properly covered by existing tests.
I'll try to write a test to exercise this path; worst case we have a new test
that makes sure it works.
> DiskStore does not encrypt serialized RDD data
> ----------------------------------------------
>
> Key: SPARK-19266
> URL: https://issues.apache.org/jira/browse/SPARK-19266
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.1.0
> Reporter: Marcelo Vanzin
>
> {{DiskStore.putBytes()}} writes serialized RDD data directly to disk, without
> encrypting (or compressing) it. So any cached blocks that are evicted to disk
> when using {{MEMORY_AND_DISK_SER}} will not be encrypted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
