Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/11805#discussion_r58131282
--- Diff:
core/src/main/scala/org/apache/spark/util/io/ChunkedByteBufferOutputStream.scala
---
@@ -18,19 +18,25 @@
package org.apache.spark.util.io
import java.io.OutputStream
+import java.nio.ByteBuffer
import scala.collection.mutable.ArrayBuffer
+import org.apache.spark.storage.StorageUtils
/**
* An OutputStream that writes to fixed-size chunks of byte arrays.
*
* @param chunkSize size of each chunk, in bytes.
*/
-private[spark]
-class ByteArrayChunkOutputStream(chunkSize: Int) extends OutputStream {
+private[spark] class ChunkedByteBufferOutputStream(
+ chunkSize: Int,
+ allocator: Int => ByteBuffer)
--- End diff --
seems like this will always be either `ByteBuffer.allocate` or
`Platform.allocateDirectBuffer`. It might be better to pass in a boolean here
instead `offHeap: Boolean = false` or something and do
```
def allocate(size: Int): ByteBuffer = {
if (offHeap) Platform.allocateDirectBuffer(size) else
ByteBuffer.allocate(size)
}
```
then you don't need to pass a closure around
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
