[
https://issues.apache.org/jira/browse/ARROW-501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593968#comment-16593968
]
Wes McKinney commented on ARROW-501:
------------------------------------
We should think up front about optimizations to enable temporary allocations to
be reused. For example, suppose you are reading 1MB at a time, then the spooler
could allocated 2 buffers so that while one buffer is being used, the other one
can be filled in the background. We would need to have an API to be able to
indicate that a buffer is "done" and can be reused (and the user would
therefore be responsible to not retain any references to that memory elsewhere
since it will be overwritten). I can try to mock up some example code if what
I'm writing is not clear
> [C++] Implement concurrent / buffering InputStream for streaming data use
> cases
> -------------------------------------------------------------------------------
>
> Key: ARROW-501
> URL: https://issues.apache.org/jira/browse/ARROW-501
> Project: Apache Arrow
> Issue Type: New Feature
> Components: C++
> Reporter: Wes McKinney
> Priority: Major
> Labels: csv
> Fix For: 0.13.0
>
>
> Related to ARROW-500, when processing an input data stream, we may wish to
> continue buffering input (up to an maximum buffer size) in between
> synchronous Read calls
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
