Steve Loughran created HADOOP-18184:
---------------------------------------
Summary: s3a prefetching stream to support unbuffer()
Key: HADOOP-18184
URL: https://issues.apache.org/jira/browse/HADOOP-18184
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.4.0
Reporter: Steve Loughran
Apache Impala uses unbuffer() to free up all client side resources held by a
stream, so allowing it to have a map of available (path -> stream) objects,
retained across queries.
This saves on having to reopen the files, with the cost of HEAD checks etc.
S3AInputStream just closes its http connection. here there is a lot more state
to discard, but all memory and file storage must be freed.
until this done, ITestS3AContractUnbuffer must skip when the prefetch stream is
used.
its notable that the other tests don't fail, even though the stream doesn't
implement the interface; the graceful degradation handles that. it should fail
if the test xml resource says the stream does it, but that the stream
capabilities say it doesn't.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
