[
https://issues.apache.org/jira/browse/HADOOP-18876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-18876.
-------------------------------------
Fix Version/s: 3.4.0
(was: 3.3.6)
Release Note:
The default value for fs.azure.data.blocks.buffer is changed from "disk" to
"bytebuffer"
This will speed up writing to azure storage, at the risk of running out of
memory
-especially if there are many threads writing to abfs at the same time and the
upload bandwidth is limited.
If jobs do run out of memory writing to abfs, change the option back to "disk"
Resolution: Fixed
> ABFS: Change default from disk to bytebuffer for fs.azure.data.blocks.buffer
> ----------------------------------------------------------------------------
>
> Key: HADOOP-18876
> URL: https://issues.apache.org/jira/browse/HADOOP-18876
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
> Affects Versions: 3.3.6
> Reporter: Anmol Asrani
> Assignee: Anmol Asrani
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Change default from disk to bytebuffer for fs.azure.data.blocks.buffer.
> Gathered from multiple workload runs, the presented data underscores a
> noteworthy enhancement in performance. The adoption of ByteBuffer for
> *reading operations* exhibited a remarkable improvement of approximately
> *64.83%* when compared to traditional disk-based reading. Similarly, the
> implementation of ByteBuffer for *write operations* yielded a substantial
> efficiency gain of about {*}60.75%{*}. These findings underscore the
> consistent and substantial advantages of integrating ByteBuffer across a
> range of workload scenarios.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
