isurika created HDFS-15570:
------------------------------
Summary: Hadoop Erasure Coding ISA-L Check Request..
Key: HDFS-15570
URL: https://issues.apache.org/jira/browse/HDFS-15570
Project: Hadoop HDFS
Issue Type: Test
Components: erasure-coding, hadoop-client
Affects Versions: 3.2.1
Environment: *-OS Version* : CentOS Linux release 8.0.1905 (Core)
*-Hadoop Version* : hadoop-3.2.1
*-CPU INFO (namenode :2 / datanode : 5)*
---------------------------------------------
namenode cpu : Intel(R) Xeon(R) CPU E5-2609 v2 @ 2.50GHz
datanode cpu : Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
---------------------------------------------
*-hadoop checknative (After source build)*
---------------------------------------------
Native library checking:
hadoop: true /nds/hadoop-3.2.1/lib/native/libhadoop.so.1.0.0
zlib: true /lib64/libz.so.1
zstd : false
snappy: true /lib64/libsnappy.so.1
lz4: true revision:10301
bzip2: false
openssl: true /lib64/libcrypto.so
{color:#0747a6}*ISA-L: true /lib64/libisal.so.2*{color}
---------------------------------------------
*-hdfs ec -listCodecs*
Erasure Coding Codecs: Codec [Coder List]
RS [RS_NATIVE, RS_JAVA]
RS-LEGACY [RS-LEGACY_JAVA]
XOR [XOR_NATIVE, XOR_JAVA]
Reporter: isurika
Fix For: 3.2.1
I am testing the performance of erasure coding in Hadoop 3.2.1 version
environment.
Apply ISA-L by referring to the manual
(https://hadoop.apache.org/docs/r3.2.1/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html)
*▶ JOB LIST (Proceed in manual order)*
# ISA-L library installed on all servers
ld ISA-L library : https://github.com/01org/isa-l
# hadoop build source (hadoop-3.2.1-src.tar.gz)
# mvn package -Pdist,native -Dtar -Drequire.isal -Drequire.snappy
# Deploy the built hadoop-3.2.1 folder to all servers
But In Test (File upload Time/Select Orc table in Hive)
There is no improvement in performance when tested.
Ask for advice on work
*[Question 1]*
Is there anything wrong with the installation? Are there any missing tasks?
*[Question 2]*
Why doesn't it speed up ? (File upload Time/Select Orc table in Hive)
*[Question 3]*
Whether to use ISA-L in Hadoop, How to check?
※ When I used the "hdfs ec -listCodecs" command, I expected to see ISA-L. But
no
※ The warn log that occurs before applying ISA-L is It no longer occurs.
----------------------------------------------
WARN org.apache.hadoop.io.erasurecode.ErasureCodeNative: ISA-L support is not
available in your platform... using builtin-java codec where applicable
----------------------------------------------
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
