Syed Shameerur Rahman created YARN-11664:
--------------------------------------------
Summary: Remove HDFS Binaries/Jars Dependency From Yarn
Key: YARN-11664
URL: https://issues.apache.org/jira/browse/YARN-11664
Project: Hadoop YARN
Issue Type: Improvement
Components: yarn
Reporter: Syed Shameerur Rahman
In principle Hadoop Yarn is independent of HDFS. It can work with any
filesystem. Currently there exists some code dependency for Yarn with HDFS.
This dependency requires Yarn to bring in some of the HDFS binaries/jars to its
class path. The idea behind this jira is to remove this dependency so that Yarn
can run without HDFS binaries/jars
*Scope*
1. Non test classes are considered
2. Some test classes which comes as transitive dependency are considered
*Out of scope*
1. All test classes in Yarn module is not considered
--------------------------------------------------------------------------------------------
A quick search in Yarn module revealed following HDFS dependencies
1. Constants
{code:java}
import
org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier;
import org.apache.hadoop.hdfs.DFSConfigKeys;{code}
2. Exception
{code:java}
import org.apache.hadoop.hdfs.protocol.DSQuotaExceededException;
import org.apache.hadoop.hdfs.protocol.QuotaExceededException; (Comes as a
transitive dependency from DSQuotaExceededException){code}
3. Utility
{code:java}
import org.apache.hadoop.hdfs.protocol.datatransfer.IOStreamPair;{code}
Both Yarn and HDFS depends on hadoop-common module, One straight forward
approach is to move all these dependencies to hadoop-common module and both
HDFS and Yarn can pick these imports.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
