Client Caller Context Through Router(RBF)

[Ayush Saxena]

Hi All,
Wanted to share and discuss a problem that we are facing in the present
situation when using Router Based Federation. Presently when a client
connects through Router to Namenode, the Namenode receives the caller
context of the router rather than being of the actual client. This
typically can cause a couple of problems, Two of which we have identified
as of now :

Firstly, The concept of data locality doesn't work correctly when
connecting through Router as the Namenode considers Router as the actual
client and performs all the optimizations/computations based on Router's
location rather than using the actual client location.

Secondly, The Namenode Retry Cache can not be used as if in case of
failover or such an event, the client retries again and connects to other
router, in that case the since the Call Id is from the Router, but not from
the actual client, the Retry Cache doesn't identify it as a repeated call
and serves it as a whole new call which creates inconsistencies.

We have been discussing and trying on solutions since a long time now and
tried out a couple of solutions :

   - Add proxy address in IPC connection (HADOOP-16254
   ) --> This had some
   security concerns for Daryn.
   - The RouterRPCServer should transfer CallerContext and client ip to
   NamenodeRpcServer (HDFS-13293
   ) --> This tend to be
   little opaque and couple of more problems stated as in HDFS-13248
    by Ajay Kumar and
   Arpit Agarwal
   - Favored Nodes -->  Pass the local node as favored node. But this isn't
   a complete solution. This doesn't take into account the fallback in case of
   non availability of local nodes and couple of more. this isn't a solution
   for the Retry Cache problem too.


The related JIRA's where most of the discussion happened, if someone tends
to follow :
HDFS-13248  :- For the
DataLocality Problem. Has a patch too in the end with Solution 3(Favored
Nodes)
HDFS-15079  , HDFS-15078
 & HDFS-15310
  : For the Retry Cache
Problem.
HADOOP-16254  :
Solution 1 : Add proxy address in IPC connection.
HDFS-13293  : Solution 2
: Passing Caller Context.

Do let us know if any help here, Any further solutions, workarounds or a
way out to unblock or improvise the tried solutions.

Thanx!!!
-Ayush

Apache Hadoop qbt Report: branch2.10+JDK7 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86/675/

No changes




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml
 
   hadoop-tools/hadoop-azure/src/config/checkstyle-suppressions.xml 
   hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml
 

FindBugs :

   module:hadoop-common-project/hadoop-minikdc 
   Possible null pointer dereference in 
org.apache.hadoop.minikdc.MiniKdc.delete(File) due to return value of called 
method Dereferenced at 
MiniKdc.java:org.apache.hadoop.minikdc.MiniKdc.delete(File) due to return value 
of called method Dereferenced at MiniKdc.java:[line 515] 

FindBugs :

   module:hadoop-common-project/hadoop-auth 
   
org.apache.hadoop.security.authentication.server.MultiSchemeAuthenticationHandler.authenticate(HttpServletRequest,
 HttpServletResponse) makes inefficient use of keySet iterator instead of 
entrySet iterator At MultiSchemeAuthenticationHandler.java:of keySet iterator 
instead of entrySet iterator At MultiSchemeAuthenticationHandler.java:[line 
192] 

FindBugs :

   module:hadoop-common-project/hadoop-common 
   org.apache.hadoop.crypto.CipherSuite.setUnknownValue(int) 
unconditionally sets the field unknownValue At CipherSuite.java:unknownValue At 
CipherSuite.java:[line 44] 
   org.apache.hadoop.crypto.CryptoProtocolVersion.setUnknownValue(int) 
unconditionally sets the field unknownValue At 
CryptoProtocolVersion.java:unknownValue At CryptoProtocolVersion.java:[line 67] 
   Possible null pointer dereference in 
org.apache.hadoop.fs.FileUtil.fullyDeleteOnExit(File) due to return value of 
called method Dereferenced at 
FileUtil.java:org.apache.hadoop.fs.FileUtil.fullyDeleteOnExit(File) due to 
return value of called method Dereferenced at FileUtil.java:[line 118] 
   Possible null pointer dereference in 
org.apache.hadoop.fs.RawLocalFileSystem.handleEmptyDstDirectoryOnWindows(Path, 
File, Path, File) due to return value of called method Dereferenced at 
RawLocalFileSystem.java:org.apache.hadoop.fs.RawLocalFileSystem.handleEmptyDstDirectoryOnWindows(Path,
 File, Path, File) due to return value of called method Dereferenced at 
RawLocalFileSystem.java:[line 383] 
   Useless condition:lazyPersist == true at this point At 
CommandWithDestination.java:[line 502] 
   org.apache.hadoop.io.DoubleWritable.compareTo(DoubleWritable) 
incorrectly handles double value At DoubleWritable.java: At 
DoubleWritable.java:[line 78] 
   org.apache.hadoop.io.DoubleWritable$Comparator.compare(byte[], int, int, 
byte[], int, int) incorrectly handles double value At DoubleWritable.java:int) 
incorrectly handles double value At DoubleWritable.java:[line 97] 
   org.apache.hadoop.io.FloatWritable.compareTo(FloatWritable) incorrectly 
handles float value At FloatWritable.java: At FloatWritable.java:[line 71] 
   org.apache.hadoop.io.FloatWritable$Comparator.compare(byte[], int, int, 
byte[], int, int) incorrectly handles float value At FloatWritable.java:int) 
incorrectly handles float value At FloatWritable.java:[line 89] 
   Possible null pointer dereference in 
org.apache.hadoop.io.IOUtils.listDirectory(File, FilenameFilter) due to return 
value of called method Dereferenced at 
IOUtils.java:org.apache.hadoop.io.IOUtils.listDirectory(File, FilenameFilter) 
due to return value of called method Dereferenced at IOUtils.java:[line 389] 
   Possible bad parsing of shift operation in 
org.apache.hadoop.io.file.tfile.Utils$Version.hashCode() At 
Utils.java:operation in 
org.apache.hadoop.io.file.tfile.Utils$Version.hashCode() At Utils.java:[line 
398] 
   
org.apache.hadoop.metrics2.lib.DefaultMetricsFactory.setInstance(MutableMetricsFactory)
 unconditionally sets the field mmfImpl At DefaultMetricsFactory.java:mmfImpl 
At DefaultMetricsFactory.java:[line 49] 
   
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.setMiniClusterMode(boolean) 
unconditionally sets the field miniClusterMode At 
DefaultMetricsSystem.java:miniClusterMode At DefaultMetricsSystem.java:[line 
92] 
   Useless object stored in variable seqOs of method 
org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.addOrUpdateToken(AbstractDelegationTokenIdentifier,
 AbstractDelegationTokenSecretManager$DelegationTokenInformation, boolean) At 
ZKDelegationTokenSecretManager.java:seqOs of method 
org.apache.hadoop.

[jira] [Created] (HDFS-15331) Remove invalid exclusions that minicluster dependency on HDFS

[Wanqiang Ji (Jira)]

Wanqiang Ji created HDFS-15331:
--

 Summary: Remove invalid exclusions that minicluster dependency on 
HDFS
 Key: HDFS-15331
 URL: https://issues.apache.org/jira/browse/HDFS-15331
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Wanqiang Ji
Assignee: Wanqiang Ji


Ozone has split into independent repo, but the invalid exclusions (kubernetes 
client) that minicluster dependency on HDFS is kept.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/130/

[May 3, 2020 3:56:29 AM] (Mingliang Liu) HDFS-15325. TestRefreshCallQueue is 
failing due to changed CallQueue constructor. Contributed by Fengnan Li
[May 3, 2020 1:39:14 PM] (Ayush Saxena) Erasure Coding: metrics xmitsInProgress 
become to negative. Contributed by maobaolong and Toshihiko Uchida.
[May 3, 2020 6:29:17 PM] (Ayush Saxena) HDFS-15328. Use DFSConfigKeys  
MONITOR_CLASS_DEFAULT  constant. Contributed by bianqi.




-1 overall


The following subsystems voted -1:
asflicense findbugs mvnsite pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

findbugs :

   module:hadoop-yarn-project/hadoop-yarn 
   Uncallable method 
org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage$1.getInstance()
 defined in anonymous class At 
TestTimelineReaderWebServicesHBaseStorage.java:anonymous class At 
TestTimelineReaderWebServicesHBaseStorage.java:[line 87] 
   Dead store to entities in 
org.apache.hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown.checkQuery(HBaseTimelineReaderImpl)
 At 
TestTimelineReaderHBaseDown.java:org.apache.hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown.checkQuery(HBaseTimelineReaderImpl)
 At TestTimelineReaderHBaseDown.java:[line 190] 
   org.apache.hadoop.yarn.server.webapp.WebServiceClient.sslFactory should 
be package protected At WebServiceClient.java: At WebServiceClient.java:[line 
42] 

findbugs :

   module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server 
   Uncallable method 
org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage$1.getInstance()
 defined in anonymous class At 
TestTimelineReaderWebServicesHBaseStorage.java:anonymous class At 
TestTimelineReaderWebServicesHBaseStorage.java:[line 87] 
   Dead store to entities in 
org.apache.hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown.checkQuery(HBaseTimelineReaderImpl)
 At 
TestTimelineReaderHBaseDown.java:org.apache.hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown.checkQuery(HBaseTimelineReaderImpl)
 At TestTimelineReaderHBaseDown.java:[line 190] 
   org.apache.hadoop.yarn.server.webapp.WebServiceClient.sslFactory should 
be package protected At WebServiceClient.java: At WebServiceClient.java:[line 
42] 

findbugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
 
   org.apache.hadoop.yarn.server.webapp.WebServiceClient.sslFactory should 
be package protected At WebServiceClient.java: At WebServiceClient.java:[line 
42] 

findbugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
 
   Uncallable method 
org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage$1.getInstance()
 defined in anonymous class At 
TestTimelineReaderWebServicesHBaseStorage.java:anonymous class At 
TestTimelineReaderWebServicesHBaseStorage.java:[line 87] 
   Dead store to entities in 
org.apache.hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown.checkQuery(HBaseTimelineReaderImpl)
 At 
TestTimelineReaderHBaseDown.java:org.apache.hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown.checkQuery(HBaseTimelineReaderImpl)
 At TestTimelineReaderHBaseDown.java:[line 190] 

findbugs :

   module:hadoop-yarn-project 
   Uncallable method 
org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage$1.getInstance()
 defined in anonymous class At 
TestTimelineReaderWebServicesHBaseStorage.java:anonymous class At 
TestTimelineReaderWebServicesHBaseStorage.java:[line 87] 
   Dead store t

[jira] [Created] (HDFS-15332) Quota Space consumed was wrong in truncate with Snapshots

[hemanthboyina (Jira)]

hemanthboyina created HDFS-15332:


 Summary: Quota Space consumed was wrong in truncate with Snapshots
 Key: HDFS-15332
 URL: https://issues.apache.org/jira/browse/HDFS-15332
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: hemanthboyina
Assignee: hemanthboyina






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-15270) Account for *env == NULL in hdfsThreadDestructor

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-15270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HDFS-15270.

Fix Version/s: 3.4.0
   Resolution: Fixed

Thanks [~babsingh] this is in the trunk. Do you have a branch in mind that you 
want this cherrypicked to?

> Account for *env == NULL in hdfsThreadDestructor
> 
>
> Key: HDFS-15270
> URL: https://issues.apache.org/jira/browse/HDFS-15270
> Project: Hadoop HDFS
>  Issue Type: Bug
> Environment: Please refer to the "steps to reproduce" the failure in 
> https://github.com/eclipse/openj9/issues/7752#issue-521732953.
>Reporter: Babneet Singh
>Assignee: Babneet Singh
>Priority: Major
> Fix For: 3.4.0
>
>
> OpenJ9 JVM properly terminates the thread before hdfsThreadDestructor is
> invoked. JNIEnv is a mirror of J9VMThread in OpenJ9. After proper thread
> termination, accessing JNIEnv in hdfsThreadDestructor (*env)->GetJavaVM,
> yields a SIGSEGV since *env is NULL after thread cleanup is performed.
> The main purpose of hdfsThreadDestructor is to invoke
> DetachCurrentThread, which performs thread cleanup in OpenJ9. Since
> OpenJ9 performs thread cleanup before hdfsThreadDestructor is invoked,
> hdfsThreadDestructor should account for *env == NULL and skip
> DetachCurrentThread.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[VOTE] Release Apache Hadoop 3.1.4 (RC0)

[Gabor Bota]

Hi folks,

I have put together a release candidate (RC0) for Hadoop 3.1.4.

The RC is available at: http://people.apache.org/~gabota/hadoop-3.1.4-RC0/
The RC tag in git is here:
https://github.com/apache/hadoop/releases/tag/release-3.1.4-RC0
The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1266/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
and http://keys.gnupg.net/pks/lookup?op=get&search=0xB86249D83539B38C

Please try the release and vote. The vote will run for 5 weekdays,
until May 11. 2020. 23:00 CET.

Thanks,
Gabor

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.1.4 (RC0)

[Wei-Chiu Chuang]

Gabor, I'm sorry there's a test failure in branch-3.1 HDFS-14599


   1.
  1. I just cherrypicked the fix to branch-3.2 branch-3.1. It's a test
  only fix so technically I could live with it. But it would be best to add
  the fix to 3.1.4 as well.


On Mon, May 4, 2020 at 3:20 PM Gabor Bota  wrote:

> Hi folks,
>
> I have put together a release candidate (RC0) for Hadoop 3.1.4.
>
> The RC is available at: http://people.apache.org/~gabota/hadoop-3.1.4-RC0/
> The RC tag in git is here:
> https://github.com/apache/hadoop/releases/tag/release-3.1.4-RC0
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1266/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> and http://keys.gnupg.net/pks/lookup?op=get&search=0xB86249D83539B38C
>
> Please try the release and vote. The vote will run for 5 weekdays,
> until May 11. 2020. 23:00 CET.
>
> Thanks,
> Gabor
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>