Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/135/

No changes




-1 overall


The following subsystems voted -1:
asflicense hadolint jshint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml 
   hadoop-build-tools/src/main/resources/checkstyle/suppressions.xml 
   
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml
 
   hadoop-tools/hadoop-azure/src/config/checkstyle-suppressions.xml 
   hadoop-tools/hadoop-azure/src/config/checkstyle.xml 
   hadoop-tools/hadoop-resourceestimator/src/config/checkstyle.xml 
   hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml
 

Failed junit tests :

   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   
hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain 
   hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat 
   hadoop.hdfs.server.federation.resolver.order.TestLocalResolver 
   hadoop.hdfs.server.federation.router.TestRouterQuota 
   hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver 
   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer 
   hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher 
   hadoop.yarn.server.resourcemanager.TestClientRMService 
   hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter 
   hadoop.tools.TestDistCpSystem 
   hadoop.yarn.sls.appmaster.TestAMSimulator 
   hadoop.yarn.sls.TestSLSRunner 
   hadoop.resourceestimator.service.TestResourceEstimatorService 
   hadoop.resourceestimator.solver.impl.TestLpSolver 
  

   jshint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/135/artifact/out/diff-patch-jshint.txt
  [208K]

   cc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/135/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/135/artifact/out/diff-compile-javac-root.txt
  [456K]

   checkstyle:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/135/artifact/out/diff-checkstyle-root.txt
  [16M]

   hadolint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/135/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   pathlen:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/135/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/135/artifact/out/diff-patch-pylint.txt
  [60K]

   shellcheck:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/135/artifact/out/diff-patch-shellcheck.txt
  [56K]

   shelldocs:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/135/artifact/out/diff-patch-shelldocs.txt
  [8.0K]

   whitespace:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/135/artifact/out/whitespace-eol.txt
  [12M]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/135/artifact/out/whitespace-tabs.txt
  [1.3M]

   xml:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/135/artifact/out/xml.txt
  [4.0K]

   javadoc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/135/artifact/out/diff-javadoc-javadoc-root.txt
  [20K]

   unit:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/135/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [280K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/135/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt
  [12K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/135/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
  [36K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/135/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [124K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/135/artifact/out/patch-unit-hadoop-yarn-project_hadoop-ya

[jira] [Resolved] (HDFS-15703) Don't generate edits for set operations that are no-op

[Jim Brennan (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jim Brennan resolved HDFS-15703.

Fix Version/s: 3.2.3
   3.1.5
   3.4.0
   3.3.1
   Resolution: Fixed

Thanks [~daryn] and [~ahussein]!  I have committed this to trunk, branch-3.3, 
branch-3.2, and branch-3.1.

 

> Don't generate edits for set operations that are no-op
> --
>
> Key: HDFS-15703
> URL: https://issues.apache.org/jira/browse/HDFS-15703
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.3.1, 3.4.0, 3.1.5, 3.2.3
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> [~daryn] reported that setting the owner, group, or permissions to what it 
> already is will generate an unnecessary edit. It should not do this to avoid 
> performance issues when users unnecessarily run routine jobs to change the 
> group and permissions of a project tree.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/344/

[Dec 2, 2020 7:59:00 PM] (noreply) HDFS-15695. NN should not let the balancer 
run in safemode (#2489). Contributed by Daryn Sharp and Ahmed Hussein
[Dec 2, 2020 9:38:20 PM] (noreply) HDFS-15703. Don't generate edits for set 
operations that are no-op (#2508). Contributed by Daryn Sharp and Ahmed Hussein
[Dec 2, 2020 11:53:09 PM] (noreply) HDFS-14904. Add Option to let Balancer 
prefer highly utilized nodes in each iteration (#2483). Contributed by Leon Gao.




-1 overall


The following subsystems voted -1:
asflicense findbugs pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

Failed junit tests :

   hadoop.hdfs.server.namenode.ha.TestHAAppend 
   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer 
   hadoop.yarn.server.router.webapp.TestRouterWebServicesREST 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
   hadoop.tools.dynamometer.TestDynamometerInfra 
   hadoop.tools.dynamometer.TestDynamometerInfra 
  

   cc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/344/artifact/out/diff-compile-cc-root.txt
  [48K]

   javac:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/344/artifact/out/diff-compile-javac-root.txt
  [568K]

   checkstyle:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/344/artifact/out/diff-checkstyle-root.txt
  [16M]

   pathlen:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/344/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/344/artifact/out/diff-patch-pylint.txt
  [60K]

   shellcheck:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/344/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/344/artifact/out/diff-patch-shelldocs.txt
  [44K]

   whitespace:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/344/artifact/out/whitespace-eol.txt
  [13M]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/344/artifact/out/whitespace-tabs.txt
  [2.0M]

   xml:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/344/artifact/out/xml.txt
  [24K]

   javadoc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/344/artifact/out/diff-javadoc-javadoc-root.txt
  [2.0M]

   findbugs:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/344/artifact/out/branch-findbugs-hadoop-tools_hadoop-gridmix.txt
  [4.0K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/344/artifact/out/branch-findbugs-hadoop-tools_hadoop-datajoin.txt
  [0]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/344/artifact/out/branch-findbugs-hadoop-tools_hadoop-extras.txt
  [8.0K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/344/artifact/out/branch-findbugs-hadoop-tools_hadoop-pipes.txt
  [0]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/344/artifact/out/branch-findbugs-hadoop-tools_hadoop-openstack.txt
  [0]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/344/artifact/out/branch-findbugs-hadoop-tools_hadoop-aws.txt
  [0]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/344/artifact/out/branch-findbugs-root.txt
  [464K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/344/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-

[jira] [Created] (HDFS-15706) HttpFS: Log more information on request failures

[Ahmed Hussein (Jira)]

Ahmed Hussein created HDFS-15706:


 Summary: HttpFS: Log more information on request failures
 Key: HDFS-15706
 URL: https://issues.apache.org/jira/browse/HDFS-15706
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: httpfs
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


[~kihwal] reported that the exception provider does not log anything for the 
requests failing with 403 or 500. This has made debugging extremely difficult. 
As more customers are using HttpFS server, this needs an improvement:
* Turn on/off logging.
* Log the full stack trace with the exception when it is on.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

HDFS upgrade with downtime questions

[Jason Wen]

Hi,

In 
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html#Upgrade_with_Downtime,
 it says following:
In a non-HA cluster, there are a NameNode (NN), a SecondaryNameNode (SNN) and 
many DataNodes (DNs). The procedure for upgrading a non-HA cluster is similar 
to upgrading an HA cluster except that Step 2 “Upgrade Active and Standby NNs” 
is changed to below:

  *   Upgrade NN and SNN
 *   Shutdown SNN
 *   Shutdown and upgrade NN.
 *   Start NN with the “-rollingUpgrade 
started”
 option.
 *   Upgrade and restart SNN
We have a HA cluster running with Hadoop 2 and we want to upgrade it to Hadoop 
3.
Here are my questions:

  1.  Should we also same steps are the above if we want to perform upgrade 
with downtime?
  2.  If yes, when restarting standby NN as in step 4 above, should we start it 
with the “-rollingUpgrade 
started”
 option. ?

My question is for step 4 when restarting SNN, do we also need start it with 
‘-rollingUpgrade started’ option?

Thanks,
Jason

[jira] [Created] (HDFS-15707) NNTop counts don't add up as expected

[Ahmed Hussein (Jira)]

Ahmed Hussein created HDFS-15707:


 Summary: NNTop counts don't add up as expected
 Key: HDFS-15707
 URL: https://issues.apache.org/jira/browse/HDFS-15707
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs, metrics, namenode
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


When we queried 
https://host-name:port/jmx?qry=Hadoop:service=NameNode,name=FSNamesystemState 
earlier today, we received surprising results. Below is a subset of the data. 
The notable thing here is that the count for {{opType='\*'}} from user 
{{user1}} is {{66652}}, but the sum of counts for other {{optype}} values by 
{{user1}} is actually larger: {{77253}}. I'd expect that {{opType='\*'}} to 
always be greater than or equal to the sum for individual operation types.

{code}
{
  "windows": [
{
  "windowLenMs": 30,
  "ops": [
{
  "totalCount": 24158,
  "opType": "rpc.complete",
  "topUsers": [{ "count": 2944, "user": "user1" }]
},
{
  "totalCount": 15921,
  "opType": "rpc.rename",
  "topUsers": [{ "count": 2891, "user": "user1" }]
},
{
  "totalCount": 3015834,
  "opType": "*",
  "topUsers": [{ "count": 66652, "user": "user1" }]
},
{
  "totalCount": 2086,
  "opType": "rpc.abandonBlock",
  "topUsers": [{ "count": 603, "user": "user1" }]
},
{
  "totalCount": 30258,
  "opType": "rpc.addBlock",
  "topUsers": [{ "count": 3182, "user": "user1" }]
},
{
  "totalCount": 101440,
  "opType": "rpc.getServerDefaults",
  "topUsers": [{ "count": 3521, "user": "user1" }]
},
{
  "totalCount": 25258,
  "opType": "rpc.create",
  "topUsers": [{ "count": 1864, "user": "user1" }]
},
{
  "totalCount": 1377563,
  "opType": "rpc.getFileInfo",
  "topUsers": [{ "count": 56541, "user": "user1" }]
},
{
  "totalCount": 60836,
  "opType": "rpc.renewLease",
  "topUsers": [{ "count": 3783, "user": "user1" }]
},
{
  "totalCount": 182212,
  "opType": "rpc.getListing",
  "topUsers": [{ "count": 1848, "user": "user1" }]
},
{
  "totalCount": 380,
  "opType": "rpc.updateBlockForPipeline",
  "topUsers": [{ "count": 58, "user": "user1" }]
},
{
  "totalCount": 215,
  "opType": "rpc.updatePipeline",
  "topUsers": [{ "count": 18, "user": "user1" }]
}
  ]
}
  ],
  "timestamp": "2017-07-11T17:20:03+"
}
{code}

The full {{TopUserOpCounts}} results from the NNTop endpoint results are 
attached to this JIRA as {{nntop.json}}. The full endpoint output is attached 
as {{endpoint.json}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-15706) HttpFS: Log more information on request failures

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-15706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HDFS-15706.

Fix Version/s: 3.2.3
   3.1.5
   3.4.0
   3.3.1
   Resolution: Fixed

Thanks [~ahussein]

> HttpFS: Log more information on request failures
> 
>
> Key: HDFS-15706
> URL: https://issues.apache.org/jira/browse/HDFS-15706
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: httpfs
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.3.1, 3.4.0, 3.1.5, 3.2.3
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> [~kihwal] reported that the exception provider does not log anything for the 
> requests failing with 403 or 500. This has made debugging extremely 
> difficult. As more customers are using HttpFS server, this needs an 
> improvement:
> * Turn on/off logging.
> * Log the full stack trace with the exception when it is on.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.2 - RC3

[Chao Sun]

Thanks Xiaoqiao. I just committed the fix to branch-3.2 (and will port it
to other branches as well later). Could you cherry-pick it to branch-3.2.2
and start another RC? really appreciate it.

Best,
Chao

On Wed, Dec 2, 2020 at 7:59 PM Xiaoqiao He  wrote:

> Thanks Chao Sun for your quick response.
>
> If no more comments, I will wait for HADOOP-16080 to be resolved and
> prepare another RC.
> BTW, If any other issues should be involved for 3.2.2, please let me know
> ASAP.  Thanks again.
>
> Regards.
> - He Xiaoqiao
>
> On Wed, Dec 2, 2020 at 2:55 PM Chao Sun  wrote:
>
>> Thanks Xiaoqiao for the work! Unfortunately we just discovered an issue
>> (HADOOP-16080) which prevents Spark from consuming this release. I'm
>> working on a PR for this and wondering if we can include the fix in the
>> release too.
>>
>> Thanks again and apologies for the late notice.
>>
>> Best,
>> Chao
>>
>> On Mon, Nov 30, 2020 at 7:37 AM Xiaoqiao He 
>> wrote:
>>
>>> Hi folks,
>>>
>>> The release candidate (RC3) for Hadoop-3.2.2 is available now.
>>> There are 22 commits[1] differences between RC3 and RC2[2].
>>>
>>> The RC3 is available at:
>>> http://people.apache.org/~hexiaoqiao/hadoop-3.2.2-RC3
>>> The RC3 tag in github is here:
>>> https://github.com/apache/hadoop/tree/release-3.2.2-RC3
>>> The maven artifacts are staged at:
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1291
>>>
>>> You can find my public key at:
>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS or
>>> https://people.apache.org/keys/committer/hexiaoqiao.asc directly.
>>>
>>> Please try the release and vote.
>>>
>>> Thanks,
>>> He Xiaoqiao
>>>
>>> [1]
>>>
>>> https://github.com/apache/hadoop/compare/release-3.2.2-RC2...release-3.2.2-RC3
>>> [2]
>>>
>>> https://lists.apache.org/thread.html/r606fff445847bdb85bd60c5a73b2fb1f0433ee31b18c456a2231fcec%40%3Chdfs-dev.hadoop.apache.org%3E
>>> [3]
>>>
>>> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948
>>>
>>

Re: [VOTE] Release Apache Hadoop 3.2.2 - RC3

[Akira Ajisaka]

I found TestCombineFileInputFormat is failing in branch-3.2.2 and I've
cherry-picked MAPREDUCE-7284 to fix the failure.

Thanks,
Akira

On Fri, Dec 4, 2020 at 9:09 AM Chao Sun  wrote:
>
> Thanks Xiaoqiao. I just committed the fix to branch-3.2 (and will port it
> to other branches as well later). Could you cherry-pick it to branch-3.2.2
> and start another RC? really appreciate it.
>
> Best,
> Chao
>
> On Wed, Dec 2, 2020 at 7:59 PM Xiaoqiao He  wrote:
>
> > Thanks Chao Sun for your quick response.
> >
> > If no more comments, I will wait for HADOOP-16080 to be resolved and
> > prepare another RC.
> > BTW, If any other issues should be involved for 3.2.2, please let me know
> > ASAP.  Thanks again.
> >
> > Regards.
> > - He Xiaoqiao
> >
> > On Wed, Dec 2, 2020 at 2:55 PM Chao Sun  wrote:
> >
> >> Thanks Xiaoqiao for the work! Unfortunately we just discovered an issue
> >> (HADOOP-16080) which prevents Spark from consuming this release. I'm
> >> working on a PR for this and wondering if we can include the fix in the
> >> release too.
> >>
> >> Thanks again and apologies for the late notice.
> >>
> >> Best,
> >> Chao
> >>
> >> On Mon, Nov 30, 2020 at 7:37 AM Xiaoqiao He 
> >> wrote:
> >>
> >>> Hi folks,
> >>>
> >>> The release candidate (RC3) for Hadoop-3.2.2 is available now.
> >>> There are 22 commits[1] differences between RC3 and RC2[2].
> >>>
> >>> The RC3 is available at:
> >>> http://people.apache.org/~hexiaoqiao/hadoop-3.2.2-RC3
> >>> The RC3 tag in github is here:
> >>> https://github.com/apache/hadoop/tree/release-3.2.2-RC3
> >>> The maven artifacts are staged at:
> >>> https://repository.apache.org/content/repositories/orgapachehadoop-1291
> >>>
> >>> You can find my public key at:
> >>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS or
> >>> https://people.apache.org/keys/committer/hexiaoqiao.asc directly.
> >>>
> >>> Please try the release and vote.
> >>>
> >>> Thanks,
> >>> He Xiaoqiao
> >>>
> >>> [1]
> >>>
> >>> https://github.com/apache/hadoop/compare/release-3.2.2-RC2...release-3.2.2-RC3
> >>> [2]
> >>>
> >>> https://lists.apache.org/thread.html/r606fff445847bdb85bd60c5a73b2fb1f0433ee31b18c456a2231fcec%40%3Chdfs-dev.hadoop.apache.org%3E
> >>> [3]
> >>>
> >>> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948
> >>>
> >>

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.2 - RC3

[Akira Ajisaka]

In addition, backported HDFS-13404 to fix
TestRouterWebHDFSContractAppend failure.

On Fri, Dec 4, 2020 at 10:43 AM Akira Ajisaka  wrote:
>
> I found TestCombineFileInputFormat is failing in branch-3.2.2 and I've
> cherry-picked MAPREDUCE-7284 to fix the failure.
>
> Thanks,
> Akira
>
> On Fri, Dec 4, 2020 at 9:09 AM Chao Sun  wrote:
> >
> > Thanks Xiaoqiao. I just committed the fix to branch-3.2 (and will port it
> > to other branches as well later). Could you cherry-pick it to branch-3.2.2
> > and start another RC? really appreciate it.
> >
> > Best,
> > Chao
> >
> > On Wed, Dec 2, 2020 at 7:59 PM Xiaoqiao He  wrote:
> >
> > > Thanks Chao Sun for your quick response.
> > >
> > > If no more comments, I will wait for HADOOP-16080 to be resolved and
> > > prepare another RC.
> > > BTW, If any other issues should be involved for 3.2.2, please let me know
> > > ASAP.  Thanks again.
> > >
> > > Regards.
> > > - He Xiaoqiao
> > >
> > > On Wed, Dec 2, 2020 at 2:55 PM Chao Sun  wrote:
> > >
> > >> Thanks Xiaoqiao for the work! Unfortunately we just discovered an issue
> > >> (HADOOP-16080) which prevents Spark from consuming this release. I'm
> > >> working on a PR for this and wondering if we can include the fix in the
> > >> release too.
> > >>
> > >> Thanks again and apologies for the late notice.
> > >>
> > >> Best,
> > >> Chao
> > >>
> > >> On Mon, Nov 30, 2020 at 7:37 AM Xiaoqiao He 
> > >> wrote:
> > >>
> > >>> Hi folks,
> > >>>
> > >>> The release candidate (RC3) for Hadoop-3.2.2 is available now.
> > >>> There are 22 commits[1] differences between RC3 and RC2[2].
> > >>>
> > >>> The RC3 is available at:
> > >>> http://people.apache.org/~hexiaoqiao/hadoop-3.2.2-RC3
> > >>> The RC3 tag in github is here:
> > >>> https://github.com/apache/hadoop/tree/release-3.2.2-RC3
> > >>> The maven artifacts are staged at:
> > >>> https://repository.apache.org/content/repositories/orgapachehadoop-1291
> > >>>
> > >>> You can find my public key at:
> > >>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS or
> > >>> https://people.apache.org/keys/committer/hexiaoqiao.asc directly.
> > >>>
> > >>> Please try the release and vote.
> > >>>
> > >>> Thanks,
> > >>> He Xiaoqiao
> > >>>
> > >>> [1]
> > >>>
> > >>> https://github.com/apache/hadoop/compare/release-3.2.2-RC2...release-3.2.2-RC3
> > >>> [2]
> > >>>
> > >>> https://lists.apache.org/thread.html/r606fff445847bdb85bd60c5a73b2fb1f0433ee31b18c456a2231fcec%40%3Chdfs-dev.hadoop.apache.org%3E
> > >>> [3]
> > >>>
> > >>> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948
> > >>>
> > >>

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-15708) TestURLConnectionFactory fails by NoClassDefFoundError in branch-3.3 and branch-3.2

[Akira Ajisaka (Jira)]

Akira Ajisaka created HDFS-15708:


 Summary: TestURLConnectionFactory fails by NoClassDefFoundError in 
branch-3.3 and branch-3.2
 Key: HDFS-15708
 URL: https://issues.apache.org/jira/browse/HDFS-15708
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: test
Reporter: Akira Ajisaka


TestURLConnectionFactory#testSSLFactoryCleanup fails:
{noformat}
[ERROR] 
testSSLFactoryCleanup(org.apache.hadoop.hdfs.web.TestURLConnectionFactory)  
Time elapsed: 0.28 s  <<< ERROR!
java.lang.NoClassDefFoundError: org/bouncycastle/x509/X509V1CertificateGenerator
at 
org.apache.hadoop.security.ssl.KeyStoreTestUtil.generateCertificate(KeyStoreTestUtil.java:86)
at 
org.apache.hadoop.security.ssl.KeyStoreTestUtil.setupSSLConfig(KeyStoreTestUtil.java:273)
at 
org.apache.hadoop.security.ssl.KeyStoreTestUtil.setupSSLConfig(KeyStoreTestUtil.java:228)
at 
org.apache.hadoop.hdfs.web.TestURLConnectionFactory.testSSLFactoryCleanup(TestURLConnectionFactory.java:83)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: java.lang.ClassNotFoundException: 
org.bouncycastle.x509.X509V1CertificateGenerator
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 29 more
{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/78/

[Dec 3, 2020 5:32:02 AM] (Wei-Chiu Chuang) HDFS-15705. Fix a typo in 
SecondaryNameNode.java. Contributed by Sixiang Ma.
[Dec 3, 2020 8:55:06 AM] (Adam Antal) YARN-9883. Reshape SchedulerHealth class. 
Contributed by D M Murali Krishna Reddy
[Dec 3, 2020 1:11:17 PM] (tmarq) HADOOP-17397: ABFS: SAS Test updates for 
version and permission update
[Dec 3, 2020 3:14:04 PM] (noreply) YARN-10511. Update 
yarn.nodemanager.env-whitelist value in docs (#2512)
[Dec 3, 2020 4:55:51 PM] (noreply) HADOOP-17392. Remote exception messages 
should not include the exception class (#2486). Contributed by Daryn Sharp and 
Ahmed Hussein
[Dec 3, 2020 8:05:20 PM] (Wei-Chiu Chuang) HADOOP-16881. KerberosAuthentication 
does not disconnect HttpURLConnection leading to CLOSE_WAIT cnxns. Contributed 
by Attila Magyar.
[Dec 3, 2020 11:14:40 PM] (noreply) HDFS-15706. HttpFS: Log more information on 
request failures. (#2515)
[Dec 4, 2020 12:33:30 AM] (noreply) HADOOP-17389. KMS should log full UGI 
principal. (#2476)
[Dec 4, 2020 1:20:09 AM] (Hui Fei) HDFS-15240. Erasure Coding: dirty buffer 
causes reconstruction block error. Contributed by HuangTao.


ERROR: File 'out/email-report.txt' does not exist

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-15709) Socket file descriptor leak in StripedBlockChecksumReconstructor

[Yushi Hayasaka (Jira)]

Yushi Hayasaka created HDFS-15709:
-

 Summary: Socket file descriptor leak in 
StripedBlockChecksumReconstructor
 Key: HDFS-15709
 URL: https://issues.apache.org/jira/browse/HDFS-15709
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: datanode, ec, erasure-coding
Reporter: Yushi Hayasaka


We found a socket file descriptor leak when we tried to get the checksum of EC 
file with reconstruction happened during the operation.
The cause of the leak seems that the StripedBlockChecksumReconstructor does not 
close StripedReader. Making the reader closed, the CLOSE_WAIT connections are 
gone.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org