[jira] [Created] (HDFS-17227) EC: Fix bug in choosing targets when racks is not enough.
Shuyan Zhang created HDFS-17227: --- Summary: EC: Fix bug in choosing targets when racks is not enough. Key: HDFS-17227 URL: https://issues.apache.org/jira/browse/HDFS-17227 Project: Hadoop HDFS Issue Type: Bug Reporter: Shuyan Zhang *Bug description* If, 1. There is a striped block blockinfo1, which has an excess replica on datanodeA. 2. blockinfo1 has an internal block that needs to be reconstruction. 3. The number of racks is less than the number of internal blocks of Blockinfo1. Then, NN may choose datanodeA to reconstruct the internal block, resulting in two internal blocks of blockinfo1 on datanodeA, causing confusion. *Root cause and solution* When we use `BlockPlacementPolicyRackFaultTolerant` for choosing targets and the racks is insufficient, `chooseEvenlyFromRemainingRacks` will be called. Currently, `chooseEvenlyFromRemainingRacks` calls `chooseOnce`, `chooseOnce` use `newExcludeNodes` as parameter instead of `excludedNodes`. When we choose targets for reconstructing internal blocks, 'newExcludeNodes' only includes those datanodes that contain live replicas, and does not include datanodes that have excess replicas. This may result in datanodes with excess replicas is chosen. I don't think we need to use 'newExcludeNodes', just pass `excludedNodes` to `chooseOnce`. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Fwd: [jira] [Created] (HADOOP-18933) upgrade netty to 4.1.100 due to CVE
Can anyone from yarn or hdfs review this before i just upvote it hoping all will be well. thanks -- Forwarded message - From: PJ Fanning (Jira) Date: Wed, 11 Oct 2023 at 20:27 Subject: [jira] [Created] (HADOOP-18933) upgrade netty to 4.1.100 due to CVE To: PJ Fanning created HADOOP-18933: --- Summary: upgrade netty to 4.1.100 due to CVE Key: HADOOP-18933 URL: https://issues.apache.org/jira/browse/HADOOP-18933 Project: Hadoop Common Issue Type: Improvement Reporter: PJ Fanning follow up to https://issues.apache.org/jira/browse/HADOOP-18783 https://netty.io/news/2023/10/10/4-1-100-Final.html -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [jira] [Created] (HADOOP-18933) upgrade netty to 4.1.100 due to CVE
POM change only & it has no test failing, so good from my side(HDFS) -Ayusg > On 16-Oct-2023, at 4:48 PM, Steve Loughran > wrote: > > Can anyone from yarn or hdfs review this before i just upvote it hoping all > will be well. thanks > > -- Forwarded message - > From: PJ Fanning (Jira) > Date: Wed, 11 Oct 2023 at 20:27 > Subject: [jira] [Created] (HADOOP-18933) upgrade netty to 4.1.100 due to CVE > To: > > > PJ Fanning created HADOOP-18933: > --- > > Summary: upgrade netty to 4.1.100 due to CVE > Key: HADOOP-18933 > URL: https://issues.apache.org/jira/browse/HADOOP-18933 > Project: Hadoop Common > Issue Type: Improvement >Reporter: PJ Fanning > > > follow up to https://issues.apache.org/jira/browse/HADOOP-18783 > > https://netty.io/news/2023/10/10/4-1-100-Final.html > > > > > > -- > This message was sent by Atlassian Jira > (v8.20.10#820010) > > - > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: common-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Join us at the Storage User Group Meetup!
Hi Please join us at the Storage Meetup at Cloudera's office next Wednesday. https://www.meetup.com/futureofdata-sanfrancisco/events/295917033/ We have HDFS developers from Uber join us to talk about optimizing HDFS for high density disks, and developers from Cloudera to talk about Apache Ozone and Apache Iceberg. I am being told this is an in-person event but it will be live streamed too. Please sign up to get more details about this event. Thanks, Wei-Chiu
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1381/ No changes -1 overall The following subsystems voted -1: blanks hadolint pathlen xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1381/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1381/artifact/out/results-compile-javac-root.txt [12K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1381/artifact/out/blanks-eol.txt [15M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1381/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1381/artifact/out/results-checkstyle-root.txt [13M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1381/artifact/out/results-hadolint.txt [20K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1381/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1381/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1381/artifact/out/results-shellcheck.txt [24K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1381/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1381/artifact/out/results-javadoc-javadoc-root.txt [244K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1182/ No changes ERROR: File 'out/email-report.txt' does not exist - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
