[jira] [Reopened] (HDFS-14158) Checkpointer ignores configured time period > 5 minutes
[
https://issues.apache.org/jira/browse/HDFS-14158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walter reopened HDFS-14158:
The explanation of [~ajisakaa] doesn't make sense to me. In my opinion, the
issue is already relevant for the functionality.
> Checkpointer ignores configured time period > 5 minutes
> ---
>
> Key: HDFS-14158
> URL: https://issues.apache.org/jira/browse/HDFS-14158
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.1
>Reporter: Timo Walter
>Priority: Minor
> Labels: checkpoint, hdfs, namenode
>
> The checkpointer always triggers a checkpoint every 5 minutes and ignores the
> flag "*dfs.namenode.checkpoint.period*", if its greater than 5 minutes.
> See the code below (in Checkpointer.java):
> {code:java}
> //Main work loop of the Checkpointer
> public void run() {
> // Check the size of the edit log once every 5 minutes.
> long periodMSec = 5 * 60; // 5 minutes
> if(checkpointConf.getPeriod() < periodMSec) {
> periodMSec = checkpointConf.getPeriod();
> }
> {code}
> If the configured period ("*dfs.namenode.checkpoint.period*") is lower than 5
> minutes, you choose use the configured one. But it always ignores it, if it's
> greater than 5 minutes.
>
> In my opinion, the if-expression should be:
> {code:java}
> if(checkpointConf.getPeriod() > periodMSec) {
> periodMSec = checkpointConf.getPeriod();
> }
> {code}
>
> Then "*dfs.namenode.checkpoint.period*" won't get ignored.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.0 - RC1
Thanks everyone for helping to vote this release! With 7 binding votes, 10 non-binding votes and no veto, this vote stands passed, I'm going to work on staging the release. Thanks, Sunil On Tue, Jan 15, 2019 at 9:59 PM Weiwei Yang wrote: > +1 (binding) > > - Setup a cluster, run teragen/terasort jobs > - Verified general readability of documentation (titles/navigations) > - Run some simple yarn commands: app/applicationattempt/container > - Checked restful APIs: RM cluster/metrics/scheduler/nodes, NM > node/apps/container > - Verified simple failover scenario > - Submitted distributed shell apps with affinity/anti-affinity constraints > - Configured conf based node attribute provider, alter attribute values > and verified the change > - Verified CLI add/list/remove node-attributes, submitted app with simple > node-attribute constraint > > -- > Weiwei >
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/ [Jan 15, 2019 3:59:59 AM] (aajisaka) YARN-9179. Fix NPE in AbstractYarnScheduler#updateNewContainerInfo. [Jan 15, 2019 1:15:18 PM] (stevel) HADOOP-16019. ZKDelegationTokenSecretManager won't log exception message [Jan 15, 2019 11:11:16 PM] (bharat) HDDS-978. Fix typo in doc : Client > S3 section. Contributed by Dinesh [Jan 16, 2019 1:28:08 AM] (aajisaka) HADOOP-15941. Addendum patch. Contributed by Takanobu Asanuma. -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes hadoop.hdfs.web.TestWebHdfsTimeouts cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/diff-compile-javac-root.txt [336K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/diff-checkstyle-root.txt [17M] hadolint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/diff-patch-hadolint.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/diff-patch-pylint.txt [60K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/whitespace-eol.txt [9.3M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/whitespace-tabs.txt [1.1M] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/branch-findbugs-hadoop-hdds_client.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/branch-findbugs-hadoop-hdds_framework.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/branch-findbugs-hadoop-hdds_tools.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/branch-findbugs-hadoop-ozone_client.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/branch-findbugs-hadoop-ozone_common.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/branch-findbugs-hadoop-ozone_objectstore-service.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/branch-findbugs-hadoop-ozone_ozonefs.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/branch-findbugs-hadoop-ozone_s3gateway.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/branch-findbugs-hadoop-ozone_tools.txt [8.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/diff-javadoc-javadoc-root.txt [752K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [328K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [84K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1018/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt [84K] https://builds.apache
Re: [VOTE] Release Apache Hadoop 3.2.0 - RC1
+1 (binding) Apologies for late vote! - Built from source for hbase profile=2.0 and deployed RM HA cluster along with ATSv2.0 enabled. - Ran sample jobs and verified all ATSv2.0 data are publishing. - Verified UI2 for ATSv2.0 data! -Rohith Sharma K S On Wed, 16 Jan 2019 at 07:00, Sunil G wrote: > Thanks everyone for helping to vote this release! > > With 7 binding votes, 10 non-binding votes and no veto, this vote stands > passed, > I'm going to work on staging the release. > > Thanks, > Sunil > > On Tue, Jan 15, 2019 at 9:59 PM Weiwei Yang wrote: > > > +1 (binding) > > > > - Setup a cluster, run teragen/terasort jobs > > - Verified general readability of documentation (titles/navigations) > > - Run some simple yarn commands: app/applicationattempt/container > > - Checked restful APIs: RM cluster/metrics/scheduler/nodes, NM > > node/apps/container > > - Verified simple failover scenario > > - Submitted distributed shell apps with affinity/anti-affinity > constraints > > - Configured conf based node attribute provider, alter attribute values > > and verified the change > > - Verified CLI add/list/remove node-attributes, submitted app with simple > > node-attribute constraint > > > > -- > > Weiwei > > >
Re: [VOTE] Release Apache Hadoop 3.2.0 - RC1
+1 (binding) Apologies for one more late vote! - Built from source and also installed the release tar. - Verified all Node Labels and node . - Ran sample jobs and verified UI Thanks and Regards, + Naga On Wed, Jan 16, 2019 at 10:34 PM Rohith Sharma K S < rohithsharm...@apache.org> wrote: > +1 (binding) Apologies for late vote! > > - Built from source for hbase profile=2.0 and deployed RM HA cluster along > with ATSv2.0 enabled. > - Ran sample jobs and verified all ATSv2.0 data are publishing. > - Verified UI2 for ATSv2.0 data! > > -Rohith Sharma K S > > On Wed, 16 Jan 2019 at 07:00, Sunil G wrote: > > > Thanks everyone for helping to vote this release! > > > > With 7 binding votes, 10 non-binding votes and no veto, this vote stands > > passed, > > I'm going to work on staging the release. > > > > Thanks, > > Sunil > > > > On Tue, Jan 15, 2019 at 9:59 PM Weiwei Yang wrote: > > > > > +1 (binding) > > > > > > - Setup a cluster, run teragen/terasort jobs > > > - Verified general readability of documentation (titles/navigations) > > > - Run some simple yarn commands: app/applicationattempt/container > > > - Checked restful APIs: RM cluster/metrics/scheduler/nodes, NM > > > node/apps/container > > > - Verified simple failover scenario > > > - Submitted distributed shell apps with affinity/anti-affinity > > constraints > > > - Configured conf based node attribute provider, alter attribute values > > > and verified the change > > > - Verified CLI add/list/remove node-attributes, submitted app with > simple > > > node-attribute constraint > > > > > > -- > > > Weiwei > > > > > >
[jira] [Created] (HDFS-14211) [Consistent Observer Reads] Allow for configurable "always msync" mode
Erik Krogen created HDFS-14211:
--
Summary: [Consistent Observer Reads] Allow for configurable
"always msync" mode
Key: HDFS-14211
URL: https://issues.apache.org/jira/browse/HDFS-14211
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Reporter: Erik Krogen
To allow for reads to be serviced from an ObserverNode (see HDFS-12943) in a
consistent way, an {{msync}} API was introduced (HDFS-13688) to allow for a
client to fetch the latest transaction ID from the Active NN, thereby ensuring
that subsequent reads from the ObserverNode will be up-to-date with the current
state of the Active.
Using this properly, however, requires application-side changes: for examples,
a NodeManager should call {{msync}} before localizing the resources for a
client, since it received notification of the existence of those resources via
communicate which is out-of-band to HDFS and thus could potentially attempt to
localize them prior to the availability of those resources on the ObserverNode.
Until such application-side changes can be made, which will be a longer-term
effort, we need to provide a mechanism for unchanged clients to utilize the
ObserverNode without exposing such a client to inconsistencies. This is
essentially phase 3 of the roadmap outlined in the [design
document|https://issues.apache.org/jira/secure/attachment/12915990/ConsistentReadsFromStandbyNode.pdf]
for HDFS-12943.
The design document proposes some heuristics based on understanding of how
common applications (e.g. MR) use HDFS for resources. As an initial pass, we
can simply have a flag which tells a client to call {{msync}} before _every
single_ read operation. This may seem counterintuitive, as it turns every read
operation into two RPCs: {{msync}} to the Active following by an actual read
operation to the Observer. However, the {{msync}} operation is extremely
lightweight, as it does not acquire the {{FSNamesystemLock}}, and in
experiments we have found that this approach can easily scale to well over
100,000 {{msync}} operations per second on the Active (while still servicing
approx. 10,000 write op/s). Combined with the fast-path edit log tailing for
standby/observer nodes (HDFS-13150), this "always msync" approach should
introduce only a few ms of extra latency to each read call.
Below are some experimental results collected from experiments which convert a
normal RPC workload into one in which all read operations are turned into an
{{msync}}. The baseline is a workload of 1.5k write op/s and 25k read op/s.
||Rate Multiplier|2|4|6|8||
||RPC Queue Avg Time (ms)|14.2|53.2|110.4|125.3||
||RPC Queue NumOps Avg (k)|51.4|102.3|147.8|177.9||
||RPC Queue NumOps Max (k)|148.8|269.5|306.3|312.4||
Results are promising up to between 4x and 6x of the baseline workload, which
is approx. 100-150k read op/s.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14212) `hdfs crypto -listZones` should list encrypted zones from all namespaces
venkata naidu udamala created HDFS-14212: Summary: `hdfs crypto -listZones` should list encrypted zones from all namespaces Key: HDFS-14212 URL: https://issues.apache.org/jira/browse/HDFS-14212 Project: Hadoop HDFS Issue Type: Improvement Reporter: venkata naidu udamala `hdfs crypto -listZones` is listing the encryption zones only from the default namespace. there is no way to list the encryption zones from other namespaces. It would be helpful if there is a way to list encryption zones from all namespaces or take any option to specify the namespace in the command -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-14212) `hdfs crypto -listZones` should list encrypted zones from all namespaces
[ https://issues.apache.org/jira/browse/HDFS-14212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] venkata naidu udamala resolved HDFS-14212. -- Resolution: Fixed Release Note: This can be done using "fs" option `hdfs crypto -fs hdfs:namespace1 -listZones` > `hdfs crypto -listZones` should list encrypted zones from all namespaces > > > Key: HDFS-14212 > URL: https://issues.apache.org/jira/browse/HDFS-14212 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: venkata naidu udamala >Priority: Major > > `hdfs crypto -listZones` is listing the encryption zones only from the > default namespace. there is no way to list the encryption zones from other > namespaces. It would be helpful if there is a way to list encryption zones > from all namespaces or take any option to specify the namespace in the command -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-981) Block allocation should involve pipeline selection and then container selection
Lokesh Jain created HDDS-981: Summary: Block allocation should involve pipeline selection and then container selection Key: HDDS-981 URL: https://issues.apache.org/jira/browse/HDDS-981 Project: Hadoop Distributed Data Store Issue Type: Bug Components: SCM Reporter: Lokesh Jain Assignee: Lokesh Jain Currently SCM maintains a list of preallocated containers and allocates blocks from these containers. This approach does not work well with dynamics of the cluster where new nodes are being added and pipelines are destroyed. New containers are not created until all the preallocated containers are exhausted. The Jira aims to establish a criteria in block allocation where first a pipeline is selected amongst the available pipelines and then a container is selected in that pipeline. In order to handle the dynamics of the cluster a fixed interval pipeline creator job can be launched which creates pipelines in the system. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
