[jira] [Resolved] (HADOOP-12387) Improve S3AFastOutputStream memory management
[ https://issues.apache.org/jira/browse/HADOOP-12387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-12387. - Resolution: Duplicate Assignee: Steve Loughran (was: Thomas Demoor) > Improve S3AFastOutputStream memory management > - > > Key: HADOOP-12387 > URL: https://issues.apache.org/jira/browse/HADOOP-12387 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Thomas Demoor >Assignee: Steve Loughran > > The use of ByteArrayOutputStream causes unexpected memory copies that causes > unintuitive memory usage for certain combinations of the configuration > parameters. Replacing it by more fine grained memory buffer management would > fix this, reduce memory usage and eliminate a memory copy. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13703) S3ABlockOutputStream to pass Yetus & Jenkins
Steve Loughran created HADOOP-13703: --- Summary: S3ABlockOutputStream to pass Yetus & Jenkins Key: HADOOP-13703 URL: https://issues.apache.org/jira/browse/HADOOP-13703 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 2.7.3 Reporter: Steve Loughran The HADOOP-13560 patches and PR has got yetus confused. This patch is purely to do test runs. h1. All discourse must continue to take place in HADOOP-13560 and/or the Pull Request. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.6.5 (RC1)
+1 (binding) - Verified signatures and digests- Built native from source- Deployed to a single-node cluster and ran some sample jobs Jason On Sunday, October 2, 2016 7:13 PM, Sangjin Lee wrote: Hi folks, I have pushed a new release candidate (R1) for the Apache Hadoop 2.6.5 release (the next maintenance release in the 2.6.x release line). RC1 contains fixes to CHANGES.txt, and is otherwise identical to RC0. Below are the details of this release candidate: The RC is available for validation at: http://home.apache.org/~sjlee/hadoop-2.6.5-RC1/. The RC tag in git is release-2.6.5-RC1 and its git commit is e8c9fe0b4c252caf2ebf1464220599650f119997. The maven artifacts are staged via repository.apache.org at: https://repository.apache.org/content/repositories/orgapachehadoop-1050/. You can find my public key at http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS. Please try the release and vote. The vote will run for the usual 5 days. I would greatly appreciate your timely vote. Thanks! Regards, Sangjin
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/190/ [Oct 9, 2016 9:07:03 AM] (kai.zheng) HADOOP-12579. Deprecate WriteableRPCEngine. Contributed by Wei Zhou [Oct 9, 2016 9:33:26 AM] (kai.zheng) MAPREDUCE-6780. Add support for HDFS directory with erasure code policy [Oct 9, 2016 9:53:36 AM] (kai.zheng) HADOOP-13641. Update UGI#spawnAutoRenewalThreadForUserCreds to reduce [Oct 10, 2016 5:55:49 AM] (kai.zheng) HDFS-10895. Update HDFS Erasure Coding doc to add how to use ISA-L based -1 overall The following subsystems voted -1: asflicense unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.ha.TestZKFailoverController hadoop.hdfs.server.namenode.TestAddBlock hadoop.hdfs.server.datanode.TestDataNodeLifeline hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/190/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/190/artifact/out/diff-compile-javac-root.txt [168K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/190/artifact/out/diff-checkstyle-root.txt [16M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/190/artifact/out/diff-patch-pylint.txt [16K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/190/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/190/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/190/artifact/out/whitespace-eol.txt [11M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/190/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/190/artifact/out/diff-javadoc-javadoc-root.txt [2.2M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/190/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [120K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/190/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [148K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/190/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/190/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [268K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/190/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt [124K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/190/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13704) S3A getContentSummary() to move to listFiles(recursive) to count children; instrument use
Steve Loughran created HADOOP-13704:
---
Summary: S3A getContentSummary() to move to listFiles(recursive)
to count children; instrument use
Key: HADOOP-13704
URL: https://issues.apache.org/jira/browse/HADOOP-13704
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 2.8.0
Reporter: Steve Loughran
Hive and a bit of Spark use {{getContentSummary()} to get some summary stats of
a filesystem. This is very expensive on S3A (and any other object store),
especially as the base implementation does the recursive tree walk.
Because of HADOOP-13208, we have a full enumeration of files under a path
without directory costs...S3A can/should switch to this to speed up those
places where the operation is called.
Also
* API call needs FS spec and contract tests
* S3A could instrument invocation, so as to enable real-world popularity to be
measured
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/120/ [Oct 9, 2016 9:53:36 AM] (kai.zheng) HADOOP-13641. Update UGI#spawnAutoRenewalThreadForUserCreds to reduce [Oct 10, 2016 5:55:49 AM] (kai.zheng) HDFS-10895. Update HDFS Erasure Coding doc to add how to use ISA-L based [Oct 10, 2016 11:32:39 AM] (stevel) HADOOP-13696. change hadoop-common dependency scope of jsch to provided. -1 overall The following subsystems voted -1: compile unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.TestWriteReadStripedFile hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA hadoop.hdfs.server.datanode.TestDataNodeLifeline hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness hadoop.hdfs.web.TestWebHDFS hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService hadoop.yarn.server.nodemanager.TestNodeManagerShutdown hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.timelineservice.storage.common.TestRowKeys hadoop.yarn.server.timelineservice.storage.common.TestKeyConverters hadoop.yarn.server.timelineservice.storage.common.TestSeparator hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing hadoop.yarn.server.resourcemanager.TestResourceTrackerService hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorage hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun hadoop.yarn.server.timelineservice.storage.TestPhoenixOfflineAggregationWriterImpl hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService hadoop.mapreduce.TestMRJobClient Timed out junit tests : org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache org.apache.hadoop.mapred.TestMRIntermediateDataEncryption compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/120/artifact/out/patch-compile-root.txt [312K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/120/artifact/out/patch-compile-root.txt [312K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/120/artifact/out/patch-compile-root.txt [312K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/120/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [260K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/120/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [52K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/120/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [52K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/120/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt [20K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/120/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [76K] https://bui
[jira] [Resolved] (HADOOP-13699) Configuration does not substitute multiple references to the same var
[
https://issues.apache.org/jira/browse/HADOOP-13699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang resolved HADOOP-13699.
--
Resolution: Fixed
Fix Version/s: 3.0.0-alpha2
Committed to trunk, thanks for reviewing Xiao!
> Configuration does not substitute multiple references to the same var
> -
>
> Key: HADOOP-13699
> URL: https://issues.apache.org/jira/browse/HADOOP-13699
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Andrew Wang
>Priority: Critical
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13699.001.patch
>
>
> Config var loop detection was originally introduced by HADOOP-6871. Due to
> cycle detection changes in the trunk patch for HADOOP-11506, resolution for
> multiple references to the same variable no longer resolved, e.g.
> {noformat}
> somekey = "${otherkey} ${otherkey}"
> {noformat}
> This loop detection business is fragile, expensive, and not in branch-2, so
> let's reduce it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13584) hadoop-aliyun: merge HADOOP-12756 branch back
[
https://issues.apache.org/jira/browse/HADOOP-13584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng resolved HADOOP-13584.
Resolution: Fixed
I have merged the branch to trunk via {{git merge --no-ff}} according to the
vote result and the discussions in HADOOP-12756
> hadoop-aliyun: merge HADOOP-12756 branch back
> -
>
> Key: HADOOP-13584
> URL: https://issues.apache.org/jira/browse/HADOOP-13584
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Reporter: shimingfei
>Assignee: Genmao Yu
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13584.001.patch, HADOOP-13584.002.patch,
> HADOOP-13584.003.patch, HADOOP-13584.004.patch, HADOOP-13584.005.patch
>
>
> We have finished a round of improvement over Hadoop-12756 branch, which
> intends to incorporate Aliyun OSS support in Hadoop. This feature provides
> basic support for data access to Aliyun OSS from Hadoop applications.
> In the implementation, we follow the style of S3 support in Hadooop. Besides
> we also provide FileSystem contract test over real Aliyun OSS environment. By
> simple configuration, it can be enabled/disabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13705) Revert HADOOP-13534 Remove unused TrashPolicy#getInstance and initialize code
Andrew Wang created HADOOP-13705: Summary: Revert HADOOP-13534 Remove unused TrashPolicy#getInstance and initialize code Key: HADOOP-13705 URL: https://issues.apache.org/jira/browse/HADOOP-13705 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.0.0-alpha1 Reporter: Andrew Wang Assignee: Andrew Wang Per discussion on HADOOP-13700, I'd like to revert HADOOP-13534. It removes a deprecated API, but the 2.x line does not have a release with the new replacement API. This places a burden on downstream applications. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.6.5 (RC1)
Thanks everyone for checking out this release! The vote passes with 15 +1's, 7 of which are binding, and no -1's. I'll go ahead and wrap up the release, and will send an announcement in the next day or two. Regards, Sangjin On Mon, Oct 10, 2016 at 6:38 AM, Jason Lowe wrote: > +1 (binding) > > - Verified signatures and digests > - Built native from source > - Deployed to a single-node cluster and ran some sample jobs > > Jason > > > On Sunday, October 2, 2016 7:13 PM, Sangjin Lee wrote: > > > Hi folks, > > I have pushed a new release candidate (R1) for the Apache Hadoop 2.6.5 > release (the next maintenance release in the 2.6.x release line). RC1 > contains fixes to CHANGES.txt, and is otherwise identical to RC0. > > Below are the details of this release candidate: > > The RC is available for validation at: > http://home.apache.org/~sjlee/hadoop-2.6.5-RC1/. > > The RC tag in git is release-2.6.5-RC1 and its git commit is > e8c9fe0b4c252caf2ebf1464220599650f119997. > > The maven artifacts are staged via repository.apache.org at: > https://repository.apache.org/content/repositories/orgapachehadoop-1050/. > > You can find my public key at > http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS. > > Please try the release and vote. The vote will run for the usual 5 days. I > would greatly appreciate your timely vote. Thanks! > > Regards, > Sangjin > > >
