Hi,
I tried to summarize all of the information from different mail threads
about the upcomming releases:
https://cwiki.apache.org/confluence/display/HADOOP/Roadmap
Please fix it / let me know if you see any invalid data. I will try to
follow the conversations and update accordingly.
Two a
TL;DR: I propose to create official hadoop images and upload them to the
dockerhub.
GOAL/SCOPE: I would like improve the existing documentation with
easy-to-use docker based recipes to start hadoop clusters with various
configuration.
The images also could be used to test experimental feat
Thanks all the feedbacks.
I created an issue:
https://issues.apache.org/jira/browse/HADOOP-14898
Let's continue the discussion there.
Thanks,
Marton
On 09/08/2017 02:45 PM, Marton, Elek wrote:
TL;DR: I propose to create official hadoop images and upload them to the
dockerhub.
GOAL/
+1 (non-binding)
* Built from the source
* Installed dockerized YARN/HADOOP cluster to a 20 node cluster
(scheduled with nomad, configured from consul, docker host networking)
* Started example yarn jobs (teragen/terasort) and hdfs dfs commands +
checking UIs
I noticed only two very minor iss
hadoop-3.0.0-alpha2.tar.gz is much more smaller than
hadoop-3.0.0-alpha1.tar.gz. (246M vs 316M)
The big difference is the generated source documentation:
find -name src-html
./hadoop-2.7.3/share/doc/hadoop/api/src-html
./hadoop-2.7.3/share/doc/hadoop/hadoop-hdfs-httpfs/apidocs/src-html
./hadoop-
]>
> minicluster is kind of weird on filesystems that don't support mixed case,
> like OS X's default HFS+.
>
> $ jar tf hadoop-client-minicluster-3.0.0-alpha3-SNAPSHOT.jar | grep -i
> license
> LICENSE.txt
> license/
> license/LICENSE
> license/LICENSE.dom-documentation.txt
> license/LICENSE.
e binary and verified the md5.
2. Deployed it on 3 node cluster with 1 ResourceManager and 2 NodeManager.
3. Set YARN to use Fair Scheduler.
4. Ran MapReduce jobs Pi
5. Verified Hadoop version command output is correct.
Best,
Yufei
On Tue, Jan 24, 2017 at 3:02 AM, Marton Elek
mailto:me...@hortonwo
Is there any reason to wait for 2.8 with 2.7.4?
Unfortunately the previous thread about release cadence has been ended without
final decision. But if I understood well, there was more or less an agreement
about that it would be great to achieve more frequent releases, if possible
(with or with
PM
To: Sangjin Lee
Cc: Marton Elek; common-...@hadoop.apache.org; yarn-...@hadoop.apache.org;
Hdfs-dev; mapreduce-...@hadoop.apache.org
Subject: Re: About 2.7.4 Release
I was planning to take this up, celebrating my return from my paternity leave
of absence for quite a while.
Marton, let me know
I think the main point here is the testing of the release script, not the
creation of the official release.
I think there should be an option to configure the release tool to use a forked
github repo and/or a private playground nexus instead of official apache repos.
In this case it would be ea
+1 (non-binding)
Tested from the released binary package
* 5 node cluster running from dockerized containers (every
namenode/datanode/nodemanager, etc. are running in separated containers)
* Bitcoin bockchain data (~100Gb) parsed and imported to HBase (1.2.4)
* Spark (2.1.0 with included hadoop)
Yeah, I suffer from the same problem as well. I will be very happy if
it's fixed.
Usually I run all of the components from my IDE (namenode, datanode,
ozone components etc.). I have two problems:
1. the provided scopes which was mentioned.
2. The components couldn't be started because the ht
[
https://issues.apache.org/jira/browse/HDDS-2287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Elek resolved HDDS-2287.
---
Resolution: Fixed
> Move ozone source code to apache/hadoop-ozone from apache/had
Marton Elek created HDDS-2371:
-
Summary: Print out the ozone version during the startup instead of
hadoop version
Key: HDDS-2371
URL: https://issues.apache.org/jira/browse/HDDS-2371
Project: Hadoop
Marton Elek created HDDS-2372:
-
Summary: Datanode pipeline is failing with NoSuchFileException
Key: HDDS-2372
URL: https://issues.apache.org/jira/browse/HDDS-2372
Project: Hadoop Distributed Data Store
[
https://issues.apache.org/jira/browse/HDDS-2348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Elek resolved HDDS-2348.
---
Fix Version/s: 0.5.0
Resolution: Fixed
> remove log4j properties for org.apache.hadoop.oz
[
https://issues.apache.org/jira/browse/HDDS-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Elek resolved HDDS-2219.
---
Fix Version/s: 0.5.0
Resolution: Fixed
> Move all the ozone dist scripts/configs to
Marton Elek created HDDS-2399:
-
Summary: Update mailing list information in CONTRIBUTION and
README files
Key: HDDS-2399
URL: https://issues.apache.org/jira/browse/HDDS-2399
Project: Hadoop Distributed
Marton Elek created HDDS-2400:
-
Summary: Enable github actions based builds for oyone
Key: HDDS-2400
URL: https://issues.apache.org/jira/browse/HDDS-2400
Project: Hadoop Distributed Data Store
[
https://issues.apache.org/jira/browse/HDDS-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Elek resolved HDDS-1515.
---
Fix Version/s: 0.5.0
Resolution: Fixed
Merged to the master. Thanks [~akkidx] the contribution
Marton Elek created HDDS-2410:
-
Summary: ozoneperf docker-compose should use privileged containers
Key: HDDS-2410
URL: https://issues.apache.org/jira/browse/HDDS-2410
Project: Hadoop Distributed Data
[
https://issues.apache.org/jira/browse/HDDS-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Elek resolved HDDS-2369.
---
Resolution: Fixed
> Fix typo in param descript
Marton Elek created HDDS-2411:
-
Summary: Create DataChunkValidator Freon test
Key: HDDS-2411
URL: https://issues.apache.org/jira/browse/HDDS-2411
Project: Hadoop Distributed Data Store
Issue
[
https://issues.apache.org/jira/browse/HDDS-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Elek resolved HDDS-2327.
---
Resolution: Fixed
> Provide new Freon test to test Ratis pipeline with pure XceiverClientRa
Marton Elek created HDDS-2412:
-
Summary: Define description/topics/merge strategy for the github
repository with .asf.yaml
Key: HDDS-2412
URL: https://issues.apache.org/jira/browse/HDDS-2412
Project
Marton Elek created HDDS-2413:
-
Summary: Set configuration variables from annotated java objects
Key: HDDS-2413
URL: https://issues.apache.org/jira/browse/HDDS-2413
Project: Hadoop Distributed Data Store
Marton Elek created HDDS-2414:
-
Summary: Simplify robot tests with removing output greps
Key: HDDS-2414
URL: https://issues.apache.org/jira/browse/HDDS-2414
Project: Hadoop Distributed Data Store
Marton Elek created HDDS-2444:
-
Summary: Remove server side dependencies from ozonefs jar files
Key: HDDS-2444
URL: https://issues.apache.org/jira/browse/HDDS-2444
Project: Hadoop Distributed Data Store
[
https://issues.apache.org/jira/browse/HDDS-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Elek resolved HDDS-1701.
---
Fix Version/s: 0.5.0
Resolution: Fixed
> Move dockerbin script to libe
[
https://issues.apache.org/jira/browse/HDDS-2325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Elek resolved HDDS-2325.
---
Fix Version/s: 0.5.0
Resolution: Fixed
> BenchMarkDatanodeDispatcher genesis test is fail
Marton Elek created HDDS-2461:
-
Summary: Logging by ChunkUtils is misleading
Key: HDDS-2461
URL: https://issues.apache.org/jira/browse/HDDS-2461
Project: Hadoop Distributed Data Store
Issue Type
Marton Elek created HDDS-2482:
-
Summary: Enable github actions for pull requests
Key: HDDS-2482
URL: https://issues.apache.org/jira/browse/HDDS-2482
Project: Hadoop Distributed Data Store
Issue
[
https://issues.apache.org/jira/browse/HDDS-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Elek resolved HDDS-2482.
---
Fix Version/s: 0.5.0
Resolution: Fixed
> Enable github actions for pull reque
Marton Elek created HDDS-2506:
-
Summary: Remove keyAllocationInfo and replication info from the
auditLog
Key: HDDS-2506
URL: https://issues.apache.org/jira/browse/HDDS-2506
Project: Hadoop Distributed
Marton Elek created HDDS-2507:
-
Summary: Remove the hard-coded exclusion of
TestMiniChaosOzoneCluster
Key: HDDS-2507
URL: https://issues.apache.org/jira/browse/HDDS-2507
Project: Hadoop Distributed Data
[
https://issues.apache.org/jira/browse/HDDS-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Elek resolved HDDS-2495.
---
Fix Version/s: 0.5.0
Resolution: Fixed
> Sonar - "notify" may not wake up th
Marton Elek created HDDS-2533:
-
Summary: Disable failing acceptance and unit tests
Key: HDDS-2533
URL: https://issues.apache.org/jira/browse/HDDS-2533
Project: Hadoop Distributed Data Store
Marton Elek created HDDS-2535:
-
Summary: TestOzoneManagerDoubleBufferWithOMResponse is flaky
Key: HDDS-2535
URL: https://issues.apache.org/jira/browse/HDDS-2535
Project: Hadoop Distributed Data Store
Marton Elek created HDDS-2541:
-
Summary: CI builds should use merged code state instead of the
forked branch
Key: HDDS-2541
URL: https://issues.apache.org/jira/browse/HDDS-2541
Project: Hadoop
Marton Elek created HDDS-2542:
-
Summary: Race condition between read and write stateMachineData
Key: HDDS-2542
URL: https://issues.apache.org/jira/browse/HDDS-2542
Project: Hadoop Distributed Data Store
[
https://issues.apache.org/jira/browse/HDDS-2533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Elek resolved HDDS-2533.
---
Fix Version/s: 0.5.0
Assignee: Marton Elek
Resolution: Fixed
> Disable fail
Marton Elek created HDDS-2587:
-
Summary: Enable sonarcloud measurement as part of CI builds
Key: HDDS-2587
URL: https://issues.apache.org/jira/browse/HDDS-2587
Project: Hadoop Distributed Data Store
Marton Elek created HDDS-2606:
-
Summary: Acceptance tests are flaky due to async pipeline creation
Key: HDDS-2606
URL: https://issues.apache.org/jira/browse/HDDS-2606
Project: Hadoop Distributed Data
[
https://issues.apache.org/jira/browse/HDFS-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Elek resolved HDFS-13510.
Resolution: Won't Fix
> Ozone: Fix precommit hook for Ozone/Hdds
Marton Elek created HDDS-2183:
-
Summary: Container and pipline subcommands of scmcli should be
grouped
Key: HDDS-2183
URL: https://issues.apache.org/jira/browse/HDDS-2183
Project: Hadoop Distributed Data
Marton Elek created HDDS-2184:
-
Summary: Rename ozone scmcli to ozone admin
Key: HDDS-2184
URL: https://issues.apache.org/jira/browse/HDDS-2184
Project: Hadoop Distributed Data Store
Issue Type
Marton Elek created HDDS-2209:
-
Summary: Checkstyle issue in OmUtils on trunk
Key: HDDS-2209
URL: https://issues.apache.org/jira/browse/HDDS-2209
Project: Hadoop Distributed Data Store
Issue
Marton Elek created HDDS-2214:
-
Summary: TestSCMContainerPlacementRackAware has an intermittent
failure
Key: HDDS-2214
URL: https://issues.apache.org/jira/browse/HDDS-2214
Project: Hadoop Distributed
Marton Elek created HDDS-2216:
-
Summary: Rename HADOOP_RUNNER_VERSION to OZONE_RUNNER_VERSION in
compose .env files
Key: HDDS-2216
URL: https://issues.apache.org/jira/browse/HDDS-2216
Project: Hadoop
Marton Elek created HDDS-2217:
-
Summary: Remove log4j and audit configuration from the
docker-config files
Key: HDDS-2217
URL: https://issues.apache.org/jira/browse/HDDS-2217
Project: Hadoop Distributed
Marton Elek created HDDS-2218:
-
Summary: Use OZONE_CLASSPATH instead of HADOOP_CLASSPATH
Key: HDDS-2218
URL: https://issues.apache.org/jira/browse/HDDS-2218
Project: Hadoop Distributed Data Store
Marton Elek created HDDS-2219:
-
Summary: Move all the ozone dist scripts/configs to one location
Key: HDDS-2219
URL: https://issues.apache.org/jira/browse/HDDS-2219
Project: Hadoop Distributed Data Store
Marton Elek created HDDS-2220:
-
Summary: HddsVolume needs a toString method
Key: HDDS-2220
URL: https://issues.apache.org/jira/browse/HDDS-2220
Project: Hadoop Distributed Data Store
Issue Type
Marton Elek created HDDS-2221:
-
Summary: Monitor datanodes in ozoneperf compose cluster
Key: HDDS-2221
URL: https://issues.apache.org/jira/browse/HDDS-2221
Project: Hadoop Distributed Data Store
Marton Elek created HDDS-2232:
-
Summary: ozone-build docker image is failing due to a missing
entrypoint script
Key: HDDS-2232
URL: https://issues.apache.org/jira/browse/HDDS-2232
Project: Hadoop
Marton Elek created HDDS-2237:
-
Summary: KeyDeletingService throws NPE if it's started too early
Key: HDDS-2237
URL: https://issues.apache.org/jira/browse/HDDS-2237
Project: Hadoop Distributed Data
Marton Elek created HDDS-2251:
-
Summary: Add an option to customize unit.sh and integration.sh
parameters
Key: HDDS-2251
URL: https://issues.apache.org/jira/browse/HDDS-2251
Project: Hadoop Distributed
[
https://issues.apache.org/jira/browse/HDDS-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Elek resolved HDDS-2251.
---
Resolution: Fixed
> Add an option to customize unit.sh and integration.sh paramet
Marton Elek created HDDS-2287:
-
Summary: Move ozone source code to apache/hadoop-ozone from
apache/hadoop
Key: HDDS-2287
URL: https://issues.apache.org/jira/browse/HDDS-2287
Project: Hadoop Distributed
Marton Elek created HDDS-2288:
-
Summary: Delete hadoop-ozone and hadoop-hdds subprojects from
apache trunk
Key: HDDS-2288
URL: https://issues.apache.org/jira/browse/HDDS-2288
Project: Hadoop Distributed
Marton Elek created HDDS-2289:
-
Summary: Put testing information and a problem description to the
github PR template
Key: HDDS-2289
URL: https://issues.apache.org/jira/browse/HDDS-2289
Project: Hadoop
Marton Elek created HDDS-2292:
-
Summary: Create Ozone specific README.md to the new hadoop-ozone
repository
Key: HDDS-2292
URL: https://issues.apache.org/jira/browse/HDDS-2292
Project: Hadoop Distributed
Marton Elek created HDDS-2293:
-
Summary: Create a new CONTRIBUTION.md for the new repository
Key: HDDS-2293
URL: https://issues.apache.org/jira/browse/HDDS-2293
Project: Hadoop Distributed Data Store
Marton Elek created HDDS-2294:
-
Summary: Create a new HISTORY.md in the new repository
Key: HDDS-2294
URL: https://issues.apache.org/jira/browse/HDDS-2294
Project: Hadoop Distributed Data Store
Marton Elek created HDDS-2295:
-
Summary: Display log of freon on the standard output
Key: HDDS-2295
URL: https://issues.apache.org/jira/browse/HDDS-2295
Project: Hadoop Distributed Data Store
Marton Elek created HDDS-2296:
-
Summary: ozoneperf compose cluster shouln't start freon by default
Key: HDDS-2296
URL: https://issues.apache.org/jira/browse/HDDS-2296
Project: Hadoop Distributed
Marton Elek created HDDS-2297:
-
Summary: Enable Opentracing for new Freon tests
Key: HDDS-2297
URL: https://issues.apache.org/jira/browse/HDDS-2297
Project: Hadoop Distributed Data Store
Issue
Marton Elek created HDDS-2298:
-
Summary: Fix maven warning about duplicated metrics-core jar
Key: HDDS-2298
URL: https://issues.apache.org/jira/browse/HDDS-2298
Project: Hadoop Distributed Data Store
Marton Elek created HDDS-2300:
-
Summary: Publish normalized Ratis metrics via the prometheus
endpoint
Key: HDDS-2300
URL: https://issues.apache.org/jira/browse/HDDS-2300
Project: Hadoop Distributed Data
Marton Elek created HDDS-2302:
-
Summary: Manage common pom versions in one common place
Key: HDDS-2302
URL: https://issues.apache.org/jira/browse/HDDS-2302
Project: Hadoop Distributed Data Store
Marton Elek created HDDS-2308:
-
Summary: Switch to centos with the apache/ozone-build docker image
Key: HDDS-2308
URL: https://issues.apache.org/jira/browse/HDDS-2308
Project: Hadoop Distributed Data
[
https://issues.apache.org/jira/browse/HDDS-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Elek resolved HDDS-2312.
---
Fix Version/s: 0.5.0
Resolution: Fixed
> Fix typo in ozone comm
Marton Elek created HDDS-2316:
-
Summary: Support to skip recon and/or ozonefs during the build
Key: HDDS-2316
URL: https://issues.apache.org/jira/browse/HDDS-2316
Project: Hadoop Distributed Data Store
[
https://issues.apache.org/jira/browse/HDDS-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Elek resolved HDDS-2221.
---
Fix Version/s: 0.5.0
Resolution: Fixed
> Monitor datanodes in ozoneperf compose clus
Marton Elek created HDDS-2325:
-
Summary: BenchMarkDatanodeDispatcher genesis test is failing with
NPE
Key: HDDS-2325
URL: https://issues.apache.org/jira/browse/HDDS-2325
Project: Hadoop Distributed Data
Marton Elek created HDDS-2326:
-
Summary: Http server of Freon is not started for new Freon tests
Key: HDDS-2326
URL: https://issues.apache.org/jira/browse/HDDS-2326
Project: Hadoop Distributed Data Store
Marton Elek created HDDS-2327:
-
Summary: Provide new Freon test to test Ratis pipeline with pure
XceiverClientRatis
Key: HDDS-2327
URL: https://issues.apache.org/jira/browse/HDDS-2327
Project: Hadoop
77 matches
Mail list logo
