If I recall this problem correctly, the root cause is the default zstd
compression block size is 256kb, and Hadoop Zstd compression will attempt
to use the OS platform default compression size, if it is available. The
recommended output size is slightly bigger than input size to account for
header
ProtobufHelper should not be a public API. Hadoop uses protobuf
serialization to expertise RPC performance with many drawbacks. The
generalized object usually require another indirection to map to usable
Java object, this is making Hadoop code messy, and that is topic for
another day. The main c
Hi Allen,
What if the commands are:
yarn application -deploy –f spec.json
yarn application -stop
yarn application -restart
yarn application -remove
and
yarn application –list will display both application list from RM as well as
docker services?
I think the development team was concerned t
The check list looks good. Some more items to add:
Kerberos
TGT renewal
SPNEGO support
Delegation token
Proxy User ACL
CVE tracking list
We might want to start a security section for Hadoop wiki for each of the
services and components.
This helps to track what has been completed.
How do
Looks good and +1 for markdown documentations to provide per release
specific information.
On Sat, Oct 21, 2017 at 8:47 AM, larry mccay wrote:
> New Revision...
>
> This revision acknowledges the reality that we often have multiple phases
> of feature lifecycle and that we need to account for ea
vice registry with a simple DNS
> > service to enable users to discover services deployed on YARN via
> standard
> > DNS lookup
> > - YARN-6419[4]. UI support for native-services on the new YARN UI
> > All these new services are optional and are
Hadoop default configuration aimed for user friendliness to increase adoption,
and security can be enabled one by one. This approach is most problematic to
security because system can be compromised before all security features are
turned on.
Larry's proposal will add some safety to remind sy
+1
On 7/5/18, 2:44 PM, "Giovanni Matteo Fumarola"
wrote:
+1
On Thu, Jul 5, 2018 at 2:41 PM, Wangda Tan wrote:
> +1
>
> On Thu, Jul 5, 2018 at 2:37 PM Subru Krishnan wrote:
>
> > Folks,
> >
> > There was a merge commit accidentally pushed to trunk
+1 on Non-routable IP idea. My preference is to start in Hadoop-common to
minimize the scope and incrementally improve. However, this will be
incompatible change for initial user experience on public cloud. What would be
the right release vehicle for this work (3.2+ or 4.x)?
Regards,
Eric
Hi Sunil,
For YARN service and docker related work, I like to propose the following
features to be merged:
YARN-7129 Application Catalog for YARN applications
YARN-7512 Support service upgrade via YARN Service API and CLI
YARN-8220 Running Tensorflow on YARN with GPU and Docker - Examples
Thank
Some lesson learn during 2.x. WebHDFS, HDFS ACL, QJM HA, rolling upgrade
are great features. Mapreduce 1.x uses resources more efficiently,
containers have rigid constraint, and applications get killed prematurely.
When a node has a lot of containers, YARN takes significant amount of
system resou
Option #4 is the most practical use case for making a release. For bleeding
edge developers, they would prefer to mix and match different version of hdfs
and mapreduce. Hence, it may be good to release the single tarball for
release, but continue to support component tarballs for developers an
By separating Hadoop docker related build into a separate git repository have
some slippery slope. It is harder to synchronize the changes between two
separate source trees. There is multi-steps process to build jar, tarball, and
docker images. This might be problematic to reproduce.
It woul
lish to the Docker Hub via Apache account?
>
>
> Thanks
> Anu
>
>
> On 1/29/19, 4:54 PM, "Eric Yang" wrote:
>
> By separating Hadoop docker related build into a separate git
repository have some slippery slope. It is harder
Thanks a lot,
Marton
On 1/30/19 6:50 PM, Eric Yang wrote:
> Hi Marton,
>
> Hi Marton,
>
> Flagging automated build on dockerhub seems conflicts with Apache release
policy. The vote and release process are manual processes of Apache Way.
Ther
Submarine is an application built for YARN framework, but it does not have
strong dependency on YARN development. For this kind of projects, it would be
best to enter Apache Incubator cycles to create a new community. Apache
commons is the only project other than Incubator that has independent
AM
To: Eric Yang
Cc: Weiwei Yang , Xun Liu , Hadoop
Common , "yarn-...@hadoop.apache.org"
, Hdfs-dev ,
"mapreduce-...@hadoop.apache.org"
Subject: Re: [DISCUSS] Making submarine to different release model like Ozone
Thanks everyone for sharing thoughts!
Eric, appreciate y
Hi Hadoop developers,
In the recent months, there were various discussions on creating docker build
process for Hadoop. There was convergence to make docker build process inline
in the mailing list last month when Ozone team is planning new repository for
Hadoop/ozone docker images. New featu
Loughran
Date: Monday, March 18, 2019 at 3:36 AM
To: Eric Yang
Cc: Hadoop Common , "yarn-...@hadoop.apache.org"
, Hdfs-dev , Eric
Badger , Eric Payne ,
Jonathan Eagles , Jim Brennan
, "Elek, Marton"
Subject: Re: [DISCUSS] Docker build process
I'm not enthusiastic abou
tifact with the same name
(eg. jdk8/jdk11). In Hadoop, profiles are used to introduce optional
steps. I think it's fine as the maven lifecycle/phase model is very
static (compare it with the tree based approach in Gradle).
Marton
[1]: https://issues.apache.org/jira/bro
or share a link to the description?
> I will make adjustment accordingly unless 7 more people comes
> out and say otherwise.
What adjustment is this?
Thanks,
Arpit
> On Mar 19, 2019, at 10:19 AM, Eric Yang wrote:
>
> Hi Ma
,
Eric
From: Jonathan Eagles
Date: Tuesday, March 19, 2019 at 11:48 AM
To: Eric Yang
Cc: "Elek, Marton" , Hadoop Common
, "yarn-...@hadoop.apache.org"
, Hdfs-dev , Eric
Badger , Eric Payne ,
Jim Brennan
Subject: Re: [DISCUSS] Docker build process
This email discussion thr
The flexibility of date appended release number is equivalent to maven snapshot
or Docker latest image convention, machine can apply timestamp better than
human. By using the Jenkins release process, this can be done with little
effort. For official release, it is best to use Docker image dige
+1
On 4/29/19, 9:05 PM, "Ajay Kumar" wrote:
Hi All,
We have created the third release candidate (RC2) for Apache Hadoop Ozone
0.4.0-alpha.
This release contains security payload for Ozone. Below are some important
features in it:
rfile would be
> adjusted).
>
> Marton
>
> ps: for the development (non published images) I am convinced that the
> optional docker profile can be an easier way to create images. Will
> create a similar plugin execution for this Dockerfile:
>
> https://github.com/apache/hadoop
Eric Yang created HDFS-8312:
---
Summary: Trash does not descent into child directories to check
for permissions
Key: HDFS-8312
URL: https://issues.apache.org/jira/browse/HDFS-8312
Project: Hadoop HDFS
Eric Yang created HDFS-4262:
---
Summary: Backport HTTPFS to Branch 1
Key: HDFS-4262
URL: https://issues.apache.org/jira/browse/HDFS-4262
Project: Hadoop HDFS
Issue Type: Improvement
Eric Yang created HDFS-7086:
---
Summary: httpfs create files default overwrite behavior is set to
true
Key: HDFS-7086
URL: https://issues.apache.org/jira/browse/HDFS-7086
Project: Hadoop HDFS
Issue
Eric Yang created HDDS-1455:
---
Summary: Inconsistent naming convention with Ozone Kerberos
configuration
Key: HDDS-1455
URL: https://issues.apache.org/jira/browse/HDDS-1455
Project: Hadoop Distributed Data
Eric Yang created HDDS-1457:
---
Summary: Sequence of configuring Ozone File System instruction
needs some work
Key: HDDS-1457
URL: https://issues.apache.org/jira/browse/HDDS-1457
Project: Hadoop Distributed
Eric Yang created HDDS-1458:
---
Summary: Create a maven profile to run fault injection tests
Key: HDDS-1458
URL: https://issues.apache.org/jira/browse/HDDS-1458
Project: Hadoop Distributed Data Store
Eric Yang created HDDS-1523:
---
Summary: Ozone Docker container phase 1
Key: HDDS-1523
URL: https://issues.apache.org/jira/browse/HDDS-1523
Project: Hadoop Distributed Data Store
Issue Type
[
https://issues.apache.org/jira/browse/HDDS-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang reopened HDDS-1520:
-
> Classpath assembly doesn't work if ozone is s
Eric Yang created HDDS-1548:
---
Summary: Jenkins precommit build is broken for Ozone
Key: HDDS-1548
URL: https://issues.apache.org/jira/browse/HDDS-1548
Project: Hadoop Distributed Data Store
Issue
[
https://issues.apache.org/jira/browse/HDDS-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang resolved HDDS-1548.
-
Resolution: Fixed
Jenkins Precommit build updated to reflect the changes.
> Jenkins precommit build
Eric Yang created HDDS-1554:
---
Summary: Create disk tests for fault injection test
Key: HDDS-1554
URL: https://issues.apache.org/jira/browse/HDDS-1554
Project: Hadoop Distributed Data Store
Issue
Eric Yang created HDDS-1563:
---
Summary: Freon error when running blockade test in maven
Key: HDDS-1563
URL: https://issues.apache.org/jira/browse/HDDS-1563
Project: Hadoop Distributed Data Store
Eric Yang created HDDS-1566:
---
Summary: Move docker image generation from dist project k8s-*
profiles
Key: HDDS-1566
URL: https://issues.apache.org/jira/browse/HDDS-1566
Project: Hadoop Distributed Data
Eric Yang created HDDS-1567:
---
Summary: Define a set of environment variables to configure Ozone
docker image
Key: HDDS-1567
URL: https://issues.apache.org/jira/browse/HDDS-1567
Project: Hadoop Distributed
Eric Yang created HDDS-1583:
---
Summary: EOFException for Ozone RPC client
Key: HDDS-1583
URL: https://issues.apache.org/jira/browse/HDDS-1583
Project: Hadoop Distributed Data Store
Issue Type: Bug
Eric Yang created HDDS-1588:
---
Summary: Update Ozone dist image to use jdk11
Key: HDDS-1588
URL: https://issues.apache.org/jira/browse/HDDS-1588
Project: Hadoop Distributed Data Store
Issue Type
Eric Yang created HDDS-1591:
---
Summary: Protocol buffer classes are incompatible between builds
Key: HDDS-1591
URL: https://issues.apache.org/jira/browse/HDDS-1591
Project: Hadoop Distributed Data Store
Eric Yang created HDDS-1609:
---
Summary: Remove hard coded uid from Ozone docker image
Key: HDDS-1609
URL: https://issues.apache.org/jira/browse/HDDS-1609
Project: Hadoop Distributed Data Store
Eric Yang created HDDS-1648:
---
Summary: Reduce Ozone docker image bloat
Key: HDDS-1648
URL: https://issues.apache.org/jira/browse/HDDS-1648
Project: Hadoop Distributed Data Store
Issue Type: Sub
[
https://issues.apache.org/jira/browse/HDDS-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang reopened HDDS-1565:
-
Reopen because the profile name was not what was agreed upon with other Hadoop
subproject. Please use
Eric Yang created HDDS-1661:
---
Summary: Consolidate hadoop-hdds and hadoop-ozone into
hadoop-ozone-project
Key: HDDS-1661
URL: https://issues.apache.org/jira/browse/HDDS-1661
Project: Hadoop Distributed
Eric Yang created HDDS-1667:
---
Summary: Docker compose file may referring to incorrect docker
image name
Key: HDDS-1667
URL: https://issues.apache.org/jira/browse/HDDS-1667
Project: Hadoop Distributed Data
Eric Yang created HDDS-1701:
---
Summary: Move dockerbin script to libexec
Key: HDDS-1701
URL: https://issues.apache.org/jira/browse/HDDS-1701
Project: Hadoop Distributed Data Store
Issue Type: Bug
Eric Yang created HDDS-1704:
---
Summary: Exercise Ozone tests in maven build
Key: HDDS-1704
URL: https://issues.apache.org/jira/browse/HDDS-1704
Project: Hadoop Distributed Data Store
Issue Type
Eric Yang created HDDS-1711:
---
Summary: Set a global reference property for Ozone image name
Key: HDDS-1711
URL: https://issues.apache.org/jira/browse/HDDS-1711
Project: Hadoop Distributed Data Store
Eric Yang created HDDS-1712:
---
Summary: Remove sudo access from Ozone docker image
Key: HDDS-1712
URL: https://issues.apache.org/jira/browse/HDDS-1712
Project: Hadoop Distributed Data Store
Issue
Eric Yang created HDDS-1734:
---
Summary: Use maven assembly to create ozone tarball image
Key: HDDS-1734
URL: https://issues.apache.org/jira/browse/HDDS-1734
Project: Hadoop Distributed Data Store
Eric Yang created HDDS-1769:
---
Summary: Update Chukwa dependency
Key: HDDS-1769
URL: https://issues.apache.org/jira/browse/HDDS-1769
Project: Hadoop Distributed Data Store
Issue Type: Bug
[
https://issues.apache.org/jira/browse/HDDS-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang resolved HDDS-1769.
-
Resolution: Invalid
Sorry, filed under wrong project.
> Update Chukwa depende
Eric Yang created HDDS-1771:
---
Summary: Add slow IO disk test to fault injection test
Key: HDDS-1771
URL: https://issues.apache.org/jira/browse/HDDS-1771
Project: Hadoop Distributed Data Store
Eric Yang created HDDS-1772:
---
Summary: Add disk full test to fault injection test
Key: HDDS-1772
URL: https://issues.apache.org/jira/browse/HDDS-1772
Project: Hadoop Distributed Data Store
Issue
Eric Yang created HDDS-1773:
---
Summary: Add intermitten IO disk test to fault injection test
Key: HDDS-1773
URL: https://issues.apache.org/jira/browse/HDDS-1773
Project: Hadoop Distributed Data Store
Eric Yang created HDDS-1774:
---
Summary: Add disk hang test to fault injection test
Key: HDDS-1774
URL: https://issues.apache.org/jira/browse/HDDS-1774
Project: Hadoop Distributed Data Store
Issue
[
https://issues.apache.org/jira/browse/HDDS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang reopened HDDS-1712:
-
Reopen because security is important.
> Remove sudo access from Ozone docker im
Eric Yang created HDDS-1825:
---
Summary: Kubernetes deployment starts only one data node by default
Key: HDDS-1825
URL: https://issues.apache.org/jira/browse/HDDS-1825
Project: Hadoop Distributed Data Store
Eric Yang created HDDS-1826:
---
Summary: External Ozone client throws exception when accessing
data in docker container
Key: HDDS-1826
URL: https://issues.apache.org/jira/browse/HDDS-1826
Project: Hadoop
Eric Yang created HDDS-1828:
---
Summary: smoke test core-site.xml is confusing to user
Key: HDDS-1828
URL: https://issues.apache.org/jira/browse/HDDS-1828
Project: Hadoop Distributed Data Store
[
https://issues.apache.org/jira/browse/HDDS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang reopened HDDS-1712:
-
> Remove sudo access from Ozone docker im
Eric Yang created HDDS-1847:
---
Summary: Datanode Kerberos principal and keytab config key looks
inconsistent
Key: HDDS-1847
URL: https://issues.apache.org/jira/browse/HDDS-1847
Project: Hadoop Distributed
Eric Yang created HDDS-1857:
---
Summary: YARN fails on mapreduce in Kerberos enabled cluster
Key: HDDS-1857
URL: https://issues.apache.org/jira/browse/HDDS-1857
Project: Hadoop Distributed Data Store
-1455
Project: Hadoop HDFS
Issue Type: New Feature
Reporter: Eric Yang
HDFS usage calculation is commonly calculated by running dfs -dus and group
directory usage by user at fix interval. This approach does not show accurate
HDFS usage if a lot of read/write
client
Environment: Java 6, CentOS 5.5
Reporter: Eric Yang
There is a hidden problem that hdfs client tries to initialize
SecurityAudit.audit log file which it shouldn't. The problem can be surfaced
by configuring HADOOP_LOG_DIR to a directory that is not writable b
[
https://issues.apache.org/jira/browse/HDFS-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang resolved HDFS-1644.
-
Resolution: Duplicate
Fix Version/s: 0.22.0
Resolved by HADOOP-6970.
> HDFS Client initial
Versions: 0.23.0
Environment: Java 6, RHEL 5.5
Reporter: Eric Yang
For trunk, the build and deployment tree look like this:
hadoop-common-0.2x.y
hadoop-hdfs-0.2x.y
hadoop-mapred-0.2x.y
Technically, hdfs's the third party dependent jar files should be fetch from
hadoop-c
Reporter: Eric Yang
Assignee: Eric Yang
This jira is corresponding to HADOOP-6255 and associated directory layout
change. The patch for creating HDFS rpm packaging should be posted here for
patch test build to verify against hdfs svn trunk.
--
This message is
Versions: 0.23.0
Environment: Java 6, Ubuntu
Reporter: Eric Yang
Assignee: Eric Yang
Fix For: 0.23.0
Java dependency for Debian package is specified as open JDK, but it should
depends on Sun version of Java. This dependency can be implicitly defined by
Affects Versions: 0.23.0
Reporter: Eric Yang
Assignee: Eric Yang
Fix For: 0.23.0
Attachments: HDFS-2277.patch
CLASSPATH for hdfs jar files should be set to HADOOP_PREFIX/share/hadoop/hdfs.
Trunk code is looking for jar files in HADOOP_PREFIX/share
72 matches
Mail list logo
