Marton please correct me I am wrong, but I believe that without this branch it
is hard for us to push to Apache DockerHub. This allows for Apache account
integration and dockerHub.
Does YARN publish to the Docker Hub via Apache account?
Thanks
Anu
On 1/29/19, 4:54 PM, "Eric Yang" <ey...@hortonworks.com> wrote:
By separating Hadoop docker related build into a separate git repository
have some slippery slope. It is harder to synchronize the changes between two
separate source trees. There is multi-steps process to build jar, tarball, and
docker images. This might be problematic to reproduce.
It would be best to arrange code such that docker image build process can
be invoked as part of maven build process. The profile is activated only if
docker is installed and running on the environment. This allows to produce
jar, tarball, and docker images all at once without hindering existing build
procedure.
YARN-7129 is one of the examples that making a subproject in YARN to build
a docker image that can run in YARN. It automatically detects presence of
docker and build docker image when docker is available. If docker is not
running, the subproject skips and proceed to next sub-project. Please try out
YARN-7129 style of build process, and see this is a possible solution to solve
docker image generation issue? Thanks
Regards,
Eric
On 1/29/19, 3:44 PM, "Arpit Agarwal" <aagar...@cloudera.com.INVALID> wrote:
I’ve requested a new repo hadoop-docker-ozone.git in gitbox.
> On Jan 22, 2019, at 4:59 AM, Elek, Marton <e...@apache.org> wrote:
>
>
>
> TLDR;
>
> I proposed to create a separated git repository for ozone docker
images
> in HDDS-851 (hadoop-docker-ozone.git)
>
> If there is no objections in the next 3 days I will ask an Apache
Member
> to create the repository.
>
>
>
>
> LONG VERSION:
>
> In HADOOP-14898 multiple docker containers and helper scripts are
> created for Hadoop.
>
> The main goal was to:
>
> 1.) help the development with easy-to-use docker images
> 2.) provide official hadoop images to make it easy to test new
features
>
> As of now we have:
>
> - apache/hadoop-runner image (which contains the required dependency
> but no hadoop)
> - apache/hadoop:2 and apache/hadoop:3 images (to try out latest hadoop
> from 2/3 lines)
>
> The base image to run hadoop (apache/hadoop-runner) is also heavily
used
> for Ozone distribution/development.
>
> The Ozone distribution contains docker-compose based cluster
definitions
> to start various type of clusters and scripts to do smoketesting. (See
> HADOOP-16063 for more details).
>
> Note: I personally believe that these definitions help a lot to start
> different type of clusters. For example it could be tricky to try out
> router based federation as it requires multiple HA clusters. But with
a
> simple docker-compose definition [1] it could be started under 3
> minutes. (HADOOP-16063 is about creating these definitions for various
> hdfs/yarn use cases)
>
> As of now we have dedicated branches in the hadoop git repository for
> the docker images (docker-hadoop-runner, docker-hadoop-2,
> docker-hadoop-3). It turns out that a separated repository would be
more
> effective as the dockerhub can use only full branch names as tags.
>
> We would like to provide ozone docker images to make the evaluation as
> easy as 'docker run -d apache/hadoop-ozone:0.3.0', therefore in
HDDS-851
> we agreed to create a separated repository for the hadoop-ozone docker
> images.
>
> If this approach works well we can also move out the existing
> docker-hadoop-2/docker-hadoop-3/docker-hadoop-runner branches from
> hadoop.git to an other separated hadoop-docker.git repository)
>
> Please let me know if you have any comments,
>
> Thanks,
> Marton
>
> 1: see
> https://github.com/flokkr/runtime-compose/tree/master/hdfs/routerfeder
> as an example
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
