On Mon, Aug 28, 2017 at 2:04 AM, Francesco Caliumi - Diennea < francesco.cali...@diennea.com> wrote:
> Hi Sijie, here the reply: > > > >> You only build the docker image locally, right? I am not sure how can > you > >> distribute the docker image in a multiple node or a cloud environment. > > > Yes, the scenario I described is only for lacal tests. If you want to > distribute the docker image you can push it to docker hub (maybe on an ad > hoc user for the purpose, like for example "bookkeeper-tests") after build, > but I've never personally set up this kind of automation though. > > However, locally you can run a whole ensemble. What is the use case you > have in mind for a cloud environment? Benchmarks? > We are trying to setup a CI/CD pipeline for bookkeeper. The ideal pipeline would : unit test => build a docker image => integration test => jepsen test => benchmark. After unit tests, it is good to deploy the bookkeeper (using the built docker image) to a small cluster (either on cloud or another other environment), run end-to-end integration test (maybe also backward compatibility testsing), failure (jepsen) tests and then benchmark. You need a nightly image for this purpose. A more ideal situation for me - for release image, we should only pick a image that pass all the CI/CD pipeline and tag it as release. The current `download-package` approach to build release docker images doesn't sound promising to me. - Sijie > > On Mon, 2017-08-28 at 01:05 -0700, Sijie Guo wrote: > > Hi Francesco, > > Sorry. I can't read what are your reply in this email. It seems the format > was lost ... > > - Sijie > > On Mon, Aug 28, 2017 at 12:32 AM, Francesco Caliumi - Diennea < > francesco.cali...@diennea.com<mailto:francesco.cali...@diennea.com>> > wrote: > > > > On Fri, 2017-08-25 at 08:02 -0700, Sijie Guo wrote: > > On Aug 25, 2017 7:33 AM, "Francesco Caliumi - Diennea" < > francesco.cali...@diennea.com<mailto:francesco.cali...@diennea.com > ><mailto:francesco.cali...@diennea.com>> > wrote: > > On Thu, 2017-08-24 at 11:22 -0700, Sijie Guo wrote: > > On Tue, Aug 22, 2017 at 12:30 AM, Francesco Caliumi - Diennea < > francesco.cali...@diennea.com<mailto:francesco.cali...@diennea.com > ><mailto:francesco.cali...@diennea.com > > > <mailto:francesco.cali...@diennea.com>> wrote: > > > > > > I'm not sure if I explained well my point. No docker hub involved, just a > tool for every developer who wants to build an image by its own. Maybe to > test the stability of a feature branch instead of master, for instance. > > > > > If we don't push to docker hub, how can this image be used for continuous > testing. The idea is to have a nightly image, so this image can be deployed > to somewhere to do integration test, jepsen test and other things. > > > > I'm not familiar with this use case, but I think the image can be builded > before starting integration tests, without any need of docker hub at all > (but it adds a couple of minutes to overall test time). > > This way integration tests could be executed on master and on branches as > well. > > > You only build the docker image locally, right? I am not sure how can you > distribute the docker image in a multiple node or a cloud environment. > > > > Yes, the scenario I described is only for lacal tests. If you want to > distribute the docker image you can push it to docker hub (maybe on an ad > hoc user for the purpose, like for example "bookkeeper-tests") after build, > but I've never personally set up this kind of automation though. > > However, locally you can run a whole ensemble. What is the use case you > have in mind for a cloud environment? Benchmarks? > > > > > > > Maybe it could help to well define the use case scenario and the target > user. If target user is a developer or tester that needs to test a bk > ensemble, having available only the one day delayed build of the master > branch could be limiting. > > > > > > > > > On Tue, 2017-08-22 at 14:38 +0800, Jia Zhai wrote: > > Hi Sijie, > For releases version, we have no permissions to do "docker push" images > into wanted place(neither under apache <https://hub.docker.com/u/apache/> > nor "Official <https://docs.docker.com/docker-hub/official_repos/>"). > > Thanks. > -Jia > > > > On Tue, Aug 22, 2017 at 12:55 PM, Sijie Guo <guosi...@gmail.com<mailto:guo > si...@gmail.com><mailto:guo > si...@gmail.com<mailto:si...@gmail.com>><mailto:guo > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com>><mailto: > guo > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com><mailto: > si...@gmail.com>>> wrote: > > > > On Aug 21, 2017 12:56 AM, "Francesco Caliumi - Diennea" < > francesco.cali...@diennea.com<mailto:francesco.cali...@diennea.com > ><mailto:francesco.cali...@diennea.com > > > <mailto:francesco.cali...@diennea.com><mailto: > > > francesco.cali...@diennea.com<mailto:francesco.cali...@diennea.com > ><mailto:francesco.cali...@diennea.com>>> > wrote: > > My view for these topics: > > 1) I'm a big fan of official docker images, so it's ok for me. > > 2) For latest build, both Jia and Enrico solutions are ok, but I think > Enrico's maven target one is the most flexible. > Who usually need the nightly build are developers, who do not care very > much to have an official or almost-official image, but they like the > flexibility to build the image of the commit / branch they want to test. > > > Having a maven target should create a workflow like this: > > - do some work or git checkout {<branch> / <commit>} > - mvn clean package docker:build > - (optional: only if working with multiple versions) docker tag > <generated_image> <dev_docker_user>/bookkeeper:mytag > - (optional: only if the image should be accessibile by others devs / > machines) docker push <dev_docker_user>/bookkeeper:mytag > - test the image > > On functionality side, this image will be slightly different from the > current one, because it will not need to download the package but it will > find it directly in the build dir. > > > Personally I'd like to have a single way to build images for nightly and > releases. It is going to be hard to maintain two solutions in long term. > > > On Tue, 2017-08-15 at 00:49 -0700, Sijie Guo wrote: > > On Tue, Aug 15, 2017 at 12:38 AM, Enrico Olivelli <eolive...@gmail.com > <mailto:eolive...@gmail.com> > <mailto:eolive...@gmail.com> > <mailto:eolive...@gmail.com> > <mailto:eolive...@gmail.com> > <mailto:eolive...@gmail.com>> > wrote: > > > > Il mar 15 ago 2017, 09:16 Sijie Guo <guosi...@gmail.com<mailto:guo > si...@gmail.com><mailto:guo > si...@gmail.com<mailto:si...@gmail.com>><mailto:guo > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com>><mailto: > guo > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com><mailto: > si...@gmail.com>><mailto: > guo > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com><mailto: > si...@gmail.com><mailto: > si...@gmail.com<mailto:si...@gmail.com>>>> ha > scritto: > > > > On Aug 14, 2017 11:18 PM, "Enrico Olivelli" <eolive...@gmail.com<mailto:eo > live...@gmail.com><mailto:eo > live...@gmail.com<mailto:live...@gmail.com>><mailto:eo > live...@gmail.com<mailto:live...@gmail.com><mailto:live...@gmail.com > >><mailto:eo > live...@gmail.com<mailto:live...@gmail.com><mailto:live...@gmail.com > ><mailto:live...@gmail.com > > > > > <mailto:eo > > > > > live...@gmail.com<mailto:live...@gmail.com><mailto:live...@gmail.com > ><mailto:live...@gmail.com > > > <mailto:live...@gmail.com>>> > > > wrote: > > Il mar 15 ago 2017, 03:29 Jia Zhai <zhaiji...@gmail.com<mailto:zh > aiji...@gmail.com><mailto:zh > aiji...@gmail.com<mailto:aiji...@gmail.com>><mailto:zh > aiji...@gmail.com<mailto:aiji...@gmail.com><mailto:aiji...@gmail.com > >><mailto:zh > aiji...@gmail.com<mailto:aiji...@gmail.com><mailto:aiji...@gmail.com > ><mailto:aiji...@gmail.com > > > > > <mailto:zh > > > > > aiji...@gmail.com<mailto:aiji...@gmail.com><mailto:aiji...@gmail.com > ><mailto:aiji...@gmail.com > > > <mailto:aiji...@gmail.com>>> ha > > > scritto: > > > > Hi Sijie, > From my view, the approach problem is whether we have permissions to > "docker push" images into wanted place. > Following the way putting images under "apache", we seems not have > permission, While following the "Official > <https://docs.docker.com/docker-hub/official_repos/>" way, seems > > > neither. > > > > Since nightly build is mainly for our development, It maybe OK to > > > > > manage > > > > > and maintain a dockerhub account by our community to hold the nightly > one(maybe also for the release images). > > > > On Tue, Aug 15, 2017 at 8:19 AM, Sijie Guo <guosi...@gmail.com<mailto:guo > si...@gmail.com><mailto:guo > si...@gmail.com<mailto:si...@gmail.com>><mailto:guo > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com>><mailto: > guo > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com><mailto: > si...@gmail.com>><mailto: > guo > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com><mailto: > si...@gmail.com><mailto: > si...@gmail.com<mailto:si...@gmail.com>>>> wrote: > > > > On Mon, Aug 14, 2017 at 3:47 AM, Jia Zhai <zhaiji...@gmail.com<mailto:zh > aiji...@gmail.com><mailto:zh > aiji...@gmail.com<mailto:aiji...@gmail.com>><mailto:zh > aiji...@gmail.com<mailto:aiji...@gmail.com><mailto:aiji...@gmail.com > >><mailto:zh > aiji...@gmail.com<mailto:aiji...@gmail.com><mailto:aiji...@gmail.com > ><mailto:aiji...@gmail.com > > > > > <mailto:zh > > > > > aiji...@gmail.com<mailto:aiji...@gmail.com><mailto:aiji...@gmail.com > ><mailto:aiji...@gmail.com > > > <mailto:aiji...@gmail.com>>> > > > > > > > > > wrote: > > > > > > > > > > Thanks Sijie for raising these good topics up. > > Regarding 1) official image, the Flink one is similar as Zookeeper > > > > > > > one, > > > > > > > which we have discussed before. If the currently approach could not > > > > > make > > > > > bookkeeper docker image official, we should go this way. > > > > > > > Regarding 2) nightly build, there is already an issue > <https://github.com/apache/bookkeeper/issues/289> opened. Seems > > > > > > > > > the > > > > > > > issue > > > is where to put the nightly build images, since for both way of > > > > > official > > > > > image, there is limited access to the dockerhub, The first thought > > > > > > > > > in > > > > > my > > > > > head is to place a nightly build somewhere, such as ( > https://dist.apache.org/repos/dist/dev/bookkeeper/), and the > > > > > > > > > current > > > > > > > > > docker > file will not changed too much, seems only some env var need > > > > > > > > > change. > > > > > > > > > > > > > currently we don't have any process to produce any nightly built > > > packages. > > > > If we are planning to use dist/dev for hosting the nightly build, we > > > > > need > > > > > to figure out how to get the credentials to do that. > Because the dist/dev is a svn repo, during a release, the release > > > > > manager > > > > > uses its own credentials to commit the new packages to the svn repo. > > > > > > Regarding 3), the build failure > <https://hub.docker.com/r/apache/bookkeeper/builds/ > bvzft3fsnpmmj5i8jnpk3fl/> > is caused by connection(from dockerhub to gpg key server) issue. It > > > > > > > is > > > > > also > > > one reason to come out PR420 <https://github.com/apache/ > bookkeeper/pull/420>, > which wanted to download local KEY to avoid gpg server connection, > > > > > > > But > > > we > > > > > agree that it is not very security. > > > > > I think there are a few drawbacks that I can see in current approach: > > - building the packages and building the docker images are managed by > > > > > two > > > > > different systems. > - the `latest` image isn't really a tag pointing to the image of > > > > > > > latest > > > > > > > release; the latest is actually building from master. any changes > > > > > pushed > > > to > > > master unnecessarily trigger auto build in docker hub, even the > > > > > > > package > > > > > > > itself isn't changed. > > > > > "latest" is some kind of place holder, I thought to use it to point to > > > the > > > nightly version, seems I was mis-understanding of it. > > > > > My two cents > IMHO latest should be v the latest stable version > > > Yes that's my thought as well. Then it should be just a tag for the image > of latest release. However in current approach, it keeps regenerating a > image everytime there is a change on master. This doesn't make any sense > > > to > > > me. > > > > > We have to touch the latest tag only as part of the release process. > > > > > Currently latest is not just a tag of 4.5.0. they are separate images > generated by docker auto builds. > > > > > > > > > > > One step back, what is the purpose of having this image? I mean an image > > > of > > > the master branch. > > > It can be used in a CD (continuous deployment) pipeline, you can deploy > > > the > > > image to a dockerized environment to verify if it is working or not. > > > > > Yup, so it is like the maven snapshots. > From the licensing point in that case all the constraints are relaxed, we > can really use a PMC managed repository > > > > > > If you only need to have a working bookkeeper server for docker we can > > > use > > > the maven docker plugin which is well integreated with maven based > environments. The problem of where to put is is still here. Sometime ago > for a project I had just created a free account on docker hub and pushed > there the image directly from the build. > Jia already created a BookKeeper account, we could use it. > Credentials will be managed by PMC > > > Enrico > > > > > > > > - building a nightly package is not trivial. > > If we step back and revisit my comment in > https://github.com/apache/bookkeeper/pull/197#issuecomment-317831799 > > > > > > > , > > > if > > > > > we > use `docker push` approach and let jenkins build and push docker > > > > > images, > > > it > > > seems to be much easier > to address the above issues. > > Thoughts? > > - Sijie > > > > > > > > -- > > > -- Enrico Olivelli > > > > -- > > > -- Enrico Olivelli > > > > -- > > Francesco Caliumi > Developer @ Diennea - MagNews > Tel.: (+39) 0546 066100 - Int. 266 > Viale G.Marconi 30/14 - 48018 Faenza (RA) > > [Magnews.it]<http://www.magnews.it/it> > > [Linkedin]<http://www.linkedin.com/company/diennea---magnews> > [Twitter] > <http://twitter.com/DienneaMagNews> [Facebook] < > http://www.facebook.com/pages/MagNews/197617841797> [Newsletter] < > http://www.magnews.it/it/iscriviti-alla-newsletter> > > ________________________________ > > Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed > email marketing! http://www.magnews.it/newsletter/ > > The information in this email is confidential and may be legally > privileged. If you are not the intended recipient please notify the sender > immediately and destroy this email. Any unauthorized, direct or indirect, > disclosure, copying, storage, distribution or other use is strictly > forbidden. > > > > -- > > Francesco Caliumi > Developer @ Diennea - MagNews > Tel.: (+39) 0546 066100 - Int. 266 > Viale G.Marconi 30/14 - 48018 Faenza (RA) > > [Magnews.it]<http://www.magnews.it/it> > > [Linkedin]<http://www.linkedin.com/company/diennea---magnews> > [Twitter] <http://twitter.com/DienneaMagNews> [Facebook] < > http://www.facebook.com/pages/MagNews/197617841797> [Newsletter] < > http://www.magnews.it/it/iscriviti-alla-newsletter> > > ________________________________ > > Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed > email marketing! http://www.magnews.it/newsletter/ > > The information in this email is confidential and may be legally > privileged. If you are not the intended recipient please notify the sender > immediately and destroy this email. Any unauthorized, direct or indirect, > disclosure, copying, storage, distribution or other use is strictly > forbidden. > > > > -- > > Francesco Caliumi > Developer @ Diennea - MagNews > Tel.: (+39) 0546 066100 - Int. 266 > Viale G.Marconi 30/14 - 48018 Faenza (RA) > > [Magnews.it]<http://www.magnews.it/it> > > [Linkedin]<http://www.linkedin.com/company/diennea---magnews> > [Twitter] > <http://twitter.com/DienneaMagNews> [Facebook] < > http://www.facebook.com/pages/MagNews/197617841797> [Newsletter] < > http://www.magnews.it/it/iscriviti-alla-newsletter> > > ________________________________ > > Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed > email marketing! http://www.magnews.it/newsletter/ > > The information in this email is confidential and may be legally > privileged. If you are not the intended recipient please notify the sender > immediately and destroy this email. Any unauthorized, direct or indirect, > disclosure, copying, storage, distribution or other use is strictly > forbidden. > > > -- > > Francesco Caliumi > Developer @ Diennea - MagNews > Tel.: (+39) 0546 066100 - Int. 266 > Viale G.Marconi 30/14 - 48018 Faenza (RA) > > [Magnews.it]<http://www.magnews.it/it> > > [Linkedin]<http://www.linkedin.com/company/diennea---magnews> > [Twitter] <http://twitter.com/DienneaMagNews> [Facebook] < > http://www.facebook.com/pages/MagNews/197617841797> [Newsletter] < > http://www.magnews.it/it/iscriviti-alla-newsletter> > > ________________________________ > > Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed > email marketing! http://www.magnews.it/newsletter/ > > The information in this email is confidential and may be legally > privileged. If you are not the intended recipient please notify the sender > immediately and destroy this email. Any unauthorized, direct or indirect, > disclosure, copying, storage, distribution or other use is strictly > forbidden. > > > > -- > > Francesco Caliumi > Developer @ Diennea - MagNews > Tel.: (+39) 0546 066100 - Int. 266 > Viale G.Marconi 30/14 - 48018 Faenza (RA) > > [Magnews.it]<http://www.magnews.it/it> > > [Linkedin]<http://www.linkedin.com/company/diennea---magnews> > [Twitter] <http://twitter.com/DienneaMagNews> [Facebook] < > http://www.facebook.com/pages/MagNews/197617841797> [Newsletter] < > http://www.magnews.it/it/iscriviti-alla-newsletter> > > ________________________________ > > Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed > email marketing! http://www.magnews.it/newsletter/ > > The information in this email is confidential and may be legally > privileged. If you are not the intended recipient please notify the sender > immediately and destroy this email. Any unauthorized, direct or indirect, > disclosure, copying, storage, distribution or other use is strictly > forbidden. >
