On Mon, 2017-08-28 at 02:19 -0700, Sijie Guo wrote: > On Mon, Aug 28, 2017 at 2:04 AM, Francesco Caliumi - Diennea < > francesco.cali...@diennea.com> wrote: > > > Hi Sijie, here the reply: > > > > > > > > You only build the docker image locally, right? I am not sure > > > > how can > > > > you > > > > distribute the docker image in a multiple node or a cloud > > > > environment. > > > > > > Yes, the scenario I described is only for lacal tests. If you want > > to > > distribute the docker image you can push it to docker hub (maybe on > > an ad > > hoc user for the purpose, like for example "bookkeeper-tests") > > after build, > > but I've never personally set up this kind of automation though. > > > > However, locally you can run a whole ensemble. What is the use case > > you > > have in mind for a cloud environment? Benchmarks? > > > > > We are trying to setup a CI/CD pipeline for bookkeeper. The ideal > pipeline > would : unit test => build a docker image => integration test => > jepsen > test => benchmark. After unit tests, it is good to deploy the > bookkeeper > (using the built docker image) to a small cluster (either on cloud or > another other environment), run end-to-end integration test (maybe > also > backward compatibility testsing), failure (jepsen) tests and then > benchmark. > Thanks for the explanation, it's now clearer what will happen.
> You need a nightly image for this purpose. A docker hub image is indeed way more easier to manage for bring up a cluster. If we manage to "docker push" the image after the build then the complexity will be the same cluster side. One thing to consider: build from docker hub can be triggered, but I don't know if it is possible to know when it finish. If it isn't, tests should be run with a certain delay after the trigger. Docker push approach doesn't have this inconvenient. > A more ideal situation for me - > for release image, we should only pick a image that pass all the > CI/CD > pipeline and tag it as release. The current `download-package` > approach to > build release docker images doesn't sound promising to me. > I agree that the current `download-package` approach is very rigid for testing. The proposal was actually to create a maven target that picks up the just built tar package and load it directly in the workspace of the image, removing the need to download it from a repository (Enrico already played with it in HerdDB project and it worked fine). The Dockerfile of this build will be simpler then the release one, so it shouldn't take much effort to create it. I think flexibility wins in test scenarios, but is up to you. > > > > > > On Mon, 2017-08-28 at 01:05 -0700, Sijie Guo wrote: > > > > Hi Francesco, > > > > Sorry. I can't read what are your reply in this email. It seems the > > format > > was lost ... > > > > - Sijie > > > > On Mon, Aug 28, 2017 at 12:32 AM, Francesco Caliumi - Diennea < > > francesco.cali...@diennea.com<mailto:francesco.cali...@diennea.com> > > > > > wrote: > > > > > > > > On Fri, 2017-08-25 at 08:02 -0700, Sijie Guo wrote: > > > > On Aug 25, 2017 7:33 AM, "Francesco Caliumi - Diennea" < > > francesco.cali...@diennea.com<mailto:francesco.cali...@diennea.com > > > <mailto:francesco.cali...@diennea.com>> > > > > wrote: > > > > On Thu, 2017-08-24 at 11:22 -0700, Sijie Guo wrote: > > > > On Tue, Aug 22, 2017 at 12:30 AM, Francesco Caliumi - Diennea < > > francesco.cali...@diennea.com<mailto:francesco.cali...@diennea.com > > > <mailto:francesco.cali...@diennea.com > > > > > > <mailto:francesco.cali...@diennea.com>> wrote: > > > > > > > > > > > > I'm not sure if I explained well my point. No docker hub involved, > > just a > > tool for every developer who wants to build an image by its own. > > Maybe to > > test the stability of a feature branch instead of master, for > > instance. > > > > > > > > > > If we don't push to docker hub, how can this image be used for > > continuous > > testing. The idea is to have a nightly image, so this image can be > > deployed > > to somewhere to do integration test, jepsen test and other things. > > > > > > > > I'm not familiar with this use case, but I think the image can be > > builded > > before starting integration tests, without any need of docker hub > > at all > > (but it adds a couple of minutes to overall test time). > > > > This way integration tests could be executed on master and on > > branches as > > well. > > > > > > You only build the docker image locally, right? I am not sure how > > can you > > distribute the docker image in a multiple node or a cloud > > environment. > > > > > > > > Yes, the scenario I described is only for lacal tests. If you want > > to > > distribute the docker image you can push it to docker hub (maybe on > > an ad > > hoc user for the purpose, like for example "bookkeeper-tests") > > after build, > > but I've never personally set up this kind of automation though. > > > > However, locally you can run a whole ensemble. What is the use case > > you > > have in mind for a cloud environment? Benchmarks? > > > > > > > > > > > > > > Maybe it could help to well define the use case scenario and the > > target > > user. If target user is a developer or tester that needs to test a > > bk > > ensemble, having available only the one day delayed build of the > > master > > branch could be limiting. > > > > > > > > > > > > > > > > > > On Tue, 2017-08-22 at 14:38 +0800, Jia Zhai wrote: > > > > Hi Sijie, > > For releases version, we have no permissions to do "docker push" > > images > > into wanted place(neither under apache <https://hub.docker.com/u/ap > > ache/> > > nor "Official <https://docs.docker.com/docker-hub/official_repos/> > > "). > > > > Thanks. > > -Jia > > > > > > > > On Tue, Aug 22, 2017 at 12:55 PM, Sijie Guo <guosi...@gmail.com<mai > > lto:guo > > si...@gmail.com><mailto:guo > > si...@gmail.com<mailto:si...@gmail.com>><mailto:guo > > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com>><ma > > ilto: > > guo > > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com><mai > > lto: > > si...@gmail.com>>> wrote: > > > > > > > > On Aug 21, 2017 12:56 AM, "Francesco Caliumi - Diennea" < > > francesco.cali...@diennea.com<mailto:francesco.cali...@diennea.com > > > <mailto:francesco.cali...@diennea.com > > > > > > <mailto:francesco.cali...@diennea.com><mailto: > > > > > > francesco.cali...@diennea.com<mailto:francesco.cali...@diennea.com > > > <mailto:francesco.cali...@diennea.com>>> > > > > wrote: > > > > My view for these topics: > > > > 1) I'm a big fan of official docker images, so it's ok for me. > > > > 2) For latest build, both Jia and Enrico solutions are ok, but I > > think > > Enrico's maven target one is the most flexible. > > Who usually need the nightly build are developers, who do not care > > very > > much to have an official or almost-official image, but they like > > the > > flexibility to build the image of the commit / branch they want to > > test. > > > > > > Having a maven target should create a workflow like this: > > > > - do some work or git checkout {<branch> / <commit>} > > - mvn clean package docker:build > > - (optional: only if working with multiple versions) docker tag > > <generated_image> <dev_docker_user>/bookkeeper:mytag > > - (optional: only if the image should be accessibile by others devs > > / > > machines) docker push <dev_docker_user>/bookkeeper:mytag > > - test the image > > > > On functionality side, this image will be slightly different from > > the > > current one, because it will not need to download the package but > > it will > > find it directly in the build dir. > > > > > > Personally I'd like to have a single way to build images for > > nightly and > > releases. It is going to be hard to maintain two solutions in long > > term. > > > > > > On Tue, 2017-08-15 at 00:49 -0700, Sijie Guo wrote: > > > > On Tue, Aug 15, 2017 at 12:38 AM, Enrico Olivelli <eolivelli@gmail. > > com > > <mailto:eolive...@gmail.com> > > <mailto:eolive...@gmail.com> > > <mailto:eolive...@gmail.com> > > <mailto:eolive...@gmail.com> > > <mailto:eolive...@gmail.com>> > > wrote: > > > > > > > > Il mar 15 ago 2017, 09:16 Sijie Guo <guosi...@gmail.com<mailto:guo > > si...@gmail.com><mailto:guo > > si...@gmail.com<mailto:si...@gmail.com>><mailto:guo > > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com>><ma > > ilto: > > guo > > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com><mai > > lto: > > si...@gmail.com>><mailto: > > guo > > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com><mai > > lto: > > si...@gmail.com><mailto: > > si...@gmail.com<mailto:si...@gmail.com>>>> ha > > scritto: > > > > > > > > On Aug 14, 2017 11:18 PM, "Enrico Olivelli" <eolive...@gmail.com<ma > > ilto:eo > > live...@gmail.com><mailto:eo > > live...@gmail.com<mailto:live...@gmail.com>><mailto:eo > > live...@gmail.com<mailto:live...@gmail.com><mailto:live...@gmail.co > > m > > > > <mailto:eo > > > > live...@gmail.com<mailto:live...@gmail.com><mailto:live...@gmail.co > > m > > > <mailto:live...@gmail.com > > > > > > > > > > <mailto:eo > > > > > > > > > > live...@gmail.com<mailto:live...@gmail.com><mailto:live...@gmail.co > > m > > > <mailto:live...@gmail.com > > > > > > <mailto:live...@gmail.com>>> > > > > > > wrote: > > > > Il mar 15 ago 2017, 03:29 Jia Zhai <zhaiji...@gmail.com<mailto:zh > > aiji...@gmail.com><mailto:zh > > aiji...@gmail.com<mailto:aiji...@gmail.com>><mailto:zh > > aiji...@gmail.com<mailto:aiji...@gmail.com><mailto:aiji...@gmail.co > > m > > > > <mailto:zh > > > > aiji...@gmail.com<mailto:aiji...@gmail.com><mailto:aiji...@gmail.co > > m > > > <mailto:aiji...@gmail.com > > > > > > > > > > <mailto:zh > > > > > > > > > > aiji...@gmail.com<mailto:aiji...@gmail.com><mailto:aiji...@gmail.co > > m > > > <mailto:aiji...@gmail.com > > > > > > <mailto:aiji...@gmail.com>>> ha > > > > > > scritto: > > > > > > > > Hi Sijie, > > From my view, the approach problem is whether we have permissions > > to > > "docker push" images into wanted place. > > Following the way putting images under "apache", we seems not > > have > > permission, While following the "Official > > <https://docs.docker.com/docker-hub/official_repos/>" way, seems > > > > > > neither. > > > > > > > > Since nightly build is mainly for our development, It maybe OK to > > > > > > > > > > manage > > > > > > > > > > and maintain a dockerhub account by our community to hold the > > nightly > > one(maybe also for the release images). > > > > > > > > On Tue, Aug 15, 2017 at 8:19 AM, Sijie Guo <guosi...@gmail.com<mail > > to:guo > > si...@gmail.com><mailto:guo > > si...@gmail.com<mailto:si...@gmail.com>><mailto:guo > > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com>><ma > > ilto: > > guo > > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com><mai > > lto: > > si...@gmail.com>><mailto: > > guo > > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com><mai > > lto: > > si...@gmail.com><mailto: > > si...@gmail.com<mailto:si...@gmail.com>>>> wrote: > > > > > > > > On Mon, Aug 14, 2017 at 3:47 AM, Jia Zhai <zhaiji...@gmail.com<mail > > to:zh > > aiji...@gmail.com><mailto:zh > > aiji...@gmail.com<mailto:aiji...@gmail.com>><mailto:zh > > aiji...@gmail.com<mailto:aiji...@gmail.com><mailto:aiji...@gmail.co > > m > > > > <mailto:zh > > > > aiji...@gmail.com<mailto:aiji...@gmail.com><mailto:aiji...@gmail.co > > m > > > <mailto:aiji...@gmail.com > > > > > > > > > > <mailto:zh > > > > > > > > > > aiji...@gmail.com<mailto:aiji...@gmail.com><mailto:aiji...@gmail.co > > m > > > <mailto:aiji...@gmail.com > > > > > > <mailto:aiji...@gmail.com>>> > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Thanks Sijie for raising these good topics up. > > > > Regarding 1) official image, the Flink one is similar as Zookeeper > > > > > > > > > > > > > > one, > > > > > > > > > > > > > > which we have discussed before. If the currently approach could not > > > > > > > > > > make > > > > > > > > > > bookkeeper docker image official, we should go this way. > > > > > > > > > > > > > > Regarding 2) nightly build, there is already an issue > > <https://github.com/apache/bookkeeper/issues/289> opened. Seems > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > issue > > > > > > is where to put the nightly build images, since for both way of > > > > > > > > > > official > > > > > > > > > > image, there is limited access to the dockerhub, The first thought > > > > > > > > > > > > > > > > > > in > > > > > > > > > > my > > > > > > > > > > head is to place a nightly build somewhere, such as ( > > https://dist.apache.org/repos/dist/dev/bookkeeper/), and the > > > > > > > > > > > > > > > > > > current > > > > > > > > > > > > > > > > > > docker > > file will not changed too much, seems only some env var need > > > > > > > > > > > > > > > > > > change. > > > > > > > > > > > > > > > > > > > > > > > > > > currently we don't have any process to produce any nightly built > > > > > > packages. > > > > > > > > If we are planning to use dist/dev for hosting the nightly build, > > we > > > > > > > > > > need > > > > > > > > > > to figure out how to get the credentials to do that. > > Because the dist/dev is a svn repo, during a release, the release > > > > > > > > > > manager > > > > > > > > > > uses its own credentials to commit the new packages to the svn > > repo. > > > > > > > > > > > > Regarding 3), the build failure > > <https://hub.docker.com/r/apache/bookkeeper/builds/ > > bvzft3fsnpmmj5i8jnpk3fl/> > > is caused by connection(from dockerhub to gpg key server) issue. It > > > > > > > > > > > > > > is > > > > > > > > > > also > > > > > > one reason to come out PR420 <https://github.com/apache/ > > bookkeeper/pull/420>, > > which wanted to download local KEY to avoid gpg server connection, > > > > > > > > > > > > > > But > > > > > > we > > > > > > > > > > agree that it is not very security. > > > > > > > > > > I think there are a few drawbacks that I can see in current > > approach: > > > > - building the packages and building the docker images are managed > > by > > > > > > > > > > two > > > > > > > > > > different systems. > > - the `latest` image isn't really a tag pointing to the image of > > > > > > > > > > > > > > latest > > > > > > > > > > > > > > release; the latest is actually building from master. any changes > > > > > > > > > > pushed > > > > > > to > > > > > > master unnecessarily trigger auto build in docker hub, even the > > > > > > > > > > > > > > package > > > > > > > > > > > > > > itself isn't changed. > > > > > > > > > > "latest" is some kind of place holder, I thought to use it to point > > to > > > > > > the > > > > > > nightly version, seems I was mis-understanding of it. > > > > > > > > > > My two cents > > IMHO latest should be v the latest stable version > > > > > > Yes that's my thought as well. Then it should be just a tag for the > > image > > of latest release. However in current approach, it keeps > > regenerating a > > image everytime there is a change on master. This doesn't make any > > sense > > > > > > to > > > > > > me. > > > > > > > > > > We have to touch the latest tag only as part of the release > > process. > > > > > > > > > > Currently latest is not just a tag of 4.5.0. they are separate > > images > > generated by docker auto builds. > > > > > > > > > > > > > > > > > > > > > > One step back, what is the purpose of having this image? I mean an > > image > > > > > > of > > > > > > the master branch. > > > > > > It can be used in a CD (continuous deployment) pipeline, you can > > deploy > > > > > > the > > > > > > image to a dockerized environment to verify if it is working or > > not. > > > > > > > > > > Yup, so it is like the maven snapshots. > > From the licensing point in that case all the constraints are > > relaxed, we > > can really use a PMC managed repository > > > > > > > > > > > > If you only need to have a working bookkeeper server for docker we > > can > > > > > > use > > > > > > the maven docker plugin which is well integreated with maven based > > environments. The problem of where to put is is still here. > > Sometime ago > > for a project I had just created a free account on docker hub and > > pushed > > there the image directly from the build. > > Jia already created a BookKeeper account, we could use it. > > Credentials will be managed by PMC > > > > > > Enrico > > > > > > > > > > > > > > > > - building a nightly package is not trivial. > > > > If we step back and revisit my comment in > > https://github.com/apache/bookkeeper/pull/197#issuecomment-31783179 > > 9 > > > > > > > > > > > > > > , > > > > > > if > > > > > > > > > > we > > use `docker push` approach and let jenkins build and push docker > > > > > > > > > > images, > > > > > > it > > > > > > seems to be much easier > > to address the above issues. > > > > Thoughts? > > > > - Sijie > > > > > > > > > > > > > > > > -- > > > > > > -- Enrico Olivelli > > > > > > > > -- > > > > > > -- Enrico Olivelli > > > > > > > > -- > > > > Francesco Caliumi > > Developer @ Diennea - MagNews > > Tel.: (+39) 0546 066100 - Int. 266 > > Viale G.Marconi 30/14 - 48018 Faenza (RA) > > > > [Magnews.it]<http://www.magnews.it/it> > > > > [Linkedin]<http://www.linkedin.com/company/diennea---magnews> > > [Twitter] > > <http://twitter.com/DienneaMagNews> [Facebook] < > > http://www.facebook.com/pages/MagNews/197617841797>; [Newslett > > er] < > > http://www.magnews.it/it/iscriviti-alla-newsletter>; > > > > ________________________________ > > > > Iscriviti alla nostra newsletter per rimanere aggiornato su digital > > ed > > email marketing! http://www.magnews.it/newsletter/ > > > > The information in this email is confidential and may be legally > > privileged. If you are not the intended recipient please notify the > > sender > > immediately and destroy this email. Any unauthorized, direct or > > indirect, > > disclosure, copying, storage, distribution or other use is strictly > > forbidden. > > > > > > > > -- > > > > Francesco Caliumi > > Developer @ Diennea - MagNews > > Tel.: (+39) 0546 066100 - Int. 266 > > Viale G.Marconi 30/14 - 48018 Faenza (RA) > > > > [Magnews.it]<http://www.magnews.it/it> > > > > [Linkedin]<http://www.linkedin.com/company/diennea---magnews> > > [Twitter] <http://twitter.com/DienneaMagNews> [Facebook] < > > http://www.facebook.com/pages/MagNews/197617841797>; [Newslett > > er] < > > http://www.magnews.it/it/iscriviti-alla-newsletter>; > > > > ________________________________ > > > > Iscriviti alla nostra newsletter per rimanere aggiornato su digital > > ed > > email marketing! http://www.magnews.it/newsletter/ > > > > The information in this email is confidential and may be legally > > privileged. If you are not the intended recipient please notify the > > sender > > immediately and destroy this email. Any unauthorized, direct or > > indirect, > > disclosure, copying, storage, distribution or other use is strictly > > forbidden. > > > > > > > > -- > > > > Francesco Caliumi > > Developer @ Diennea - MagNews > > Tel.: (+39) 0546 066100 - Int. 266 > > Viale G.Marconi 30/14 - 48018 Faenza (RA) > > > > [Magnews.it]<http://www.magnews.it/it> > > > > [Linkedin]<http://www.linkedin.com/company/diennea---magnews> > > [Twitter] > > <http://twitter.com/DienneaMagNews> [Facebook] < > > http://www.facebook.com/pages/MagNews/197617841797>; [Newslett > > er] < > > http://www.magnews.it/it/iscriviti-alla-newsletter>; > > > > ________________________________ > > > > Iscriviti alla nostra newsletter per rimanere aggiornato su digital > > ed > > email marketing! http://www.magnews.it/newsletter/ > > > > The information in this email is confidential and may be legally > > privileged. If you are not the intended recipient please notify the > > sender > > immediately and destroy this email. Any unauthorized, direct or > > indirect, > > disclosure, copying, storage, distribution or other use is strictly > > forbidden. > > > > > > -- > > > > Francesco Caliumi > > Developer @ Diennea - MagNews > > Tel.: (+39) 0546 066100 - Int. 266 > > Viale G.Marconi 30/14 - 48018 Faenza (RA) > > > > [Magnews.it]<http://www.magnews.it/it> > > > > [Linkedin]<http://www.linkedin.com/company/diennea---magnews> > > [Twitter] <http://twitter.com/DienneaMagNews> [Facebook] < > > http://www.facebook.com/pages/MagNews/197617841797>; [Newslett > > er] < > > http://www.magnews.it/it/iscriviti-alla-newsletter>; > > > > ________________________________ > > > > Iscriviti alla nostra newsletter per rimanere aggiornato su digital > > ed > > email marketing! http://www.magnews.it/newsletter/ > > > > The information in this email is confidential and may be legally > > privileged. If you are not the intended recipient please notify the > > sender > > immediately and destroy this email. Any unauthorized, direct or > > indirect, > > disclosure, copying, storage, distribution or other use is strictly > > forbidden. > > > > > > > > -- > > > > Francesco Caliumi > > Developer @ Diennea - MagNews > > Tel.: (+39) 0546 066100 - Int. 266 > > Viale G.Marconi 30/14 - 48018 Faenza (RA) > > > > [Magnews.it]<http://www.magnews.it/it> > > > > [Linkedin]<http://www.linkedin.com/company/diennea---magnews> > > [Twitter] <http://twitter.com/DienneaMagNews> [Facebook] < > > http://www.facebook.com/pages/MagNews/197617841797>; [Newslett > > er] < > > http://www.magnews.it/it/iscriviti-alla-newsletter>; > > > > ________________________________ > > > > Iscriviti alla nostra newsletter per rimanere aggiornato su digital > > ed > > email marketing! http://www.magnews.it/newsletter/ > > > > The information in this email is confidential and may be legally > > privileged. If you are not the intended recipient please notify the > > sender > > immediately and destroy this email. Any unauthorized, direct or > > indirect, > > disclosure, copying, storage, distribution or other use is strictly > > forbidden. > > > > -- Francesco Caliumi Developer @ Diennea - MagNews Tel.: (+39) 0546 066100 - Int. 266 Viale G.Marconi 30/14 - 48018 Faenza (RA) ________________________________ Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email marketing! http://www.magnews.it/newsletter/ The information in this email is confidential and may be legally privileged. If you are not the intended recipient please notify the sender immediately and destroy this email. Any unauthorized, direct or indirect, disclosure, copying, storage, distribution or other use is strictly forbidden.