hmm, I am not sure. I changed the permissions to admin for you and Jia. Let me know if you have permissions now?
- Sijie On Thu, Aug 31, 2017 at 11:45 PM, Francesco Caliumi - Diennea < francesco.cali...@diennea.com> wrote: > Hi Sijie, for sure. > > I see apachebookkeeper in my organizations, but I can't modify anything. > Maybe my user lacks write access? > > On Fri, 2017-09-01 at 14:11 +0800, Jia Zhai wrote: > > sure , I would like to. my docker account is: zhaijia > > On Fri, Sep 1, 2017 at 1:45 PM, Sijie Guo <guosi...@gmail.com<mailto:guo > si...@gmail.com>> wrote: > > > > Hi Jia, Francesco, > > Regarding nightly build, let's put this aside for now. > > Let's try to make an official docker image. I've created an > `apachebookkeeper` organization - https://hub.docker.com/r/ > apachebookkeeper/bookkeeper/~/settings/collaborators/ > > I've added you guys to the team. Can you guys help making it the place for > hosting official docker image? > > Other committers/contributors, if you are willing to help with this, let me > know I will add you to the list as well. > > - Sijie > > On Mon, Aug 28, 2017 at 3:00 AM, Francesco Caliumi - Diennea < > francesco.cali...@diennea.com<mailto:francesco.cali...@diennea.com>> > wrote: > > > > On Mon, 2017-08-28 at 02:19 -0700, Sijie Guo wrote: > > > On Mon, Aug 28, 2017 at 2:04 AM, Francesco Caliumi - Diennea < > francesco.cali...@diennea.com<mailto:francesco.cali...@diennea.com>> > wrote: > > > > Hi Sijie, here the reply: > > > > > > > You only build the docker image locally, right? I am not sure > how can > > > > > > you > > > > > distribute the docker image in a multiple node or a cloud > environment. > > > > > > > Yes, the scenario I described is only for lacal tests. If you want > to > distribute the docker image you can push it to docker hub (maybe on > an ad > hoc user for the purpose, like for example "bookkeeper-tests") > after build, > but I've never personally set up this kind of automation though. > > However, locally you can run a whole ensemble. What is the use case > you > have in mind for a cloud environment? Benchmarks? > > > > > > We are trying to setup a CI/CD pipeline for bookkeeper. The ideal > pipeline > would : unit test => build a docker image => integration test => > jepsen > test => benchmark. After unit tests, it is good to deploy the > bookkeeper > (using the built docker image) to a small cluster (either on cloud or > another other environment), run end-to-end integration test (maybe > also > backward compatibility testsing), failure (jepsen) tests and then > benchmark. > > > > Thanks for the explanation, it's now clearer what will happen. > > > > You need a nightly image for this purpose. > > > > A docker hub image is indeed way more easier to manage for bring up > a cluster. If we manage to "docker push" the image after the build then > the complexity will be the same cluster side. > > One thing to consider: build from docker hub can be triggered, but I > don't know if it is possible to know when it finish. If it isn't, tests > should be run with a certain delay after the trigger. Docker push > approach doesn't have this inconvenient. > > > > A more ideal situation for me - > for release image, we should only pick a image that pass all the > CI/CD > pipeline and tag it as release. The current `download-package` > approach to > build release docker images doesn't sound promising to me. > > > > > I agree that the current `download-package` approach is very rigid > for testing. The proposal was actually to create a maven target that > picks up the just built tar package and load it directly in the > workspace of the image, removing the need to download it from a > repository (Enrico already played with it in HerdDB project and it > worked fine). > > The Dockerfile of this build will be simpler then the release one, so > it shouldn't take much effort to create it. I think flexibility wins in > test scenarios, but is up to you. > > > > > > > > > On Mon, 2017-08-28 at 01:05 -0700, Sijie Guo wrote: > > Hi Francesco, > > Sorry. I can't read what are your reply in this email. It seems the > format > was lost ... > > - Sijie > > On Mon, Aug 28, 2017 at 12:32 AM, Francesco Caliumi - Diennea < > francesco.cali...@diennea.com<mailto:francesco.cali...@diennea.com > ><mailto:francesco.cali...@diennea.com> > > > > > > wrote: > > > > On Fri, 2017-08-25 at 08:02 -0700, Sijie Guo wrote: > > On Aug 25, 2017 7:33 AM, "Francesco Caliumi - Diennea" < > francesco.cali...@diennea.com<mailto:francesco.cali...@diennea.com > ><mailto:francesco.cali...@diennea.com > > > <mailto:francesco.cali...@diennea.com>> > > > > wrote: > > On Thu, 2017-08-24 at 11:22 -0700, Sijie Guo wrote: > > On Tue, Aug 22, 2017 at 12:30 AM, Francesco Caliumi - Diennea < > francesco.cali...@diennea.com<mailto:francesco.cali...@diennea.com > ><mailto:francesco.cali...@diennea.com > > > <mailto:francesco.cali...@diennea.com > > > > > <mailto:francesco.cali...@diennea.com>> wrote: > > > > > > I'm not sure if I explained well my point. No docker hub involved, > just a > tool for every developer who wants to build an image by its own. > Maybe to > test the stability of a feature branch instead of master, for > instance. > > > > > If we don't push to docker hub, how can this image be used for > continuous > testing. The idea is to have a nightly image, so this image can be > deployed > to somewhere to do integration test, jepsen test and other things. > > > > I'm not familiar with this use case, but I think the image can be > builded > before starting integration tests, without any need of docker hub > at all > (but it adds a couple of minutes to overall test time). > > This way integration tests could be executed on master and on > branches as > well. > > > You only build the docker image locally, right? I am not sure how > can you > distribute the docker image in a multiple node or a cloud > environment. > > > > Yes, the scenario I described is only for lacal tests. If you want > to > distribute the docker image you can push it to docker hub (maybe on > an ad > hoc user for the purpose, like for example "bookkeeper-tests") > after build, > but I've never personally set up this kind of automation though. > > However, locally you can run a whole ensemble. What is the use case > you > have in mind for a cloud environment? Benchmarks? > > > > > > > Maybe it could help to well define the use case scenario and the > target > user. If target user is a developer or tester that needs to test a > bk > ensemble, having available only the one day delayed build of the > master > branch could be limiting. > > > > > > > > > On Tue, 2017-08-22 at 14:38 +0800, Jia Zhai wrote: > > Hi Sijie, > For releases version, we have no permissions to do "docker push" > images > into wanted place(neither under apache <https://hub.docker.com/u/ap > ache/> > nor "Official <https://docs.docker.com/docker-hub/official_repos/> > "). > > Thanks. > -Jia > > > > On Tue, Aug 22, 2017 at 12:55 PM, Sijie Guo <guosi...@gmail.com<mailto:guo > si...@gmail.com><mai > lto:guo > si...@gmail.com<mailto:si...@gmail.com>><mailto:guo > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com>><mailto: > guo > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com><mailto: > si...@gmail.com>><ma > ilto: > guo > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com><mailto: > si...@gmail.com><mai > lto: > si...@gmail.com<mailto:si...@gmail.com>>>> wrote: > > > > On Aug 21, 2017 12:56 AM, "Francesco Caliumi - Diennea" < > francesco.cali...@diennea.com<mailto:francesco.cali...@diennea.com > ><mailto:francesco.cali...@diennea.com > > > <mailto:francesco.cali...@diennea.com > > > > > <mailto:francesco.cali...@diennea.com><mailto: > > > francesco.cali...@diennea.com<mailto:francesco.cali...@diennea.com > ><mailto:francesco.cali...@diennea.com > > > <mailto:francesco.cali...@diennea.com>>> > > > > wrote: > > My view for these topics: > > 1) I'm a big fan of official docker images, so it's ok for me. > > 2) For latest build, both Jia and Enrico solutions are ok, but I > think > Enrico's maven target one is the most flexible. > Who usually need the nightly build are developers, who do not care > very > much to have an official or almost-official image, but they like > the > flexibility to build the image of the commit / branch they want to > test. > > > Having a maven target should create a workflow like this: > > - do some work or git checkout {<branch> / <commit>} > - mvn clean package docker:build > - (optional: only if working with multiple versions) docker tag > <generated_image> <dev_docker_user>/bookkeeper:mytag > - (optional: only if the image should be accessibile by others devs > / > machines) docker push <dev_docker_user>/bookkeeper:mytag > - test the image > > On functionality side, this image will be slightly different from > the > current one, because it will not need to download the package but > it will > find it directly in the build dir. > > > Personally I'd like to have a single way to build images for > nightly and > releases. It is going to be hard to maintain two solutions in long > term. > > > On Tue, 2017-08-15 at 00:49 -0700, Sijie Guo wrote: > > On Tue, Aug 15, 2017 at 12:38 AM, Enrico Olivelli <eolivelli@gmail. > com > <mailto:eolive...@gmail.com> > <mailto:eolive...@gmail.com> > <mailto:eolive...@gmail.com> > <mailto:eolive...@gmail.com> > <mailto:eolive...@gmail.com>> > wrote: > > > > Il mar 15 ago 2017, 09:16 Sijie Guo <guosi...@gmail.com<mailto:guo > si...@gmail.com><mailto:guo > si...@gmail.com<mailto:si...@gmail.com>><mailto:guo > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com>><mailto: > guo > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com><mailto: > si...@gmail.com>><ma > ilto: > guo > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com><mailto: > si...@gmail.com><mai > lto: > si...@gmail.com<mailto:si...@gmail.com>>><mailto: > guo > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com><mailto: > si...@gmail.com><mai > lto: > si...@gmail.com<mailto:si...@gmail.com>><mailto: > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com>>>> ha > scritto: > > > > On Aug 14, 2017 11:18 PM, "Enrico Olivelli" <eolive...@gmail.com<mailto:eo > live...@gmail.com><ma > ilto:eo > live...@gmail.com<mailto:live...@gmail.com>><mailto:eo > live...@gmail.com<mailto:live...@gmail.com><mailto:live...@gmail.com > >><mailto:eo > live...@gmail.com<mailto:live...@gmail.com><mailto:live...@gmail.com > ><mailto:live...@gmail.co > m > > > > > <mailto:eo > > > > > > live...@gmail.com<mailto:live...@gmail.com><mailto:live...@gmail.com > ><mailto:live...@gmail.co > m > > > <mailto:live...@gmail.com > > > > > > > <mailto:eo > > > > > live...@gmail.com<mailto:live...@gmail.com><mailto:live...@gmail.com > ><mailto:live...@gmail.co > m > > > <mailto:live...@gmail.com > > > > > <mailto:live...@gmail.com>>> > > > wrote: > > Il mar 15 ago 2017, 03:29 Jia Zhai <zhaiji...@gmail.com<mailto:zh > aiji...@gmail.com><mailto:zh > aiji...@gmail.com<mailto:aiji...@gmail.com>><mailto:zh > aiji...@gmail.com<mailto:aiji...@gmail.com><mailto:aiji...@gmail.com > >><mailto:zh > aiji...@gmail.com<mailto:aiji...@gmail.com><mailto:aiji...@gmail.com > ><mailto:aiji...@gmail.co > m > > > > > <mailto:zh > > > > > > aiji...@gmail.com<mailto:aiji...@gmail.com><mailto:aiji...@gmail.com > ><mailto:aiji...@gmail.co > m > > > <mailto:aiji...@gmail.com > > > > > > > <mailto:zh > > > > > aiji...@gmail.com<mailto:aiji...@gmail.com><mailto:aiji...@gmail.com > ><mailto:aiji...@gmail.co > m > > > <mailto:aiji...@gmail.com > > > > > <mailto:aiji...@gmail.com>>> ha > > > scritto: > > > > Hi Sijie, > From my view, the approach problem is whether we have permissions > to > "docker push" images into wanted place. > Following the way putting images under "apache", we seems not > have > permission, While following the "Official > <https://docs.docker.com/docker-hub/official_repos/>" way, seems > > > neither. > > > > Since nightly build is mainly for our development, It maybe OK to > > > > > manage > > > > > and maintain a dockerhub account by our community to hold the > nightly > one(maybe also for the release images). > > > > On Tue, Aug 15, 2017 at 8:19 AM, Sijie Guo <guosi...@gmail.com<mailto:guo > si...@gmail.com><mail > to:guo > si...@gmail.com<mailto:si...@gmail.com>><mailto:guo > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com>><mailto: > guo > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com><mailto: > si...@gmail.com>><ma > ilto: > guo > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com><mailto: > si...@gmail.com><mai > lto: > si...@gmail.com<mailto:si...@gmail.com>>><mailto: > guo > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com><mailto: > si...@gmail.com><mai > lto: > si...@gmail.com<mailto:si...@gmail.com>><mailto: > si...@gmail.com<mailto:si...@gmail.com><mailto:si...@gmail.com>>>> wrote: > > > > On Mon, Aug 14, 2017 at 3:47 AM, Jia Zhai <zhaiji...@gmail.com<mailto:zh > aiji...@gmail.com><mail > to:zh > aiji...@gmail.com<mailto:aiji...@gmail.com>><mailto:zh > aiji...@gmail.com<mailto:aiji...@gmail.com><mailto:aiji...@gmail.com > >><mailto:zh > aiji...@gmail.com<mailto:aiji...@gmail.com><mailto:aiji...@gmail.com > ><mailto:aiji...@gmail.co > m > > > > > <mailto:zh > > > > > > aiji...@gmail.com<mailto:aiji...@gmail.com><mailto:aiji...@gmail.com > ><mailto:aiji...@gmail.co > m > > > <mailto:aiji...@gmail.com > > > > > > > <mailto:zh > > > > > aiji...@gmail.com<mailto:aiji...@gmail.com><mailto:aiji...@gmail.com > ><mailto:aiji...@gmail.co > m > > > <mailto:aiji...@gmail.com > > > > > <mailto:aiji...@gmail.com>>> > > > > > > > > > wrote: > > > > > > > > > > Thanks Sijie for raising these good topics up. > > Regarding 1) official image, the Flink one is similar as Zookeeper > > > > > > > one, > > > > > > > which we have discussed before. If the currently approach could not > > > > > make > > > > > bookkeeper docker image official, we should go this way. > > > > > > > Regarding 2) nightly build, there is already an issue > <https://github.com/apache/bookkeeper/issues/289> opened. Seems > > > > > > > > > the > > > > > > > issue > > > is where to put the nightly build images, since for both way of > > > > > official > > > > > image, there is limited access to the dockerhub, The first thought > > > > > > > > > in > > > > > my > > > > > head is to place a nightly build somewhere, such as ( > https://dist.apache.org/repos/dist/dev/bookkeeper/), and the > > > > > > > > > current > > > > > > > > > docker > file will not changed too much, seems only some env var need > > > > > > > > > change. > > > > > > > > > > > > > currently we don't have any process to produce any nightly built > > > packages. > > > > If we are planning to use dist/dev for hosting the nightly build, > we > > > > > need > > > > > to figure out how to get the credentials to do that. > Because the dist/dev is a svn repo, during a release, the release > > > > > manager > > > > > uses its own credentials to commit the new packages to the svn > repo. > > > > > > Regarding 3), the build failure > <https://hub.docker.com/r/apache/bookkeeper/builds/ > bvzft3fsnpmmj5i8jnpk3fl/> > is caused by connection(from dockerhub to gpg key server) issue. It > > > > > > > is > > > > > also > > > one reason to come out PR420 <https://github.com/apache/ > bookkeeper/pull/420>, > which wanted to download local KEY to avoid gpg server connection, > > > > > > > But > > > we > > > > > agree that it is not very security. > > > > > I think there are a few drawbacks that I can see in current > approach: > > - building the packages and building the docker images are managed > by > > > > > two > > > > > different systems. > - the `latest` image isn't really a tag pointing to the image of > > > > > > > latest > > > > > > > release; the latest is actually building from master. any changes > > > > > pushed > > > to > > > master unnecessarily trigger auto build in docker hub, even the > > > > > > > package > > > > > > > itself isn't changed. > > > > > "latest" is some kind of place holder, I thought to use it to point > to > > > the > > > nightly version, seems I was mis-understanding of it. > > > > > My two cents > IMHO latest should be v the latest stable version > > > Yes that's my thought as well. Then it should be just a tag for the > image > of latest release. However in current approach, it keeps > regenerating a > image everytime there is a change on master. This doesn't make any > sense > > > to > > > me. > > > > > We have to touch the latest tag only as part of the release > process. > > > > > Currently latest is not just a tag of 4.5.0. they are separate > images > generated by docker auto builds. > > > > > > > > > > > One step back, what is the purpose of having this image? I mean an > image > > > of > > > the master branch. > > > It can be used in a CD (continuous deployment) pipeline, you can > deploy > > > the > > > image to a dockerized environment to verify if it is working or > not. > > > > > Yup, so it is like the maven snapshots. > From the licensing point in that case all the constraints are > relaxed, we > can really use a PMC managed repository > > > > > > If you only need to have a working bookkeeper server for docker we > can > > > use > > > the maven docker plugin which is well integreated with maven based > environments. The problem of where to put is is still here. > Sometime ago > for a project I had just created a free account on docker hub and > pushed > there the image directly from the build. > Jia already created a BookKeeper account, we could use it. > Credentials will be managed by PMC > > > Enrico > > > > > > > > - building a nightly package is not trivial. > > If we step back and revisit my comment in > https://github.com/apache/bookkeeper/pull/197#issuecomment-31783179 > 9 > > > > > > > , > > > if > > > > > we > use `docker push` approach and let jenkins build and push docker > > > > > images, > > > it > > > seems to be much easier > to address the above issues. > > Thoughts? > > - Sijie > > > > > > > > -- > > > -- Enrico Olivelli > > > > -- > > > -- Enrico Olivelli > > > > -- > > Francesco Caliumi > Developer @ Diennea - MagNews > Tel.: (+39) 0546 066100 - Int. 266 > Viale G.Marconi 30/14 - 48018 Faenza (RA) > > [Magnews.it]<http://www.magnews.it/it> > > [Linkedin]<http://www.linkedin.com/company/diennea---magnews> > [Twitter] > <http://twitter.com/DienneaMagNews> [Facebook] < > http://www.facebook.com/pages/MagNews/197617841797>; [Newslett > er] < > http://www.magnews.it/it/iscriviti-alla-newsletter>; > > ________________________________ > > Iscriviti alla nostra newsletter per rimanere aggiornato su digital > ed > email marketing! http://www.magnews.it/newsletter/ > > The information in this email is confidential and may be legally > privileged. If you are not the intended recipient please notify the > sender > immediately and destroy this email. Any unauthorized, direct or > indirect, > disclosure, copying, storage, distribution or other use is strictly > forbidden. > > > > -- > > Francesco Caliumi > Developer @ Diennea - MagNews > Tel.: (+39) 0546 066100 - Int. 266 > Viale G.Marconi 30/14 - 48018 Faenza (RA) > > [Magnews.it]<http://www.magnews.it/it> > > [Linkedin]<http://www.linkedin.com/company/diennea---magnews> > [Twitter] <http://twitter.com/DienneaMagNews> [Facebook] < > http://www.facebook.com/pages/MagNews/197617841797>; [Newslett > er] < > http://www.magnews.it/it/iscriviti-alla-newsletter>; > > ________________________________ > > Iscriviti alla nostra newsletter per rimanere aggiornato su digital > ed > email marketing! http://www.magnews.it/newsletter/ > > The information in this email is confidential and may be legally > privileged. If you are not the intended recipient please notify the > sender > immediately and destroy this email. Any unauthorized, direct or > indirect, > disclosure, copying, storage, distribution or other use is strictly > forbidden. > > > > -- > > Francesco Caliumi > Developer @ Diennea - MagNews > Tel.: (+39) 0546 066100 - Int. 266 > Viale G.Marconi 30/14 - 48018 Faenza (RA) > > [Magnews.it]<http://www.magnews.it/it> > > [Linkedin]<http://www.linkedin.com/company/diennea---magnews> > [Twitter] > <http://twitter.com/DienneaMagNews> [Facebook] < > http://www.facebook.com/pages/MagNews/197617841797>; [Newslett > er] < > http://www.magnews.it/it/iscriviti-alla-newsletter>; > > ________________________________ > > Iscriviti alla nostra newsletter per rimanere aggiornato su digital > ed > email marketing! http://www.magnews.it/newsletter/ > > The information in this email is confidential and may be legally > privileged. If you are not the intended recipient please notify the > sender > immediately and destroy this email. Any unauthorized, direct or > indirect, > disclosure, copying, storage, distribution or other use is strictly > forbidden. > > > -- > > Francesco Caliumi > Developer @ Diennea - MagNews > Tel.: (+39) 0546 066100 - Int. 266 > Viale G.Marconi 30/14 - 48018 Faenza (RA) > > [Magnews.it]<http://www.magnews.it/it> > > [Linkedin]<http://www.linkedin.com/company/diennea---magnews> > [Twitter] <http://twitter.com/DienneaMagNews> [Facebook] < > http://www.facebook.com/pages/MagNews/197617841797>; [Newslett > er] < > http://www.magnews.it/it/iscriviti-alla-newsletter>; > > ________________________________ > > Iscriviti alla nostra newsletter per rimanere aggiornato su digital > ed > email marketing! http://www.magnews.it/newsletter/ > > The information in this email is confidential and may be legally > privileged. If you are not the intended recipient please notify the > sender > immediately and destroy this email. Any unauthorized, direct or > indirect, > disclosure, copying, storage, distribution or other use is strictly > forbidden. > > > > -- > > Francesco Caliumi > Developer @ Diennea - MagNews > Tel.: (+39) 0546 066100 - Int. 266 > Viale G.Marconi 30/14 - 48018 Faenza (RA) > > [Magnews.it]<http://www.magnews.it/it> > > [Linkedin]<http://www.linkedin.com/company/diennea---magnews> > [Twitter] <http://twitter.com/DienneaMagNews> [Facebook] < > http://www.facebook.com/pages/MagNews/197617841797>; [Newslett > er] < > http://www.magnews.it/it/iscriviti-alla-newsletter>; > > ________________________________ > > Iscriviti alla nostra newsletter per rimanere aggiornato su digital > ed > email marketing! http://www.magnews.it/newsletter/ > > The information in this email is confidential and may be legally > privileged. If you are not the intended recipient please notify the > sender > immediately and destroy this email. Any unauthorized, direct or > indirect, > disclosure, copying, storage, distribution or other use is strictly > forbidden. > > > > > > > > -- > Francesco Caliumi > Developer @ Diennea - MagNews > Tel.: (+39) 0546 066100 - Int. 266 > Viale G.Marconi 30/14 - 48018 Faenza (RA) > > > > ________________________________ > > Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed > email marketing! http://www.magnews.it/newsletter/ > > The information in this email is confidential and may be legally > privileged. If you are not the intended recipient please notify the > > > sender > > > immediately and destroy this email. Any unauthorized, direct or indirect, > disclosure, copying, storage, distribution or other use is strictly > forbidden. > > > > > > > -- > > Francesco Caliumi > Developer @ Diennea - MagNews > Tel.: (+39) 0546 066100 - Int. 266 > Viale G.Marconi 30/14 - 48018 Faenza (RA) > > [Magnews.it]<http://www.magnews.it/it> > > [Linkedin]<http://www.linkedin.com/company/diennea---magnews> > [Twitter] <http://twitter.com/DienneaMagNews> [Facebook] < > http://www.facebook.com/pages/MagNews/197617841797> [Newsletter] < > http://www.magnews.it/it/iscriviti-alla-newsletter> > > ________________________________ > > Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed > email marketing! http://www.magnews.it/newsletter/ > > The information in this email is confidential and may be legally > privileged. If you are not the intended recipient please notify the sender > immediately and destroy this email. Any unauthorized, direct or indirect, > disclosure, copying, storage, distribution or other use is strictly > forbidden. >
