just elaborate a bit more on why slow build is ok but no resource is not: Say I submit a build request at PST 9am, no other requests exist and mine is the queue head, currently it means it still cannot get built until 4 or 5pm.
> On Jun 26, 2019, at 12:28, Bowen Li <bowenl...@gmail.com> wrote: > > Hi, > > @Dawid, I think the "long test running" as I mentioned in the first email, > also as you guys said, belongs to "a big effort which is much harder to > accomplish in a short period of time and may deserve its own separate > discussion". Thus I didn't include it in what we can do in a foreseeable > short term. > > Besides, I don't think that's the ultimate reason for lack of build > resources. Even if the build is shortened to something like 2h, the problems > of no build machine works about 6 or more hours in PST daytime that I > described will still happen, because no machine from ASF INFRA's pool is > allocated to Flink. As I have paid close attention to the build queue in the > past few weekdays, it's a pretty clear pattern now. > > **The ultimate root cause** for that is - we don't have any **dedicated** > build resources that we can stably rely on. I'm actually ok to wait for a > long time if there are build requests running, it means at least we are > making progress. But I'm not ok with no build resource. A better place I > think we should aim at in short term is to always have at least a central > pool (can be 3 or 5) of machines dedicated to build Flink at any time, or > maybe use users resources. > > @Chesnay @Robert I synced with Jeff offline that Zeppelin community is using > a Jenkins job to automatically build on users' travis account and link the > result back to github PR. I guess the Jenkins job would fetch latest upstream > master and build the PR against it. Jeff has filed tickets to learn and get > access to the Jenkins infra. It'll better to fully understand it first before > judging this approach. > > I also heard good things about CircleCI, and ASF INFRA seems to have a pool > of build capacity there too. Can be an alternative to consider. > > > > > > > > > >> On Wed, Jun 26, 2019 at 12:44 AM Dawid Wysakowicz <dwysakow...@apache.org> >> wrote: >> Sorry to jump in late, but I think Bowen missed the most important point >> from Chesnay's previous message in the summary. The ultimate reason for >> all the problems is that the tests take close to 2 hours to run already. >> I fully support this claim: "Unless people start caring about test times >> before adding them, this issue cannot be solved" >> >> This is also another reason why using user's Travis account won't help. >> Every few weeks we reach the user's time limit for a single profile. >> This makes the user's builds simply fail, until we either properly >> decrease the time the tests take (which I am not sure we ever did) or >> postpone the problem by splitting into more profiles. (Note that the ASF >> Travis account has higher time limits) >> >> Best, >> >> Dawid >> >> On 26/06/2019 09:36, Robert Metzger wrote: >> > Do we know if using "the best" available hardware would improve the build >> > times? >> > Imagine we would run the build on machines with plenty of main memory to >> > mount everything to ramdisk + the latest CPU architecture? >> > >> > Throwing hardware at the problem could help reduce the time of an >> > individual build, and using our own infrastructure would remove our >> > dependency on Apache's Travis account (with the obvious downside of having >> > to maintain the infrastructure) >> > We could use an open source travis alternative, to have a similar >> > experience and make the migration easy. >> > >> > >> > On Wed, Jun 26, 2019 at 9:34 AM Chesnay Schepler <ches...@apache.org> >> > wrote: >> > >> >> From what I gathered, there's no special sauce that the Zeppelin >> >> project uses which actually integrates a users Travis account into the PR. >> >> >> >> They just disabled Travis for PRs. And that's kind of it. >> >> >> >> Naturally we can do this (duh) and safe the ASF a fair amount of >> >> resources, but there are downsides: >> >> >> >> The discoverability of the Travis check takes a nose-dive. Either we >> >> require every contributor to always, an every commit, also post a Travis >> >> build, or we have the reviewer sift through the contributors account to >> >> find it. >> >> >> >> This is rather cumbersome. Additionally, it's also not equivalent to >> >> having a PR build. >> >> >> >> A normal branch build takes a branch as is and tests it. A PR build >> >> merges the branch into master, and then runs it. (Fun fact: This is why >> >> a PR without merge conflicts is not being run on Travis.) >> >> >> >> And ultimately, everyone can already make use of this approach anyway. >> >> >> >> On 25/06/2019 08:02, Jark Wu wrote: >> >>> Hi Jeff, >> >>> >> >>> Thanks for sharing the Zeppelin approach. I think it's a good idea to >> >>> leverage user's travis account. >> >>> In this way, we can have almost unlimited concurrent build jobs and >> >>> developers can restart build by themselves (currently only committers >> >>> can restart PR's build). >> >>> >> >>> But I'm still not very clear how to integrate user's travis build into >> >>> the Flink pull request's build automatically. Can you explain more in >> >>> detail? >> >>> >> >>> Another question: does travis only build branches for user account? >> >>> My concern is that builds for PRs will rebase user's commits against >> >>> current master branch. >> >>> This will help us to find problems before merge. Builds for branches >> >>> will lose the impact of new commits in master. >> >>> How does Zeppelin solve this problem? >> >>> >> >>> Thanks again for sharing the idea. >> >>> >> >>> Regards, >> >>> Jark >> >>> >> >>> On Tue, 25 Jun 2019 at 11:01, Jeff Zhang <zjf...@gmail.com >> >>> <mailto:zjf...@gmail.com>> wrote: >> >>> >> >>> Hi Folks, >> >>> >> >>> Zeppelin meet this kind of issue before, we solve it by delegating >> >>> each >> >>> one's PR build to his travis account (Everyone can have 5 free >> >>> slot for >> >>> travis build). >> >>> Apache account travis build is only triggered when PR is merged. >> >>> >> >>> >> >>> >> >>> Kurt Young <ykt...@gmail.com <mailto:ykt...@gmail.com>> >> >>> 于2019年6月25日周二 上午10:16写道： >> >>> >> >>> > (Forgot to cc George) >> >>> > >> >>> > Best, >> >>> > Kurt >> >>> > >> >>> > >> >>> > On Tue, Jun 25, 2019 at 10:16 AM Kurt Young <ykt...@gmail.com >> >>> <mailto:ykt...@gmail.com>> wrote: >> >>> > >> >>> > > Hi Bowen, >> >>> > > >> >>> > > Thanks for bringing this up. We actually have discussed about >> >>> this, and I >> >>> > > think Till and George have >> >>> > > already spend sometime investigating it. I have cced both of >> >>> them, and >> >>> > > maybe they can share >> >>> > > their findings. >> >>> > > >> >>> > > Best, >> >>> > > Kurt >> >>> > > >> >>> > > >> >>> > > On Tue, Jun 25, 2019 at 10:08 AM Jark Wu <imj...@gmail.com >> >>> <mailto:imj...@gmail.com>> wrote: >> >>> > > >> >>> > >> Hi Bowen, >> >>> > >> >> >>> > >> Thanks for bringing this. We also suffered from the long >> >>> build time. >> >>> > >> I agree that we should focus on solving build capacity >> >>> problem in the >> >>> > >> thread. >> >>> > >> >> >>> > >> My observation is there is only one build is running, all the >> >>> others >> >>> > >> (other >> >>> > >> PRs, master) are pending. >> >>> > >> The pricing plan[1] of travis shows it can support concurrent >> >>> build >> >>> > jobs. >> >>> > >> But I don't know which plan we are using, might be the free >> >>> plan for >> >>> > open >> >>> > >> source. >> >>> > >> >> >>> > >> I cc-ed Chesnay who may have some experience on Travis. >> >>> > >> >> >>> > >> Regards, >> >>> > >> Jark >> >>> > >> >> >>> > >> [1]: https://travis-ci.com/plans >> >>> > >> >> >>> > >> On Tue, 25 Jun 2019 at 08:11, Bowen Li <bowenl...@gmail.com >> >>> <mailto:bowenl...@gmail.com>> wrote: >> >>> > >> >> >>> > >> > Hi Steven, >> >>> > >> > >> >>> > >> > I think you may not read what I wrote. The discussion is about >> >>> > "unstable >> >>> > >> > build **capacity**", in another word "unstable / lack of build >> >>> > >> resources", >> >>> > >> > not "unstable build". >> >>> > >> > >> >>> > >> > On Mon, Jun 24, 2019 at 4:40 PM Steven Wu >> >>> <stevenz...@gmail.com <mailto:stevenz...@gmail.com>> >> >>> > wrote: >> >>> > >> > >> >>> > >> > > long and sometimes unstable build is definitely a pain >> >> point. >> >>> > >> > > >> >>> > >> > > I suspect the build failure here in flink-connector-kafka >> >>> is not >> >>> > >> related >> >>> > >> > to >> >>> > >> > > my change. but there is no easy re-run the build on >> >>> travis UI. >> >>> > Google >> >>> > >> > > search showed a trick of close-and-open the PR will >> >>> trigger rebuild. >> >>> > >> but >> >>> > >> > > that could add noises to the PR activities. >> >>> > >> > > https://travis-ci.org/apache/flink/jobs/545555519 >> >>> > >> > > >> >>> > >> > > travis-ci for my personal repo often failed with >> >>> exceeding time >> >>> > limit >> >>> > >> > after >> >>> > >> > > 4+ hours. >> >>> > >> > > The job exceeded the maximum time limit for jobs, and has >> >>> been >> >>> > >> > terminated. >> >>> > >> > > >> >>> > >> > > On Mon, Jun 24, 2019 at 4:15 PM Bowen Li >> >>> <bowenl...@gmail.com <mailto:bowenl...@gmail.com>> >> >>> > wrote: >> >>> > >> > > >> >>> > >> > > > https://travis-ci.org/apache/flink/builds/549681530 >> >>> This build >> >>> > >> > request >> >>> > >> > > > has >> >>> > >> > > > been sitting at **HEAD of the queue** since I first saw >> >>> it at PST >> >>> > >> > 10:30am >> >>> > >> > > > (not sure how long it's been there before 10:30am). >> >>> It's PST >> >>> > 4:12pm >> >>> > >> now >> >>> > >> > > and >> >>> > >> > > > it hasn't started yet. >> >>> > >> > > > >> >>> > >> > > > On Mon, Jun 24, 2019 at 2:48 PM Bowen Li >> >>> <bowenl...@gmail.com <mailto:bowenl...@gmail.com>> >> >>> > >> wrote: >> >>> > >> > > > >> >>> > >> > > > > Hi devs, >> >>> > >> > > > > >> >>> > >> > > > > I've been experiencing the pain resulting from lack >> >>> of stable >> >>> > >> build >> >>> > >> > > > > capacity on Travis for Flink PRs [1]. Specifically, I >> >>> noticed >> >>> > >> often >> >>> > >> > > that >> >>> > >> > > > no >> >>> > >> > > > > build in the queue is making any progress for hours, and >> >>> > suddenly >> >>> > >> 5 >> >>> > >> > or >> >>> > >> > > 6 >> >>> > >> > > > > builds kick off all together after the long pause. >> >>> I'm at PST >> >>> > >> > (UTC-08) >> >>> > >> > > > time >> >>> > >> > > > > zone, and I've seen pause can be as long as 6 hours >> >>> from PST 9am >> >>> > >> to >> >>> > >> > 3pm >> >>> > >> > > > > (let alone the time needed to drain the queue >> >>> afterwards). >> >>> > >> > > > > >> >>> > >> > > > > I think this has greatly impacted our productivity. I've >> >>> > >> experienced >> >>> > >> > > that >> >>> > >> > > > > PRs submitted in the early morning of PST time zone >> >>> won't finish >> >>> > >> > their >> >>> > >> > > > > build until late night of the same day. >> >>> > >> > > > > >> >>> > >> > > > > So my questions are: >> >>> > >> > > > > >> >>> > >> > > > > - Has anyone else experienced the same problem or >> >>> have similar >> >>> > >> > > > observation >> >>> > >> > > > > on TravisCI? (I suspect it has things to do with time >> >>> zone) >> >>> > >> > > > > >> >>> > >> > > > > - What pricing plan of TravisCI is Flink currently >> >>> using? Is it >> >>> > >> the >> >>> > >> > > free >> >>> > >> > > > > plan for open source projects? What are the >> >>> guaranteed build >> >>> > >> capacity >> >>> > >> > > of >> >>> > >> > > > > the current plan? >> >>> > >> > > > > >> >>> > >> > > > > - If the current pricing plan (either free or paid) >> >> can't >> >>> > provide >> >>> > >> > > stable >> >>> > >> > > > > build capacity, can we upgrade to a higher priced >> >>> plan with >> >>> > larger >> >>> > >> > and >> >>> > >> > > > more >> >>> > >> > > > > stable build capacity? >> >>> > >> > > > > >> >>> > >> > > > > BTW, another factor that contribute to the >> >>> productivity problem >> >>> > is >> >>> > >> > that >> >>> > >> > > > > our build is slow - we run full build for every PR and a >> >>> > >> successful >> >>> > >> > > full >> >>> > >> > > > > build takes ~5h. We definitely have more options to >> >>> solve it, >> >>> > for >> >>> > >> > > > instance, >> >>> > >> > > > > modularize the build graphs and reuse artifacts from the >> >>> > previous >> >>> > >> > > build. >> >>> > >> > > > > But I think that can be a big effort which is much >> >>> harder to >> >>> > >> > accomplish >> >>> > >> > > > in >> >>> > >> > > > > a short period of time and may deserve its own separate >> >>> > >> discussion. >> >>> > >> > > > > >> >>> > >> > > > > [1] https://travis-ci.org/apache/flink/pull_requests >> >>> > >> > > > > >> >>> > >> > > > > >> >>> > >> > > > >> >>> > >> > > >> >>> > >> > >> >>> > >> >> >>> > > >> >>> > >> >>> >> >>> >> >>> -- >> >>> Best Regards >> >>> >> >>> Jeff Zhang >> >>> >> >> >>
