+1 and thanks for Chesnay' work on this. Best, Zhijiang ------------------------------------------------------------------ From:Haibo Sun <sunhaib...@163.com> Send Time:2019年7月4日(星期四) 18:21 To:dev <dev@flink.apache.org> Cc:priv...@flink.apache.org <priv...@flink.apache.org> Subject:Re:Re: [VOTE] Migrate to sponsored Travis account
+1. Thank Chesnay for pushing this forward. Best, Haibo At 2019-07-04 17:58:28, "Kurt Young" <ykt...@gmail.com> wrote: >+1 and great thanks Chesnay for pushing this. > >Best, >Kurt > > >On Thu, Jul 4, 2019 at 5:44 PM Aljoscha Krettek <aljos...@apache.org> wrote: > >> +1 >> >> Aljoscha >> >> > On 4. Jul 2019, at 11:09, Stephan Ewen <se...@apache.org> wrote: >> > >> > +1 to move to a private Travis account. >> > >> > I can confirm that Ververica will sponsor a Travis CI plan that is >> > equivalent or a bit higher than the previous ASF quota (10 concurrent >> build >> > queues) >> > >> > Best, >> > Stephan >> > >> > On Thu, Jul 4, 2019 at 10:46 AM Chesnay Schepler <ches...@apache.org> >> wrote: >> > >> >> I've raised a JIRA >> >> <https://issues.apache.org/jira/browse/INFRA-18703>with INFRA to >> inquire >> >> whether it would be possible to switch to a different Travis account, >> >> and if so what steps would need to be taken. >> >> We need a proper confirmation from INFRA since we are not in full >> >> control of the flink repository (for example, we cannot access the >> >> settings page). >> >> >> >> If this is indeed possible, Ververica is willing sponsor a Travis >> >> account for the Flink project. >> >> This would provide us with more than enough resources than we need. >> >> >> >> Since this makes the project more reliant on resources provided by >> >> external companies I would like to vote on this. >> >> >> >> Please vote on this proposal, as follows: >> >> [ ] +1, Approve the migration to a Ververica-sponsored Travis account, >> >> provided that INFRA approves >> >> [ ] -1, Do not approach the migration to a Ververica-sponsored Travis >> >> account >> >> >> >> The vote will be open for at least 24h, and until we have confirmation >> >> from INFRA. The voting period may be shorter than the usual 3 days since >> >> our current is effectively not working. >> >> >> >> On 04/07/2019 06:51, Bowen Li wrote: >> >>> Re: > Are they using their own Travis CI pool, or did the switch to an >> >>> entirely different CI service? >> >>> >> >>> I reached out to Wes and Krisztián from Apache Arrow PMC. They are >> >>> currently moving away from ASF's Travis to their own in-house metal >> >>> machines at [1] with custom CI application at [2]. They've seen >> >>> significant improvement w.r.t both much higher performance and >> >>> basically no resource waiting time, "night-and-day" difference quoting >> >>> Wes. >> >>> >> >>> Re: > If we can just switch to our own Travis pool, just for our >> >>> project, then this might be something we can do fairly quickly? >> >>> >> >>> I believe so, according to [3] and [4] >> >>> >> >>> >> >>> [1] https://ci.ursalabs.org/ <https://ci.ursalabs.org/#/> >> >>> [2] https://github.com/ursa-labs/ursabot >> >>> [3] >> >>> >> https://docs.travis-ci.com/user/migrate/open-source-repository-migration >> >>> [4] >> https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com >> >>> >> >>> >> >>> >> >>> On Wed, Jul 3, 2019 at 12:01 AM Chesnay Schepler <ches...@apache.org >> >>> <mailto:ches...@apache.org>> wrote: >> >>> >> >>> Are they using their own Travis CI pool, or did the switch to an >> >>> entirely different CI service? >> >>> >> >>> If we can just switch to our own Travis pool, just for our >> >>> project, then >> >>> this might be something we can do fairly quickly? >> >>> >> >>> On 03/07/2019 05:55, Bowen Li wrote: >> >>>> I responded in the INFRA ticket [1] that I believe they are >> >>> using a wrong >> >>>> metric against Flink and the total build time is a completely >> >>> different >> >>>> thing than guaranteed build capacity. >> >>>> >> >>>> My response: >> >>>> >> >>>> "As mentioned above, since I started to pay attention to Flink's >> >>> build >> >>>> queue a few tens of days ago, I'm in Seattle and I saw no build >> >>> was kicking >> >>>> off in PST daytime in weekdays for Flink. Our teammates in China >> >>> and Europe >> >>>> have also reported similar observations. So we need to evaluate >> >>> how the >> >>>> large total build time came from - if 1) your number and 2) our >> >>>> observations from three locations that cover pretty much a full >> >>> day, are >> >>>> all true, I **guess** one reason can be that - highly likely the >> >>> extra >> >>>> build time came from weekends when other Apache projects may be >> >>> idle and >> >>>> Flink just drains hard its congested queue. >> >>>> >> >>>> Please be aware of that we're not complaining about the lack of >> >>> resources >> >>>> in general, I'm complaining about the lack of **stable, dedicated** >> >>>> resources. An example for the latter one is, currently even if >> >>> no build is >> >>>> in Flink's queue and I submit a request to be the queue head in PST >> >>>> morning, my build won't even start in 6-8+h. That is an absurd >> >>> amount of >> >>>> waiting time. >> >>>> >> >>>> That's saying, if ASF INFRA decides to adopt a quota system and >> >>> grants >> >>>> Flink five DEDICATED servers that runs all the time only for >> >>> Flink, that'll >> >>>> be PERFECT and can totally solve our problem now. >> >>>> >> >>>> Please be aware of that we're not complaining about the lack of >> >>> resources >> >>>> in general, I'm complaining about the lack of **stable, dedicated** >> >>>> resources. An example for the latter one is, currently even if >> >>> no build is >> >>>> in Flink's queue and I submit a request to be the queue head in PST >> >>>> morning, my build won't even start in 6-8+h. That is an absurd >> >>> amount of >> >>>> waiting time. >> >>>> >> >>>> >> >>>> That's saying, if ASF INFRA decides to adopt a quota system and >> >>> grants >> >>>> Flink five DEDICATED servers that runs all the time only for >> >>> Flink, that'll >> >>>> be PERFECT and can totally solve our problem now. >> >>>> >> >>>> I feel what's missing in the ASF INFRA's Travis resource pool is >> >>> some level >> >>>> of build capacity SLAs and certainty" >> >>>> >> >>>> >> >>>> Again, I believe there are differences in nature of these two >> >>> problems, >> >>>> long build time v.s. lack of dedicated build resource. That's >> >>> saying, >> >>>> shortening build time may relieve the situation, and may not. >> >>> I'm sightly >> >>>> negative on disabling IT cases for PRs, due to the downside is >> >>> that we are >> >>>> at risk of any potential bugs in PR that UTs doesn't catch, and >> >>> may cost a >> >>>> lot more to fix and if it slows others down or even block >> >>> others, but am >> >>>> open to others opinions on it. >> >>>> >> >>>> AFAICT from INFRA ticket[1], donating to ASF INFRA won't be >> >>> feasible to >> >>>> solve our problem since INFRA's pool is fully shared and they >> >>> have no >> >>>> control and finer insights over resource allocation to a >> >>> specific Apache >> >>>> project. As mentioned in [1], Apache Arrow is moving away from >> >>> ASF INFRA >> >>>> Travis pool (they are actually surprised Flink hasn't plan to do >> >>> so). I >> >>>> know that Spark is on its own build infra. If we all agree that >> >>> funding our >> >>>> own build infra, I'd be glad to help investigate any potential >> >>> options >> >>>> after releasing 1.9 since I'm super busy with 1.9 now. >> >>>> >> >>>> [1] https://issues.apache.org/jira/browse/INFRA-18533 >> >>>> >> >>>> >> >>>> >> >>>> On Tue, Jul 2, 2019 at 4:46 AM Chesnay Schepler >> >>> <ches...@apache.org <mailto:ches...@apache.org>> wrote: >> >>>> >> >>>>> As a short-term stopgap, since we can assume this issue to >> >>> become much >> >>>>> worse in the following days/weeks, we could disable IT cases in >> >>> PRs and >> >>>>> only run them on master. >> >>>>> >> >>>>> On 02/07/2019 12:03, Chesnay Schepler wrote: >> >>>>>> People really have to stop thinking that just because >> >>> something works >> >>>>>> for us it is also a good solution. >> >>>>>> Also, please remember that our builds run for 2h from start to >> >>> finish, >> >>>>>> and not the 14 _minutes_ it takes for zeppelin. >> >>>>>> We are dealing with an entirely different scale here, both in >> >>> terms of >> >>>>>> build times and number of builds. >> >>>>>> >> >>>>>> In this very thread people have been complaining about long queue >> >>>>>> times for their builds. Surprise, other Apache projects have been >> >>>>>> suffering the very same thing due to us not controlling our build >> >>>>>> times. While switching services (be it Jenkins, CircleCI or >> >>> whatever) >> >>>>>> will possibly work for us (and these options are actually >> >>> attractive, >> >>>>>> like CircleCI's proper support for build artifacts), it will also >> >>>>>> result in us likely negatively affecting other projects in >> >>> significant >> >>>>>> ways. >> >>>>>> >> >>>>>> Sure, the Jenkins setup has a good user experience for us, at >> >>> the cost >> >>>>>> of blocking Jenkins workers for a _lot_ of time. Right now we >> >>> have 25 >> >>>>>> PR's in our queue; that's possibly 50h we'd consume of Jenkins >> >>>>>> resources, and the European contributors haven't even really >> >>> started yet. >> >>>>>> >> >>>>>> FYI, the latest INFRA response from INFRA-18533: >> >>>>>> >> >>>>>> "Our rough metrics shows that Flink used over 5800 hours of >> >>> build time >> >>>>>> last month. That is equal to EIGHT servers running 24/7 for >> >>> the ENTIRE >> >>>>>> MONTH. EIGHT. nonstop. >> >>>>>> When we discovered this last night, we discussed it some and >> >>> are going >> >>>>>> to tune down Flink to allow only five executors maximum. We >> >> cannot >> >>>>>> allow Flink to consume so much of a Foundation shared resource." >> >>>>>> >> >>>>>> So yes, we either >> >>>>>> a) have to heavily reduce our CI usage or >> >>>>>> b) fund our own, either maintaining it ourselves or donating >> >>> to Apache. >> >>>>>> >> >>>>>> On 02/07/2019 05:11, Bowen Li wrote: >> >>>>>>> By looking at the git history of the Jenkins script, its core >> >>> part >> >>>>>>> was finished in March 2017 (and only two minor update in >> >>> 2017/2018), >> >>>>>>> so it's been running for over two years now and feels like >> >>> Zepplin >> >>>>>>> community has been quite happy with it. @Jeff Zhang >> >>>>>>> <mailto:zjf...@gmail.com <mailto:zjf...@gmail.com>> can you >> >>> share your insights and user >> >>>>>>> experience with the Jenkins+Travis approach? >> >>>>>>> >> >>>>>>> Things like: >> >>>>>>> >> >>>>>>> - has the approach completely solved the resource capacity >> >>> problem >> >>>>>>> for Zepplin community? is Zepplin community happy with the >> >>> result? >> >>>>>>> - is the whole configuration chain stable (e.g. uptime) enough? >> >>>>>>> - how often do you need to maintain the Jenkins infra? how many >> >>>>>>> people are usually involved in maintenance and bug-fixes? >> >>>>>>> >> >>>>>>> The downside of this approach seems mostly to be on the >> >>> maintenance >> >>>>>>> to me - maintain the script and Jenkins infra. >> >>>>>>> >> >>>>>>> ** Having Our Own Travis-CI.com Account ** >> >>>>>>> >> >>>>>>> Another alternative I've been thinking of is to have our own >> >>>>>>> travis-ci.com <http://travis-ci.com> <http://travis-ci.com> >> >>> account with paid dedicated >> >>>>>>> resources. Note travis-ci.org <http://travis-ci.org> >> >>> <http://travis-ci.org> is the free >> >>>>>>> version and travis-ci.com <http://travis-ci.com> >> >>> <http://travis-ci.com> is the commercial >> >>>>>>> version. We currently use a shared resource pool managed by >> >>> ASK INFRA >> >>>>>>> team on travis-ci.org <http://travis-ci.org> >> >>> <http://travis-ci.org>, but we have no control >> >>>>>>> over it - we can't see how it's configured, how much >> >>> resources are >> >>>>>>> available, how resources are allocated among Apache projects, >> >>> etc. >> >>>>>>> The nice thing about having an account on travis-ci.com >> >>> <http://travis-ci.com> >> >>>>>>> <http://travis-ci.com> are: >> >>>>>>> >> >>>>>>> - relatively low cost with much better resource guarantee >> >>> than what >> >>>>>>> we currently have [1]: $249/month with 5 dedicated concurrency, >> >>>>>>> $489/month with 10 concurrency >> >>>>>>> - low maintenance work compared to using Jenkins >> >>>>>>> - (potentially) no migration cost according to Travis's doc [2] >> >>>>>>> (pending verification) >> >>>>>>> - full control over the build capacity/configuration compared to >> >>>>>>> using ASF INFRA's pool >> >>>>>>> >> >>>>>>> I'd be surprised if we as such a vibrant community cannot >> >>> find and >> >>>>>>> fund $249*12=$2988 a year in exchange for a much better >> >> developer >> >>>>>>> experience and much higher productivity. >> >>>>>>> >> >>>>>>> [1] https://travis-ci.com/plans >> >>>>>>> [2] >> >>>>>>> >> >>>>> >> >>> >> >> >> https://docs.travis-ci.com/user/migrate/open-source-repository-migration >> >>>>>>> On Sat, Jun 29, 2019 at 8:39 AM Chesnay Schepler >> >>> <ches...@apache.org <mailto:ches...@apache.org> >> >>>>>>> <mailto:ches...@apache.org <mailto:ches...@apache.org>>> wrote: >> >>>>>>> >> >>>>>>> So yes, the Jenkins job keeps pulling the state from >> >>> Travis until it >> >>>>>>> finishes. >> >>>>>>> >> >>>>>>> Note sure I'm comfortable with the idea of using Jenkins >> >>> workers >> >>>>>>> just to >> >>>>>>> idle for a several hours. >> >>>>>>> >> >>>>>>> On 29/06/2019 14:56, Jeff Zhang wrote: >> >>>>>>>> Here's what zeppelin community did, we make a python >> >>> script to >> >>>>>>> check the >> >>>>>>>> build status of pull request. >> >>>>>>>> Here's script: >> >>>>>>>> >> >>> https://github.com/apache/zeppelin/blob/master/travis_check.py >> >>>>>>>> >> >>>>>>>> And this is the script we used in Jenkins build job. >> >>>>>>>> >> >>>>>>>> if [ -f "travis_check.py" ]; then >> >>>>>>>> git log -n 1 >> >>>>>>>> STATUS=$(curl -s $BUILD_URL | grep -e "GitHub pull >> >>>>>>> request.*from.*" | sed >> >>>>>>>> 's/.*GitHub pull request <a >> >>>>>>>> href=\"\(https[^"]*\).*from[^"]*.\(https[^"]*\).*/\1 >> >>> \2/g') >> >>>>>>>> AUTHOR=$(echo $STATUS | sed 's/.*[/]\(.*\)$/\1/g') >> >>>>>>>> PR=$(echo $STATUS | awk '{print $1}' | sed >> >>>>>>> 's/.*[/]\(.*\)$/\1/g') >> >>>>>>>> #COMMIT=$(git log -n 1 | grep "^Merge:" | awk >> >>> '{print $3}') >> >>>>>>>> #if [ -z $COMMIT ]; then >> >>>>>>>> # COMMIT=$(curl -s >> >>>>>>> https://api.github.com/repos/apache/zeppelin/pulls/$PR >> >>>>>>>> | grep -e "\"label\":" -e "\"ref\":" -e "\"sha\":" | >> >>> tr '\n' ' ' >> >>>>>>> | sed >> >>>>>>>> 's/\(.*sha[^,]*,\)\(.*ref.*\)/\1 = \2/g' | tr = '\n' | >> >>> grep -v >> >>>>>>> "apache:" | >> >>>>>>>> sed 's/.*sha.[^"]*["]\([^"]*\).*/\1/g') >> >>>>>>>> #fi >> >>>>>>>> >> >>>>>>>> # get commit hash from PR >> >>>>>>>> COMMIT=$(curl -s >> >>>>>>> https://api.github.com/repos/apache/zeppelin/pulls/$PR | >> >>>>>>>> grep -e "\"label\":" -e "\"ref\":" -e "\"sha\":" | tr >> >>> '\n' ' ' >> >>>>>>> | sed >> >>>>>>>> 's/\(.*sha[^,]*,\)\(.*ref.*\)/\1 = \2/g' | tr = '\n' | >> >>> grep -v >> >>>>>>> "apache:" | >> >>>>>>>> sed 's/.*sha.[^"]*["]\([^"]*\).*/\1/g') >> >>>>>>>> sleep 30 # sleep few moment to wait travis starts >> >>> the build >> >>>>>>>> RET_CODE=0 >> >>>>>>>> python ./travis_check.py ${AUTHOR} ${COMMIT} || >> >>> RET_CODE=$? >> >>>>>>>> if [ $RET_CODE -eq 2 ]; then # try with repository >> >>> name when >> >>>>>>> travis-ci is >> >>>>>>>> not available in the account >> >>>>>>>> RET_CODE=0 >> >>>>>>>> AUTHOR=$(curl -s >> >>>>>>> https://api.github.com/repos/apache/zeppelin/pulls/$PR >> >>>>>>>> | grep '"full_name":' | grep -v "apache/zeppelin" | sed >> >>>>>>>> 's/.*[:][^"]*["]\([^/]*\).*/\1/g') >> >>>>>>>> python ./travis_check.py ${AUTHOR} ${COMMIT} || >> >>> RET_CODE=$? >> >>>>>>>> fi >> >>>>>>>> >> >>>>>>>> if [ $RET_CODE -eq 2 ]; then # fail with can't find >> >>> build >> >>>>>>> information in >> >>>>>>>> the travis >> >>>>>>>> set +x >> >>>>>>>> echo >> >>> "-----------------------------------------------------" >> >>>>>>>> echo "Looks like travis-ci is not configured for >> >>> your fork." >> >>>>>>>> echo "Please setup by swich on 'zeppelin' >> >>> repository at >> >>>>>>>> https://travis-ci.org/profile and travis-ci." >> >>>>>>>> echo "And then make sure 'Build branch updates' >> >>> option is >> >>>>>>> enabled in >> >>>>>>>> the settings >> >>> https://travis-ci.org/${AUTHOR}/zeppelin/settings >> >>> <https://travis-ci.org/$%7BAUTHOR%7D/zeppelin/settings> >> >>>>>>> <https://travis-ci.org/$%7BAUTHOR%7D/zeppelin/settings>." >> >>>>>>>> echo "" >> >>>>>>>> echo "To trigger CI after setup, you will need >> >>> ammend your >> >>>>>>> last commit >> >>>>>>>> with" >> >>>>>>>> echo "git commit --amend" >> >>>>>>>> echo "git push your-remote HEAD --force" >> >>>>>>>> echo "" >> >>>>>>>> echo "See >> >>>>>>>> >> >>>>>>> >> >>>>> >> >>> >> >> >> http://zeppelin.apache.org/contribution/contributions.html#continuous-integration >> >>>>>>>> ." >> >>>>>>>> fi >> >>>>>>>> >> >>>>>>>> exit $RET_CODE >> >>>>>>>> else >> >>>>>>>> set +x >> >>>>>>>> echo "travis_check.py does not exists" >> >>>>>>>> exit 1 >> >>>>>>>> fi >> >>>>>>>> >> >>>>>>>> Chesnay Schepler <ches...@apache.org >> >>> <mailto:ches...@apache.org> >> >>>>>>> <mailto:ches...@apache.org <mailto:ches...@apache.org>>> >> >>> 于2019年6月29日周六 下午3:17写道： >> >>>>>>>> >> >>>>>>>>> Does this imply that a Jenkins job is active as long >> >>> as the >> >>>>>>> Travis build >> >>>>>>>>> runs? >> >>>>>>>>> >> >>>>>>>>> On 26/06/2019 21:28, Bowen Li wrote: >> >>>>>>>>>> Hi, >> >>>>>>>>>> >> >>>>>>>>>> @Dawid, I think the "long test running" as I >> >>> mentioned in the >> >>>>>>> first >> >>>>>>>>> email, >> >>>>>>>>>> also as you guys said, belongs to "a big effort >> >>> which is much >> >>>>>>> harder to >> >>>>>>>>>> accomplish in a short period of time and may deserve >> >>> its own >> >>>>>>> separate >> >>>>>>>>>> discussion". Thus I didn't include it in what we can >> >>> do in a >> >>>>>>> foreseeable >> >>>>>>>>>> short term. >> >>>>>>>>>> >> >>>>>>>>>> Besides, I don't think that's the ultimate reason >> >>> for lack of >> >>>>>>> build >> >>>>>>>>>> resources. Even if the build is shortened to >> >>> something like >> >>>>>>> 2h, the >> >>>>>>>>>> problems of no build machine works about 6 or more >> >>> hours in >> >>>>>>> PST daytime >> >>>>>>>>>> that I described will still happen, because no >> >>> machine from >> >>>>>>> ASF INFRA's >> >>>>>>>>>> pool is allocated to Flink. As I have paid close >> >>> attention to >> >>>>>>> the build >> >>>>>>>>>> queue in the past few weekdays, it's a pretty clear >> >>> pattern now. >> >>>>>>>>>> >> >>>>>>>>>> **The ultimate root cause** for that is - we don't >> >>> have any >> >>>>>>> **dedicated** >> >>>>>>>>>> build resources that we can stably rely on. I'm >> >>> actually ok to >> >>>>>>> wait for a >> >>>>>>>>>> long time if there are build requests running, it >> >>> means at >> >>>>>>> least we are >> >>>>>>>>>> making progress. But I'm not ok with no build >> >>> resource. A >> >>>>>>> better place I >> >>>>>>>>>> think we should aim at in short term is to always >> >>> have at >> >>>>>>> least a central >> >>>>>>>>>> pool (can be 3 or 5) of machines dedicated to build >> >>> Flink at >> >>>>>>> any time, or >> >>>>>>>>>> maybe use users resources. >> >>>>>>>>>> >> >>>>>>>>>> @Chesnay @Robert I synced with Jeff offline that >> >>> Zeppelin >> >>>>>>> community is >> >>>>>>>>>> using a Jenkins job to automatically build on users' >> >>> travis >> >>>>>>> account and >> >>>>>>>>>> link the result back to github PR. I guess the >> >>> Jenkins job >> >>>>>>> would fetch >> >>>>>>>>>> latest upstream master and build the PR against it. >> >>> Jeff has >> >>>>>>> filed >> >>>>>>>>> tickets >> >>>>>>>>>> to learn and get access to the Jenkins infra. It'll >> >>> better to >> >>>>>>> fully >> >>>>>>>>>> understand it first before judging this approach. >> >>>>>>>>>> >> >>>>>>>>>> I also heard good things about CircleCI, and ASF >> >>> INFRA seems >> >>>>>>> to have a >> >>>>>>>>> pool >> >>>>>>>>>> of build capacity there too. Can be an alternative >> >>> to consider. >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> On Wed, Jun 26, 2019 at 12:44 AM Dawid Wysakowicz < >> >>>>>>>>> dwysakow...@apache.org >> >>> <mailto:dwysakow...@apache.org> <mailto:dwysakow...@apache.org >> >>> <mailto:dwysakow...@apache.org>>> >> >>>>>>>>>> wrote: >> >>>>>>>>>> >> >>>>>>>>>>> Sorry to jump in late, but I think Bowen missed the >> >>> most >> >>>>>>> important point >> >>>>>>>>>>> from Chesnay's previous message in the summary. The >> >>> ultimate >> >>>>>>> reason for >> >>>>>>>>>>> all the problems is that the tests take close to 2 >> >>> hours to >> >>>>>>> run already. >> >>>>>>>>>>> I fully support this claim: "Unless people start >> >>> caring about >> >>>>>>> test times >> >>>>>>>>>>> before adding them, this issue cannot be solved" >> >>>>>>>>>>> >> >>>>>>>>>>> This is also another reason why using user's Travis >> >>> account >> >>>>>>> won't help. >> >>>>>>>>>>> Every few weeks we reach the user's time limit for >> >>> a single >> >>>>>>> profile. >> >>>>>>>>>>> This makes the user's builds simply fail, until we >> >>> either >> >>>>>>> properly >> >>>>>>>>>>> decrease the time the tests take (which I am not >> >>> sure we ever >> >>>>>>> did) or >> >>>>>>>>>>> postpone the problem by splitting into more >> >>> profiles. (Note >> >>>>>>> that the ASF >> >>>>>>>>>>> Travis account has higher time limits) >> >>>>>>>>>>> >> >>>>>>>>>>> Best, >> >>>>>>>>>>> >> >>>>>>>>>>> Dawid >> >>>>>>>>>>> >> >>>>>>>>>>> On 26/06/2019 09:36, Robert Metzger wrote: >> >>>>>>>>>>>> Do we know if using "the best" available hardware >> >>> would >> >>>>>>> improve the >> >>>>>>>>> build >> >>>>>>>>>>>> times? >> >>>>>>>>>>>> Imagine we would run the build on machines with >> >>> plenty of >> >>>>>>> main memory >> >>>>>>>>> to >> >>>>>>>>>>>> mount everything to ramdisk + the latest CPU >> >>> architecture? >> >>>>>>>>>>>> >> >>>>>>>>>>>> Throwing hardware at the problem could help reduce >> >>> the time >> >>>>>>> of an >> >>>>>>>>>>>> individual build, and using our own infrastructure >> >>> would >> >>>>>>> remove our >> >>>>>>>>>>>> dependency on Apache's Travis account (with the >> >>> obvious >> >>>>>>> downside of >> >>>>>>>>>>> having >> >>>>>>>>>>>> to maintain the infrastructure) >> >>>>>>>>>>>> We could use an open source travis alternative, to >> >>> have a >> >>>>>>> similar >> >>>>>>>>>>>> experience and make the migration easy. >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> On Wed, Jun 26, 2019 at 9:34 AM Chesnay Schepler >> >>>>>>> <ches...@apache.org <mailto:ches...@apache.org> >> >>> <mailto:ches...@apache.org <mailto:ches...@apache.org>>> >> >>>>>>>>>>> wrote: >> >>>>>>>>>>>>>> From what I gathered, there's no special >> >>> sauce that the >> >>>>>>> Zeppelin >> >>>>>>>>>>>>> project uses which actually integrates a users >> >> Travis >> >>>>>>> account into the >> >>>>>>>>>>> PR. >> >>>>>>>>>>>>> They just disabled Travis for PRs. And that's >> >>> kind of it. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Naturally we can do this (duh) and safe the ASF a >> >>> fair >> >>>>>>> amount of >> >>>>>>>>>>>>> resources, but there are downsides: >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> The discoverability of the Travis check takes a >> >>> nose-dive. >> >>>>>>> Either we >> >>>>>>>>>>>>> require every contributor to always, an every >> >>> commit, also >> >>>>>>> post a >> >>>>>>>>> Travis >> >>>>>>>>>>>>> build, or we have the reviewer sift through the >> >>>>>>> contributors account >> >>>>>>>>> to >> >>>>>>>>>>>>> find it. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> This is rather cumbersome. Additionally, it's >> >>> also not >> >>>>>>> equivalent to >> >>>>>>>>>>>>> having a PR build. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> A normal branch build takes a branch as is and >> >>> tests it. A >> >>>>>>> PR build >> >>>>>>>>>>>>> merges the branch into master, and then runs it. >> >>> (Fun fact: >> >>>>>>> This is >> >>>>>>>>> why >> >>>>>>>>>>>>> a PR without merge conflicts is not being run on >> >>> Travis.) >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> And ultimately, everyone can already make use of >> >> this >> >>>>>>> approach anyway. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> On 25/06/2019 08:02, Jark Wu wrote: >> >>>>>>>>>>>>>> Hi Jeff, >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Thanks for sharing the Zeppelin approach. I >> >>> think it's a >> >>>>>>> good idea to >> >>>>>>>>>>>>>> leverage user's travis account. >> >>>>>>>>>>>>>> In this way, we can have almost unlimited >> >>> concurrent build >> >>>>>>> jobs and >> >>>>>>>>>>>>>> developers can restart build by themselves >> >>> (currently only >> >>>>>>> committers >> >>>>>>>>>>>>>> can restart PR's build). >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> But I'm still not very clear how to integrate >> >> user's >> >>>>>>> travis build >> >>>>>>>>> into >> >>>>>>>>>>>>>> the Flink pull request's build automatically. >> >>> Can you >> >>>>>>> explain more in >> >>>>>>>>>>>>>> detail? >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Another question: does travis only build >> >>> branches for user >> >>>>>>> account? >> >>>>>>>>>>>>>> My concern is that builds for PRs will rebase >> >> user's >> >>>>>>> commits against >> >>>>>>>>>>>>>> current master branch. >> >>>>>>>>>>>>>> This will help us to find problems before >> >>> merge. Builds >> >>>>>>> for branches >> >>>>>>>>>>>>>> will lose the impact of new commits in master. >> >>>>>>>>>>>>>> How does Zeppelin solve this problem? >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Thanks again for sharing the idea. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Regards, >> >>>>>>>>>>>>>> Jark >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> On Tue, 25 Jun 2019 at 11:01, Jeff Zhang >> >>> <zjf...@gmail.com <mailto:zjf...@gmail.com> >> >>>>>>> <mailto:zjf...@gmail.com <mailto:zjf...@gmail.com>> >> >>>>>>>>>>>>>> <mailto:zjf...@gmail.com >> >>> <mailto:zjf...@gmail.com> <mailto:zjf...@gmail.com >> >>> <mailto:zjf...@gmail.com>>>> wrote: >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Hi Folks, >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Zeppelin meet this kind of issue before, we solve >> >>>>>>> it by >> >>>>>>>>> delegating >> >>>>>>>>>>>>>> each >> >>>>>>>>>>>>>> one's PR build to his travis account >> >>> (Everyone can >> >>>>>>> have 5 free >> >>>>>>>>>>>>>> slot for >> >>>>>>>>>>>>>> travis build). >> >>>>>>>>>>>>>> Apache account travis build is only triggered when >> >>>>>>> PR is merged. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Kurt Young <ykt...@gmail.com >> >>> <mailto:ykt...@gmail.com> >> >>>>>>> <mailto:ykt...@gmail.com <mailto:ykt...@gmail.com>> >> >>> <mailto:ykt...@gmail.com <mailto:ykt...@gmail.com> >> >>>>>>> <mailto:ykt...@gmail.com <mailto:ykt...@gmail.com>>>> >> >>>>>>>>>>>>>> 于2019年6月25日周二 上午10:16写道： >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> (Forgot to cc George) >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Best, >> >>>>>>>>>>>>>>> Kurt >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> On Tue, Jun 25, 2019 at 10:16 AM Kurt Young >> >>>>>>> <ykt...@gmail.com <mailto:ykt...@gmail.com> >> >>> <mailto:ykt...@gmail.com <mailto:ykt...@gmail.com>> >> >>>>>>>>>>>>>> <mailto:ykt...@gmail.com >> >>> <mailto:ykt...@gmail.com> <mailto:ykt...@gmail.com >> >>> <mailto:ykt...@gmail.com>>>> >> >>>>>>> wrote: >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> Hi Bowen, >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> Thanks for bringing this up. We >> >>> actually have >> >>>>>>> discussed >> >>>>>>>>> about >> >>>>>>>>>>>>>> this, and I >> >>>>>>>>>>>>>>>> think Till and George have >> >>>>>>>>>>>>>>>> already spend sometime investigating >> >>> it. I have >> >>>>>>> cced both of >> >>>>>>>>>>>>>> them, and >> >>>>>>>>>>>>>>>> maybe they can share >> >>>>>>>>>>>>>>>> their findings. >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> Best, >> >>>>>>>>>>>>>>>> Kurt >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> On Tue, Jun 25, 2019 at 10:08 AM Jark Wu >> >>>>>>> <imj...@gmail.com <mailto:imj...@gmail.com> >> >>> <mailto:imj...@gmail.com <mailto:imj...@gmail.com>> >> >>>>>>>>>>>>>> <mailto:imj...@gmail.com >> >>> <mailto:imj...@gmail.com> <mailto:imj...@gmail.com >> >>> <mailto:imj...@gmail.com>>>> >> >>>>>>> wrote: >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> Hi Bowen, >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> Thanks for bringing this. We also >> >>> suffered from >> >>>>>>> the long >> >>>>>>>>>>>>>> build time. >> >>>>>>>>>>>>>>>>> I agree that we should focus on >> >>> solving build >> >>>>>>> capacity >> >>>>>>>>>>>>>> problem in the >> >>>>>>>>>>>>>>>>> thread. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> My observation is there is only one >> >>> build is >> >>>>>>> running, all >> >>>>>>>>> the >> >>>>>>>>>>>>>> others >> >>>>>>>>>>>>>>>>> (other >> >>>>>>>>>>>>>>>>> PRs, master) are pending. >> >>>>>>>>>>>>>>>>> The pricing plan[1] of travis shows >> >>> it can >> >>>>>>> support >> >>>>>>>>> concurrent >> >>>>>>>>>>>>>> build >> >>>>>>>>>>>>>>> jobs. >> >>>>>>>>>>>>>>>>> But I don't know which plan we are >> >>> using, might >> >>>>>>> be the free >> >>>>>>>>>>>>>> plan for >> >>>>>>>>>>>>>>> open >> >>>>>>>>>>>>>>>>> source. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> I cc-ed Chesnay who may have some >> >>> experience on >> >>>>>>> Travis. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> Regards, >> >>>>>>>>>>>>>>>>> Jark >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> [1]: https://travis-ci.com/plans >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> On Tue, 25 Jun 2019 at 08:11, Bowen Li < >> >>>>>>>>> bowenl...@gmail.com <mailto:bowenl...@gmail.com> >> >>> <mailto:bowenl...@gmail.com <mailto:bowenl...@gmail.com>> >> >>>>>>>>>>>>>> <mailto:bowenl...@gmail.com >> >>> <mailto:bowenl...@gmail.com> >> >>>>>>> <mailto:bowenl...@gmail.com >> >>> <mailto:bowenl...@gmail.com>>>> wrote: >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> Hi Steven, >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> I think you may not read what I >> >>> wrote. The >> >>>>>>> discussion is >> >>>>>>>>>>> about >> >>>>>>>>>>>>>>> "unstable >> >>>>>>>>>>>>>>>>>> build **capacity**", in another word >> >>>>>>> "unstable / lack of >> >>>>>>>>>>> build >> >>>>>>>>>>>>>>>>> resources", >> >>>>>>>>>>>>>>>>>> not "unstable build". >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> On Mon, Jun 24, 2019 at 4:40 PM >> >>> Steven Wu >> >>>>>>>>>>>>>> <stevenz...@gmail.com >> >>> <mailto:stevenz...@gmail.com> <mailto:stevenz...@gmail.com >> >>> <mailto:stevenz...@gmail.com>> >> >>>>>>> <mailto:stevenz...@gmail.com >> >>> <mailto:stevenz...@gmail.com> <mailto:stevenz...@gmail.com >> >>> <mailto:stevenz...@gmail.com>>>> >> >>>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> long and sometimes unstable build is >> >>>>>>> definitely a pain >> >>>>>>>>>>>>> point. >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> I suspect the build failure here in >> >>>>>>>>> flink-connector-kafka >> >>>>>>>>>>>>>> is not >> >>>>>>>>>>>>>>>>> related >> >>>>>>>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>>>>> my change. but there is no easy >> >>> re-run the >> >>>>>>> build on >> >>>>>>>>>>>>>> travis UI. >> >>>>>>>>>>>>>>> Google >> >>>>>>>>>>>>>>>>>>> search showed a trick of >> >>> close-and-open the >> >>>>>>> PR will >> >>>>>>>>>>>>>> trigger rebuild. >> >>>>>>>>>>>>>>>>> but >> >>>>>>>>>>>>>>>>>>> that could add noises to the PR >> >>> activities. >> >>>>>>>>>>>>>>>>>>> >> >>>>>>> https://travis-ci.org/apache/flink/jobs/545555519 >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> travis-ci for my personal repo >> >>> often failed >> >>>>>>> with >> >>>>>>>>>>>>>> exceeding time >> >>>>>>>>>>>>>>> limit >> >>>>>>>>>>>>>>>>>> after >> >>>>>>>>>>>>>>>>>>> 4+ hours. >> >>>>>>>>>>>>>>>>>>> The job exceeded the maximum time >> >>> limit for >> >>>>>>> jobs, and >> >>>>>>>>> has >> >>>>>>>>>>>>>> been >> >>>>>>>>>>>>>>>>>> terminated. >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> On Mon, Jun 24, 2019 at 4:15 PM >> >>> Bowen Li >> >>>>>>>>>>>>>> <bowenl...@gmail.com >> >>> <mailto:bowenl...@gmail.com> <mailto:bowenl...@gmail.com >> >>> <mailto:bowenl...@gmail.com>> >> >>>>>>> <mailto:bowenl...@gmail.com <mailto:bowenl...@gmail.com> >> >>> <mailto:bowenl...@gmail.com <mailto:bowenl...@gmail.com>>>> >> >>>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>> https://travis-ci.org/apache/flink/builds/549681530 >> >>>>>>>>>>>>>> This build >> >>>>>>>>>>>>>>>>>> request >> >>>>>>>>>>>>>>>>>>>> has >> >>>>>>>>>>>>>>>>>>>> been sitting at **HEAD of the >> >>> queue** >> >>>>>>> since I first >> >>>>>>>>> saw >> >>>>>>>>>>>>>> it at PST >> >>>>>>>>>>>>>>>>>> 10:30am >> >>>>>>>>>>>>>>>>>>>> (not sure how long it's been >> >>> there before >> >>>>>>> 10:30am). >> >>>>>>>>>>>>>> It's PST >> >>>>>>>>>>>>>>> 4:12pm >> >>>>>>>>>>>>>>>>> now >> >>>>>>>>>>>>>>>>>>> and >> >>>>>>>>>>>>>>>>>>>> it hasn't started yet. >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> On Mon, Jun 24, 2019 at 2:48 PM >> >>> Bowen Li >> >>>>>>>>>>>>>> <bowenl...@gmail.com >> >>> <mailto:bowenl...@gmail.com> <mailto:bowenl...@gmail.com >> >>> <mailto:bowenl...@gmail.com>> >> >>>>>>> <mailto:bowenl...@gmail.com <mailto:bowenl...@gmail.com> >> >>> <mailto:bowenl...@gmail.com <mailto:bowenl...@gmail.com>>>> >> >>>>>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> Hi devs, >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> I've been experiencing the pain >> >>>>>>> resulting from lack >> >>>>>>>>>>>>>> of stable >> >>>>>>>>>>>>>>>>> build >> >>>>>>>>>>>>>>>>>>>>> capacity on Travis for Flink >> >>> PRs [1]. >> >>>>>>>>> Specifically, I >> >>>>>>>>>>>>>> noticed >> >>>>>>>>>>>>>>>>> often >> >>>>>>>>>>>>>>>>>>> that >> >>>>>>>>>>>>>>>>>>>> no >> >>>>>>>>>>>>>>>>>>>>> build in the queue is making any >> >>>>>>> progress for >> >>>>>>>>> hours, >> >>>>>>>>>>> and >> >>>>>>>>>>>>>>> suddenly >> >>>>>>>>>>>>>>>>> 5 >> >>>>>>>>>>>>>>>>>> or >> >>>>>>>>>>>>>>>>>>> 6 >> >>>>>>>>>>>>>>>>>>>>> builds kick off all together >> >>> after the >> >>>>>>> long pause. >> >>>>>>>>>>>>>> I'm at PST >> >>>>>>>>>>>>>>>>>> (UTC-08) >> >>>>>>>>>>>>>>>>>>>> time >> >>>>>>>>>>>>>>>>>>>>> zone, and I've seen pause can >> >>> be as >> >>>>>>> long as 6 hours >> >>>>>>>>>>>>>> from PST 9am >> >>>>>>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>>>> 3pm >> >>>>>>>>>>>>>>>>>>>>> (let alone the time needed to >> >>> drain the >> >>>>>>> queue >> >>>>>>>>>>>>>> afterwards). >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> I think this has greatly >> >>> impacted our >> >>>>>>> productivity. >> >>>>>>>>>>> I've >> >>>>>>>>>>>>>>>>> experienced >> >>>>>>>>>>>>>>>>>>> that >> >>>>>>>>>>>>>>>>>>>>> PRs submitted in the early >> >>> morning of >> >>>>>>> PST time zone >> >>>>>>>>>>>>>> won't finish >> >>>>>>>>>>>>>>>>>> their >> >>>>>>>>>>>>>>>>>>>>> build until late night of the >> >>> same day. >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> So my questions are: >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> - Has anyone else experienced >> >>> the same >> >>>>>>> problem or >> >>>>>>>>>>>>>> have similar >> >>>>>>>>>>>>>>>>>>>> observation >> >>>>>>>>>>>>>>>>>>>>> on TravisCI? (I suspect it >> >>> has things >> >>>>>>> to do with >> >>>>>>>>> time >> >>>>>>>>>>>>>> zone) >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> - What pricing plan of >> >>> TravisCI is >> >>>>>>> Flink currently >> >>>>>>>>>>>>>> using? Is it >> >>>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>> free >> >>>>>>>>>>>>>>>>>>>>> plan for open source >> >>> projects? What >> >>>>>>> are the >> >>>>>>>>>>>>>> guaranteed build >> >>>>>>>>>>>>>>>>> capacity >> >>>>>>>>>>>>>>>>>>> of >> >>>>>>>>>>>>>>>>>>>>> the current plan? >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> - If the current pricing plan >> >>> (either >> >>>>>>> free or paid) >> >>>>>>>>>>>>> can't >> >>>>>>>>>>>>>>> provide >> >>>>>>>>>>>>>>>>>>> stable >> >>>>>>>>>>>>>>>>>>>>> build capacity, can we >> >>> upgrade to a >> >>>>>>> higher priced >> >>>>>>>>>>>>>> plan with >> >>>>>>>>>>>>>>> larger >> >>>>>>>>>>>>>>>>>> and >> >>>>>>>>>>>>>>>>>>>> more >> >>>>>>>>>>>>>>>>>>>>> stable build capacity? >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> BTW, another factor that >> >>> contribute to >> >>>>>>> the >> >>>>>>>>>>>>>> productivity problem >> >>>>>>>>>>>>>>> is >> >>>>>>>>>>>>>>>>>> that >> >>>>>>>>>>>>>>>>>>>>> our build is slow - we run >> >>> full build >> >>>>>>> for every PR >> >>>>>>>>>>> and a >> >>>>>>>>>>>>>>>>> successful >> >>>>>>>>>>>>>>>>>>> full >> >>>>>>>>>>>>>>>>>>>>> build takes ~5h. We >> >>> definitely have >> >>>>>>> more options to >> >>>>>>>>>>>>>> solve it, >> >>>>>>>>>>>>>>> for >> >>>>>>>>>>>>>>>>>>>> instance, >> >>>>>>>>>>>>>>>>>>>>> modularize the build graphs >> >>> and reuse >> >>>>>>> artifacts >> >>>>>>>>> from >> >>>>>>>>>>> the >> >>>>>>>>>>>>>>> previous >> >>>>>>>>>>>>>>>>>>> build. >> >>>>>>>>>>>>>>>>>>>>> But I think that can be a big >> >>> effort >> >>>>>>> which is much >> >>>>>>>>>>>>>> harder to >> >>>>>>>>>>>>>>>>>> accomplish >> >>>>>>>>>>>>>>>>>>>> in >> >>>>>>>>>>>>>>>>>>>>> a short period of time and >> >>> may deserve >> >>>>>>> its own >> >>>>>>>>>>> separate >> >>>>>>>>>>>>>>>>> discussion. >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> [1] >> >>>>>>>>> https://travis-ci.org/apache/flink/pull_requests >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> -- >> >>>>>>>>>>>>>> Best Regards >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Jeff Zhang >> >>>>>>>>>>>>>> >> >>>>>>>>> >> >>>>>>> >> >>>>>> >> >>>>> >> >>> >> >> >> >> >> >>
