Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-08 Thread Krisztián Szűcs
xec cpp sh -c 'echo hello > /tmp/hello.txt' > WARNING: The CI_ARROW_SHA variable is not set. Defaulting to a blank > string. > WARNING: The CI_ARROW_BRANCH variable is not set. Defaulting to a blank > string. > % docker-compose exec cpp cat /tmp/hello.txt > WARNING:

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-07 Thread Sutou Kouhei
ARNING: The CI_ARROW_BRANCH variable is not set. Defaulting to a blank string. % docker-compose exec cpp cat /tmp/hello.txt WARNING: The CI_ARROW_SHA variable is not set. Defaulting to a blank string. WARNING: The CI_ARROW_BRANCH variable is not set. Defaulting to a blank string. hello % docker-compo

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-06 Thread Krisztián Szűcs
On Fri, Sep 6, 2019 at 7:56 PM Wes McKinney wrote: > On Fri, Sep 6, 2019 at 3:18 AM Krisztián Szűcs > wrote: > > > > Hey Wes, > > > > On Fri, Sep 6, 2019 at 12:23 AM Wes McKinney > wrote: > > > > > hi Krisztian, > > > > > > Anyone who's developing in the project can see that the Buildbot setup

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-06 Thread Wes McKinney
On Fri, Sep 6, 2019 at 3:18 AM Krisztián Szűcs wrote: > > Hey Wes, > > On Fri, Sep 6, 2019 at 12:23 AM Wes McKinney wrote: > > > hi Krisztian, > > > > Anyone who's developing in the project can see that the Buildbot setup > > is working well (at least for Linux builds) and giving much more > > ti

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-06 Thread Antoine Pitrou
Le 06/09/2019 à 16:18, Krisztián Szűcs a écrit : > On Fri, Sep 6, 2019 at 12:48 PM Antoine Pitrou wrote: > >> On Fri, 6 Sep 2019 12:41:15 +0200 >> Krisztián Szűcs wrote: I get the impression that it is a complicated and fragile solution to the problem. >>> Ursabot has a bu

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-06 Thread Krisztián Szűcs
On Fri, Sep 6, 2019 at 12:48 PM Antoine Pitrou wrote: > On Fri, 6 Sep 2019 12:41:15 +0200 > Krisztián Szűcs wrote: > > > > > > I get the impression that it is a complicated and fragile solution to > > > the problem. > > > > > Ursabot has a bunch of tests to ensure that we don't brake any of the

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-06 Thread Antoine Pitrou
On Fri, 6 Sep 2019 12:41:15 +0200 Krisztián Szűcs wrote: > > > > I get the impression that it is a complicated and fragile solution to > > the problem. > > > Ursabot has a bunch of tests to ensure that we don't brake any of the > functionality, > so fragility can be avoided by testing it. Testi

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-06 Thread Krisztián Szűcs
On Fri, Sep 6, 2019 at 12:15 PM Antoine Pitrou wrote: > > Le 06/09/2019 à 12:13, Krisztián Szűcs a écrit : > > On Fri, Sep 6, 2019 at 12:01 PM Antoine Pitrou > wrote: > > > >> > >> Le 06/09/2019 à 10:07, Krisztián Szűcs a écrit : > >>> For example trigger a builder for changes affecting files un

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-06 Thread Antoine Pitrou
Le 06/09/2019 à 12:13, Krisztián Szűcs a écrit : > On Fri, Sep 6, 2019 at 12:01 PM Antoine Pitrou wrote: > >> >> Le 06/09/2019 à 10:07, Krisztián Szűcs a écrit : >>> For example trigger a builder for changes affecting files under arrow/ci >>> which reloads the builder object within the build ma

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-06 Thread Krisztián Szűcs
On Fri, Sep 6, 2019 at 12:01 PM Antoine Pitrou wrote: > > Le 06/09/2019 à 10:07, Krisztián Szűcs a écrit : > > For example trigger a builder for changes affecting files under arrow/ci > > which reloads the builder object within the build master's process. > > I am asking you how this affects only

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-06 Thread Antoine Pitrou
Le 06/09/2019 à 10:07, Krisztián Szűcs a écrit : > For example trigger a builder for changes affecting files under arrow/ci > which reloads the builder object within the build master's process. I am asking you how this affects only the current build and not other concurrent builds. Regards Ant

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-06 Thread Krisztián Szűcs
On Fri, Sep 6, 2019 at 12:23 AM Wes McKinney wrote: > hi Krisztian, > > Anyone who's developing in the project can see that the Buildbot setup > is working well (at least for Linux builds) and giving much more > timely feedback, which has been very helpful. > > I'm concerned about the "ursabot" a

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-06 Thread Krisztián Szűcs
Hey Wes, On Fri, Sep 6, 2019 at 12:23 AM Wes McKinney wrote: > hi Krisztian, > > Anyone who's developing in the project can see that the Buildbot setup > is working well (at least for Linux builds) and giving much more > timely feedback, which has been very helpful. > > I'm concerned about the "

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-06 Thread Krisztián Szűcs
For example trigger a builder for changes affecting files under arrow/ci which reloads the builder object within the build master's process. We are not limited to shell commands, arbitrary python functions can be executed too, but the semantics would be similar to running MasterShellCommand [1]. [

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-05 Thread Wes McKinney
hi Krisztian, Anyone who's developing in the project can see that the Buildbot setup is working well (at least for Linux builds) and giving much more timely feedback, which has been very helpful. I'm concerned about the "ursabot" approach for a few reasons: * If we are to centralize our tooling

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-05 Thread Antoine Pitrou
Le 05/09/2019 à 15:04, Krisztián Szűcs a écrit : >> >> If going with buildbot, this means that the various build steps need to >> be generic like in Travis-CI (e.g. "install", "setup", "before-test", >> "test", "after-test"...) and their contents expressed outside of the >> buildmaster configurat

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-05 Thread Krisztián Szűcs
Hey Antoine, On Thu, Sep 5, 2019 at 2:54 PM Antoine Pitrou wrote: > > Le 05/09/2019 à 14:43, Uwe L. Korn a écrit : > > Hello Krisztián, > > > >> Am 05.09.2019 um 14:22 schrieb Krisztián Szűcs < > szucs.kriszt...@gmail.com>: > >> > >>> * The build configuration is automatically updated on a merge

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-05 Thread Antoine Pitrou
Le 05/09/2019 à 14:43, Uwe L. Korn a écrit : > Hello Krisztián, > >> Am 05.09.2019 um 14:22 schrieb Krisztián Szűcs : >> >>> * The build configuration is automatically updated on a merge to master? >>> >> Not yet, but this can be automatized too with buildbot itself. > > This is something I wou

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-05 Thread Uwe L. Korn
Hello Krisztián, > Am 05.09.2019 um 14:22 schrieb Krisztián Szűcs : > >> * The build configuration is automatically updated on a merge to master? >> > Not yet, but this can be automatized too with buildbot itself. This is something I would actually like to have before getting rid of the Travi

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-05 Thread Krisztián Szűcs
Hey Uwe, On Thu, Sep 5, 2019 at 1:49 PM Uwe L. Korn wrote: > Hello Krisztián, > > I like this proposal. CI coverage and response time is a crucial thing for > the health of the project. In general I like the consolidation and local > reproducibility of tge builds. Some questions I wanted to ask

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-05 Thread Uwe L. Korn
Hello Krisztián, I like this proposal. CI coverage and response time is a crucial thing for the health of the project. In general I like the consolidation and local reproducibility of tge builds. Some questions I wanted to ask to make sure I understand your proposal correctly (hopefully they a

[PROPOSAL] Consolidate Arrow's CI configuration

2019-08-29 Thread Krisztián Szűcs
Hi, Arrow's current continuous integration setup utilizes multiple CI providers, tools, and scripts: - Unit tests are running on Travis and Appveyor - Binary packaging builds are running on crossbow, an abstraction over multiple CI providers driven through a GitHub repository - For local te