hi Jacques, I think we need to share the concerns that many PMC members have over the constraints that INFRA is placing on us. Can we rephrase the concern in a way that is more helpful?
Firstly, I respect and appreciate the ASF's desire to limit write access to committers only from an IP provenance perspective. I understand that GitHub webhooks are used to log actions taken in repositories to secure IP provenance. I do not think a third party application should be given the ability to commit or modify a repository -- all write operations on the .git repository should be initiated by committers. However, GitHub is the main platform for producing open source software, and tools are being created to help produce open source more efficiently. It is frustrating for us to not be able to take advantage of the tools that are available to everyone else on GitHub. I brought up the recent request about Buildkite as being representative of this (after learning that Google has been making a lot of use of it), but we have previously been denied use of CircleCI and Azure Pipelines since those services require even more permissions (AFAIK) than in the case of Buildkite. From our use in https://github.com/ursa-labs/crossbow CircleCI and Azure seem to be a lot better than Travis CI and Appveyor I think the ASF is going to face an existential crisis in the near future whether it wants to live in 2020 or 2000. It feels like GitHub is treated somewhat as ersatz SVN "because people want to use git + GitHub instead of SVN" In the same way that the cloud revolutionized software startups, enabling small groups of developers to build large SaaS applications, the same kind of leverage is becoming available to open source developers to set up infrastructure to automate and scale open source projects. I think projects considering joining the Foundation are going to look at these issues around App usage and decide that they would rather be in control of their own infrastructure. I can set aside even more time and money from my non-profit organization's modest budget to do CI work for Apache Arrow. The amount that we have invested already is very large, and continues to grow. I'm raising these issues because as Member of the Foundation I'm concerned that fast-growing projects like ours are not being adequately served by INFRA, and we probably aren't the only project that will face these issues. All that is needed is for INFRA to let us use third party GitHub Apps and monitor any potentially destructive actions that they may take, such as modifying unrelated repository webhooks related to IP provenance. - Wes On Wed, Oct 9, 2019 at 9:33 PM Jacques Nadeau <jacq...@apache.org> wrote: > > I think we need to more direct in listing issues for the board. > > What have we done? What do we want them to do? > > In general, any large org is going to be slow to add new deep integrations > into GitHub. I don't think we should expect Apache to be any different (it > took several years before we could merge things through github for > example). If I were on the INFRA side, I think I would look and see how > many different people are asking for BuildKite before considering > integration. It seems like we only opened the JIRA 6 days ago and no other > projects have requested access to this? > > I'm not clear why this is a board issue. What do we think the board can do > for us that we can't solve ourselves and need them to solve? Remember, a > board solution to a problem is typically very removed from what matters to > individuals on a project. > > > > > > > On Tue, Oct 8, 2019 at 7:03 AM Wes McKinney <wesmck...@gmail.com> wrote: > > > New draft > > > > ## Description: > > The mission of Apache Arrow is the creation and maintenance of software > > related > > to columnar in-memory processing and data interchange > > > > ## Issues: > > > > * We are struggling with Continuous Integration scalability as the project > > has > > definitely outgrown what Travis CI and Appveyor can do for us. Some > > contributors have shown reluctance to submit patches they aren't sure > > about > > because they don't want to pile on the build queue. We are exploring > > alternative solutions such as Buildbot, Buildkite, and GitHub Actions to > > provide a path to migrate away from Travis CI / Appveyor. In our request > > to > > Infrastructure INFRA-19217, some of us were alarmed to find that an CI/CD > > service like Buildkite may not be able to be connected to the @apache > > GitHub > > account on account of requiring admin access to repository webhooks, but > > no > > ability to modify source code. There are workarounds (building custom > > OAuth > > bots) that could enable us to use Buildkite, but it would require extra > > development and result in a less refined experience for community > > members. > > > > ## Membership Data: > > * Apache Arrow was founded 2016-01-19 (4 years ago) > > * There are currently 48 committers and 28 PMC members in this project. > > * The Committer-to-PMC ratio is roughly 3:2. > > > > Community changes, past quarter: > > - Micah Kornfield was added to the PMC on 2019-08-21 > > - Sebastien Binet was added to the PMC on 2019-08-21 > > - Ben Kietzman was added as committer on 2019-09-07 > > - David Li was added as committer on 2019-08-30 > > - Kenta Murata was added as committer on 2019-09-05 > > - Neal Richardson was added as committer on 2019-09-05 > > - Praveen Kumar was added as committer on 2019-07-14 > > > > ## Project Activity: > > > > * The project has just made a 0.15.0 release. > > * We are discussing ways to make the Arrow libraries as accessible as > > possible > > to downstream projects for minimal use cases while allowing the > > development > > of more comprehensive "standard libraries" with larger dependency stacks > > in > > the project > > * We plan to make a 1.0.0 release as our next major release, at which time > > we > > will declare that the Arrow binary protocol is stable with forward and > > backward compatibility guarantees > > > > ## Community Health: > > > > * The community is overall healthy, with the aforementioned concerns > > around CI > > scalability. New contributors frequently take notice of the long build > > queue > > times when submitting pull requests. > > > > On Tue, Oct 8, 2019 at 8:58 AM Wes McKinney <wesmck...@gmail.com> wrote: > > > > > > Yes, I agree with raising the issue to the board. > > > > > > On Tue, Oct 8, 2019 at 8:31 AM Antoine Pitrou <anto...@python.org> > > wrote: > > > > > > > > > > > > I agree. Especially given that the constraints imposed by Infra don't > > > > help solving the problem. > > > > > > > > Regards > > > > > > > > Antoine. > > > > > > > > > > > > Le 08/10/2019 à 15:02, Uwe L. Korn a écrit : > > > > > I'm not sure what qualifies for "board attention" but it seems that > > CI is a critical problem in Apache projects, not just Arrow. Should we > > raise that? > > > > > > > > > > Uwe > > > > > > > > > > On Tue, Oct 8, 2019, at 12:00 AM, Wes McKinney wrote: > > > > >> Here is a start for our Q3 board report > > > > >> > > > > >> ## Description: > > > > >> The mission of Apache Arrow is the creation and maintenance of > > software related > > > > >> to columnar in-memory processing and data interchange > > > > >> > > > > >> ## Issues: > > > > >> There are no issues requiring board attention at this time > > > > >> > > > > >> ## Membership Data: > > > > >> * Apache Arrow was founded 2016-01-19 (4 years ago) > > > > >> * There are currently 48 committers and 28 PMC members in this > > project. > > > > >> * The Committer-to-PMC ratio is roughly 3:2. > > > > >> > > > > >> Community changes, past quarter: > > > > >> - Micah Kornfield was added to the PMC on 2019-08-21 > > > > >> - Sebastien Binet was added to the PMC on 2019-08-21 > > > > >> - Ben Kietzman was added as committer on 2019-09-07 > > > > >> - David Li was added as committer on 2019-08-30 > > > > >> - Kenta Murata was added as committer on 2019-09-05 > > > > >> - Neal Richardson was added as committer on 2019-09-05 > > > > >> - Praveen Kumar was added as committer on 2019-07-14 > > > > >> > > > > >> ## Project Activity: > > > > >> > > > > >> * The project has just made a 0.15.0 release. > > > > >> * We are discussing ways to make the Arrow libraries as accessible > > as possible > > > > >> to downstream projects for minimal use cases while allowing the > > development > > > > >> of more comprehensive "standard libraries" with larger dependency > > stacks in > > > > >> the project > > > > >> * We plan to make a 1.0.0 release as our next major release, at > > which time we > > > > >> will declare that the Arrow binary protocol is stable with > > forward and > > > > >> backward compatibility guarantees > > > > >> * We are struggling with Continuous Integration scalability as the > > project has > > > > >> definitely outgrown what Travis CI and Appveyor can do for us. We > > are > > > > >> exploring alternative solutions such as Buildbot, Buildkite (see > > > > >> INFRA-19217), and GitHub Actions to provide a path to migrate > > away from > > > > >> Travis CI / Appveyor > > > > >> > > > > >> ## Community Health: > > > > >> > > > > >> * The community is overall healthy, with the aforementioned > > concerns around CI > > > > >> scalability. New contributors frequently take notice of the long > > build queue > > > > >> times when submitting pull requests. > > > > >> > >
