Proposed report update below. LMK your thoughts.
## Description:
The mission of Apache Arrow is the creation and maintenance of software
related to columnar in-memory processing and data interchange
## Issues:
* We are struggling with Continuous Integration scalability as the project
has
definitely outgrown what Travis CI and Appveyor can do for us. Some
contributors have shown reluctance to submit patches they aren't sure
about
because they don't want to pile on the build queue. We are exploring
alternative solutions such as Buildbot, Buildkite, and GitHub Actions to
provide a path to migrate away from Travis CI / Appveyor. In our request
to
Infrastructure INFRA-19217, some of us were alarmed to find that an CI/CD
service like Buildkite may not be able to be connected to the @apache
GitHub
account on account of requiring admin access to repository webhooks, but
no
ability to modify source code. There are workarounds (building custom
OAuth
bots) that could enable us to use Buildkite, but it would require extra
development and result in a less refined experience for community members.
## Membership Data:
* Apache Arrow was founded 2016-01-19 (4 years ago)
* There are currently 48 committers and 28 PMC members in this project.
* The Committer-to-PMC ratio is roughly 3:2.
Community changes, past quarter:
- Micah Kornfield was added to the PMC on 2019-08-21
- Sebastien Binet was added to the PMC on 2019-08-21
- Ben Kietzman was added as committer on 2019-09-07
- David Li was added as committer on 2019-08-30
- Kenta Murata was added as committer on 2019-09-05
- Neal Richardson was added as committer on 2019-09-05
- Praveen Kumar was added as committer on 2019-07-14
## Project Activity:
* The project has just made a 0.15.0 release.
* We are discussing ways to make the Arrow libraries as accessible as
possible
to downstream projects for minimal use cases while allowing the
development
of more comprehensive "standard libraries" with larger dependency stacks
in
the project
* We plan to make a 1.0.0 release as our next major release, at which time
we
will declare that the Arrow binary protocol is stable with forward and
backward compatibility guarantees
## Community Health:
* The community is continuing to grow at a great rate. We see good adoption
among many other projects and fast growth of key metrics.
* Many contributors are struggling with the slowness of pre-commit CI. Arrow
has a large number of different platforms and components and a complex
build
matrix. As new commits come in, they frequently take a long time to
complete. The community is trying several ways to solve this. There is
bubbling frustration in the community around the GitHub repo rules for
using
third party services. This is especially challenging when there are free
solutions to relieve the community pressure but the community is unable to
access these resources. This frustration is greatest among people who work
on many non-asf OSS projects which don't have such restrictive rules
around GitHub. Some examples of ways the community has tried to resolve
these have included:
* Try to use CircleCI, rejected in INFRA-15964
* Try to use Azure Pipelines, rejected in INFRA-17030
* Try to resolves Issues with Travis CI capacity: INFRA-18533 &
https://s.apache.org/ci-capacity (no resolution beyond "find donations")
* The creation of new infrastructure design (in progress but a huge
amount of
thankless work)
* While the community has seen great growth in contribution (more than 300
unique contributors at this point), the vast majority are casual
contributors. The daily active committers (the workhorses of the project
that bear the load committing the constant PRs, more than 5000 closed at
this point) have been growing slower than adoption. This is despite the
fact
that the community has been very aggressive at being inclusive of new
committers (with likelihood to have more than 50 in the next week). The
community is still continuing to try to brainstorm ways to improve this.
