Perhaps meson is also worth exploring?
Le 15/10/2019 à 23:06, Micah Kornfield a écrit :
Hi Wes,
I agree on both accounts that it won't be a done in the short term, and it
makes sense to tackle in incrementally. Like I said I don't have much
bandwidth at the moment but might be able to re-arrange a few things on my
plate. I think some people have asked on the mailing list how they might
be able to help, this might be one area that doesn't require a lot of
in-depth knowledge of C++ at least for a proof of concept. I'll try to
open up some JIRAs soon.
Thanks,
Micah
On Tue, Oct 15, 2019 at 10:33 AM Wes McKinney <wesmck...@gmail.com> wrote:
hi Micah,
Definitely Bazel is worth exploring, but we must be realistic about
the amount of energy (several hundred hours or more) that's been
invested in the build system we have now. So a new build system will
be a large endeavor, but hopefully can make things simpler.
Aside from the requirements gathering process, if it is felt that
Bazel is a possible path forward in the future, it may be good to try
to break up the work into more tractable pieces. For example, a first
step would be to set up Bazel configurations to build the project's
thirdparty toolchain. Since we're reliant in ExternalProject in CMake
to do a lot of heavy lifting there for us, I imagine this (taking care
of what ThirdpartyToolchain.cmake does not) will take up a lot of the
energy
- Wes
On Sun, Oct 13, 2019 at 1:06 PM Micah Kornfield <emkornfi...@gmail.com>
wrote:
This might be taking the thread on more of a tangent, but maybe we
should
start collecting requirements for the C++ build system in general and see
if there might be better solution that can address some of these
concerns?
In particular, Bazel at least on the surface seems like it might be a
better fit for some of the use cases discussed here. I know this is a
big
project (and I currently don't have much bandwidth for it) but I think if
CMake is lacking in these areas it might be worth at least exploring
instead of going down the path of building our own meta-build system on
top
of CMake.
Requirements that I think we are targeting:
1. Be able to provide an out of box build system that requires as close
to
zero dependencies beyond a standard C++ toolchain (e.g. "$BUILD minimal"
works on any C++ developers desktop without additional requirements)
2. The build system should limit configuration knobs in favor of implied
dependencies (e.g. "$BUILD python" automatically builds "compute",
"filesystem", "ipc")
3. The build system should be configurable to use (and have the user
specify) one of "System packages", "Conda packages" or source packages
for
providing dependencies (and fallback options between the three).
4. The build system should be able to treat some dependencies as
optional
(e.g. different compression libraries or allocators).
5. Easily allow developers to limit building unnecessary code for their
particular task at hand.
6. The build system must work across the following toolchains/platforms:
- Linux: g++ and clang. x86 and ARM
- Mac
- Windows (msys2 and MSVC)
Thanks,
Micah
On Thu, Oct 10, 2019 at 6:09 AM Antoine Pitrou <anto...@python.org>
wrote:
Yes, we could express dependencies in a Python script and have it
generate a CMake module of if/else chains in cmake_modules (which we
would check in git to avoid having people depend on a Python install,
perhaps).
Still, that is an additional maintenance burden.
Regards
Antoine.
Le 10/10/2019 à 14:50, Wes McKinney a écrit :
I guess one question we should first discuss is: who is the C++ build
system for?
The users who are most sensitive to benchmark-driven decision making
will generally be consuming the project through pre-built binaries,
like our Python or R packages. If C++ developers build the project
from source and don't do a minimal read of the documentation to see
what a "recommended configuration" looks like, I would say that is
more their fault than ours. In the case of the ARROW_JEMALLOC option,
I think it's important for C++ system integrators to be aware of the
impact of the choice of memory allocator.
The concern I have with the current "out of the box" experience is
that people are getting the impression that "I have to build $X, $Y,
and $Z -- which I don't necessarily need -- to have $CORE_FEATURE_1".
They can, of course, read the documentation and learn that those
things can be toggled off, but I think the user that reaches for a
self-built source install is much different in general than someone
who uses the project through the Linux binary packages, for example.
On the subject of managing intraproject dependencies and
relationships, I think we should develop a better way to express
relationships between components than we have now.
As an example, building the Python library assumes that various
components are enabled
- ARROW_COMPUTE=ON
- ARROW_FILESYSTEM=ON
- ARROW_IPC=ON
Somewhere in the code we might have some code like
if (ARROW_PYTHON)
set(ARROW_COMPUTE ON)
...
endif()
This doesn't strike me as that scalable. I would rather see a
dependency file like
component_dependencies = {
...
'python': ['compute', 'filesystem', 'ipc'],
...
}
A helper Python script as part of the build could be used to give
CMake (because CMake is a bit poor as a programming language) the
list
of required components based on what the user has indicated to CMake.
On Thu, Oct 10, 2019 at 7:36 AM Francois Saint-Jacques
<fsaintjacq...@gmail.com> wrote:
There's always the route of vendoring some library and not exposing
external CMake options. This would achieve the goal of
compile-out-of-the-box and enable important feature in the basic
build. We also simplify dependencies requirements (benefits CI or
developer). The downside is following security patches and grumpy
reaction from package maintainers. I think we should explore this
route for dependencies that match the following criteria:
- libarrow*.so don't export any of the symbols of the dependency and
not referenced in any public headers
- dependency is lightweight, e.g. excludes boost, openssl, grpc,
llvm,
thrift, protobuf
- dependency is not-ubiquitous on major platform and have a stable
API, e.g. excludes libz and openssl
A small list of candidates:
- RapidJSON (enables JSON)
- DoubleConversion (enables CSV)
There's a precedent, arrow already vendors small C++ libraries
(datetime, utf8cpp, variant, xxhash).
François
On Thu, Oct 10, 2019 at 6:03 AM Antoine Pitrou <anto...@python.org>
wrote:
Hi all,
I'm a bit concerned that we're planning to add many additional
build
options in the quest to have a core zero-dependency build in C++.
See for example https://issues.apache.org/jira/browse/ARROW-6633
or
https://issues.apache.org/jira/browse/ARROW-6612.
The problem is that this is creating many possible configurations
and
we
will only be testing a tiny subset of them. Inevitably, users
will try
other option combinations and they'll fail building for some random
reason. It will not be a very good user experience.
Another related issue is user perception when doing a default
build.
For example https://issues.apache.org/jira/browse/ARROW-6638
proposes
to
build with jemalloc disabled by default. Inevitably, people will
be
doing benchmarks with this (publicly or not) and they'll conclude
Arrow
is not as performant as it claims to be.
Perhaps we should look for another approach instead?
For example we could have a single ARROW_BARE_CORE (whatever the
name)
option that when enabled (not by default) builds the tiniest
minimal
subset of Arrow. It's more inflexible, but at least it's something
that
we can reasonably test.
Regards
Antoine.
