Ah yeah, that's correct. Arrow goes through significant effort to hide the generated files as much as it can to try to avoid conflicts if someone were to dynamically link both Arrow and something else that uses Substrait together. You also won't find the Substrait headers be exposed by the Arrow install for this reason. I don't think the current method solves all edge cases due to libprotobuf's descriptor table (in particular, if the two Substrait versions differ, libprotobuf may or may not invoke HALT_AND_CATCH_FIRE), but that is at least the intention.
On Mon, 18 Jul 2022 at 21:32, Li Jin <ice.xell...@gmail.com> wrote: > Thanks both. This appears to work! > > With regard to linking, I also have libsubstrait.a under build/debug, but > not in dist/lib - I suppose maybe the substrait classes are statically > linked into libarrow.so? > > Li > > On Mon, Jul 18, 2022 at 3:19 PM Jeroen van Straten < > jeroen.van.stra...@gmail.com> wrote: > > > I think the SHA in versions.txt is for the tarball specifically (prior to > > decompression), so I think it'll complain if you set it to a repository > > directory instead. > > > > Jeroen > > > > > > On Mon, 18 Jul 2022 at 20:40, Yaron Gvili <rt...@hotmail.com> wrote: > > > > > ARROW_SUBSTRAIT_URL works for both a *.tar.gz file and a repository > > > directory. In my experience, there is no need to also set any > > > sha256-related setting. > > > > > > > > > Yaron. > > > ________________________________ > > > From: Li Jin <ice.xell...@gmail.com> > > > Sent: Monday, July 18, 2022 2:34 PM > > > To: dev@arrow.apache.org <dev@arrow.apache.org> > > > Subject: Re: [C++] Question about substrait dependency in C++ > > > > > > Thanks both! Let me try changing ARROW_SUBSTRAIT_URL. Should I set > > > ARROW_SUBSTRAIT_URL just to local substrait tarball or sth else? > > > > > > On Mon, Jul 18, 2022 at 2:28 PM Yaron Gvili <rt...@hotmail.com> wrote: > > > > > > > Hi Li, > > > > > > > > I was just writing this. > > > > > > > > AFAIK, currently the Arrow C++ build system does not take prebuilt > > > > Substrait C++ classes. The usual way is rebuilding Arrow C++ with a > > > custom > > > > Substrait repository, which is done by setting ARROW_SUBSTRAIT_URL > to a > > > > local Substrait repository. You can download this repository offline > > and > > > > make it available to your internal build system. With this, the Arrow > > C++ > > > > build system auto-generates the Substrait C++ classes for the local > > > > Substrait repository and builds into > > "libarrow_bundled_dependencies.a". I > > > > wouldn't recommend doing these auto-generation and build steps on > your > > > own, > > > > as you'd have to change the Arrow C++ build system, and specifically > > > > "cpp/cmake_modules/ThirdpartyToolchain.cmake", to enable this. > > > > > > > > > > > > Yaron. > > > > ________________________________ > > > > From: Jeroen van Straten <jeroen.van.stra...@gmail.com> > > > > Sent: Monday, July 18, 2022 2:27 PM > > > > To: dev@arrow.apache.org <dev@arrow.apache.org> > > > > Subject: Re: [C++] Question about substrait dependency in C++ > > > > > > > > Hi, > > > > > > > > I'm not sure I completely understand what you're trying to do, but if > > > lack > > > > of internet access is the only problem, I think you should just be > able > > > to > > > > override the URL it tries to download by setting the > > ARROW_SUBSTRAIT_URL > > > > environment variable to some local file:// URL. I think it should > work > > as > > > > long as the hash matches what Arrow's build system expects from > > > > thirdparty/versions.txt. > > > > > > > > I'm not sure why you're not seeing the library. With `--preset > > > > ninja-debug`, I'm getting a libsubstrait.a in build/debug. I'm not > > > familiar > > > > enough with Arrow's build system to provide more help there. > > > > > > > > Regards, > > > > Jeroen > > > > > > > > On Mon, 18 Jul 2022 at 18:00, Li Jin <ice.xell...@gmail.com> wrote: > > > > > > > > > Hello! > > > > > > > > > > I am working on integrating the latest Arrow C++ into our internal > > > build > > > > > system. Currently I am planning to build substrait C++ classes > > > > > independently and provide header locations and so files to the > Arrow > > > > > Cmakefile - I wonder if that is a good approach? (We cannot > download > > > the > > > > > substrait tarball in the internal build system when building Arrow) > > > > > > > > > > Also, I don't see a libsubstrait.so or libsubstrait.a in the build > > > > > directory - is that statically linked? > > > > > > > > > > Thanks, > > > > > Li > > > > > > > > > > > > > > >
