Hi all, Please see https://docs.google.com/document/d/1y25kRrXlORnUD9p7wTMOWjC6wONEyI9rU-Pv4q1udZ8/edit?usp=sharing for a copy of this email with proper formatting.
thanks. On Mon, May 2, 2022 at 4:23 PM Larry White <ljw1...@gmail.com> wrote: > Hi all, > > > I would like to request your feedback on incorporating Windows binaries in > those Maven packages that have native Arrow dependencies, while drawing > your attention to the likely impact on jar size. > > > Five of the 23 arrow packages on Maven Central have native dependencies. > Four of those five have bundled native libraries included in the maven > package jar itself. (The exception is the plasma package.) For the others, > both .so (Linux shared-object) and .dylib (OSX dynamic library) files are > provided in the same jar. Windows native libraries are not included. > > > The packages in question are: > > - > > arrow-dataset > > > - > > arrow-orc > - > > arrow-c > - > > Arrow-gandiva > > > For developers using Arrow on OSX or Linux, the experience using the > arrow-dataset jar with its bundled native library is the same as using a > pure Java library. Including Windows binaries in the jars would expand the > community of developers who could use Arrow features like datasets “out-of > the box.” > > > Moreover, it is not trivial for devs on Windows to create their own > solution. To the best of my knowledge, pre-compiled JNI DLLs are not > available for download, and there are no build scripts or instructions, > as there are for Linux and Mac users (see > https://arrow.apache.org/docs/dev/developers/java/building.html#building-arrow-jni-modules > ). > Effort > > To produce the JNI DLLs, the main effort will be to create new > Windows-focused build scripts similar to: *arrow > <https://github.com/apache/arrow>/ci > <https://github.com/apache/arrow/tree/master/ci>/scripts > <https://github.com/apache/arrow/tree/master/ci/scripts>/java_jni_macos_build.sh, > a*nd incorporate them into the larger build process. > > > Creating these build files is a prerequisite for the suggested packaging > changes, but is also desirable in its own right, even if the proposed > packaging change is not implemented. > File size concern > > The downside of including Windows binaries is that these files are large. > In the 7.0.0 release, the two native library files included in the dataset > jar total 78 MB on disk, which is roughly 100% of the total size of the > jar. See table below for more details. > > module > > .dylib (size in MB) > > .so (size in MB) > > Combined > > dataset > > 34.6 > > 43.7 > > 78.3 > > ORC > > 29.3 > > 37.9 > > 67.2 > > Gandiva > > 77.4 > > 87.1 > > 164.5 > > c-data > > <1.0 > > <1.0 > > <`1.0 > > Total > > 141.3 > > 167.7 > > > > It’s estimated that DLLs would be slightly larger than the dylib files, so > that the proposed change would increase the size of the dataset jar from > 78.3 MB to about 114 MB. > > For reference, here are the native Arrow libraries (.so) in a PyArrow > x86-64 wheel: > > Dataset > > 2.3 > > Flight > > 13.0 > > Python > > 2.1 > > Python-flight > > 0.1 > > Plasma > > 0.2 > > Parquet > > 4.3 > > Arrow > > 49.0 > > Total > > 71.0 > > Note that this isn't an apples-to-apples comparison: the PyArrow libraries > do not include Gandiva, while the Java libraries do not include Flight, > Plasma, Parque, or (presumably) some amount of the code in the Arrow file. > > As more C++ functionality is used by Java code the number of modules with > native dependencies may rise, and the size of the individual libraries may > increase. > > For the sake of simplicity, it is preferable to produce a single Jar for > each module that contains binaries for the three platforms: Windows, OSX, > and Linux. If file size is a significant concern, there are several options: > > > > - > > Stripping some symbols (`strip -x`) on the Linux dataset JNI library > brings it down from 43 to 34 MB, at the cost of debug information. It may > be worth considering this option for release builds. > - > > It may be possible to combine modules to reduce the amount of > duplicated code for projects that need more than one module with native > dependencies. > - > > OS-specific Maven packages could be built > > > Thank you for your feedback, > > larry >
