Hi All, I am following up to close the loop on this. Apologies for the delay. We had to work through some technical and procedural issues before releasing the code.
Updates: 1. We decided to release the code under the BSD 3-Clause [1], rather than the BSD 2-Clause license. If there are any concerns about this licensing change, please let us know. Our understanding is that BSD 3-Clause license should still be compatible with the licensing of the upstream Arrow codebase. We apologize for the confusion. 2. Initial code has been released under the MathWorks GitHub organization in a repository named "libmexclass" [2]. We chose the name "libmexclass" because the project aims to enable users to implement MATLAB classes in terms of calls to corresponding C++ classes using MEX [3]. The code is under active development and is not yet ready for integration with the Arrow codebase. However, we wanted to get the code on GitHub as soon as possible so that anyone who is interested can feel free to follow development progress. We welcome any contributions from Arrow community members! Once the code has matured a bit more, we will work with the Arrow community to update the build infrastructure for the MATLAB Interface to Arrow to make use of libmexclass. Our hope is that using libmexclass will help unblock and streamline development efforts for the MATLAB interface. Thank you again to the community for providing helpful feedback and enabling us to move forward. [1] https://github.com/mathworks/libmexclass/blob/main/LICENSE [2] https://github.com/mathworks/libmexclass [3] https://www.mathworks.com/help/matlab/call-mex-files-1.html Best Regards, Kevin Gurney ________________________________ From: Sutou Kouhei <k...@clear-code.com> Sent: Sunday, June 12, 2022 10:12 PM To: dev@arrow.apache.org <dev@arrow.apache.org> Cc: Fiona La <fion...@mathworks.com>; Jeremy Hughes <jhug...@mathworks.com>; Nick Haddad <nhad...@mathworks.com>; Kevin Gurney <kgur...@mathworks.com> Subject: Re: [MATLAB] Integrating a framework for connecting MATLAB and C++ objects using MEX +1 In <mn2pr05mb6496c7164ff07547514ffaf7ae...@mn2pr05mb6496.namprd05.prod.outlook.com> "Re: [MATLAB] Integrating a framework for connecting MATLAB and C++ objects using MEX" on Fri, 10 Jun 2022 18:22:47 +0000, Kevin Gurney <kgur...@mathworks.com> wrote: > Hi Kou, > > Thank you for helping to clear up our confusion. > >> How do we install the object dispatch layer to use it in >> apache/arrow? I assumed that something like the following: >> >> ---- >> $ git clone >> https://github.com/mathworks/object-dispatch-layer.git<https://github.com/mathworks/object-dispatch-layer.git> >> $ cd object-dispatch-layer >> $ cmake -S . -B build ... >> $ cmake --build build >> $ cmake --install build >> $ git clone >> https://github.com/apache/arrow.git<https://github.com/apache/arrow.git> >> $ cd apache/matlab >> $ cmake -S . -B build # This find installed object-dispatch-layer >> $ cmake --build build >> $ cmake --install build >> ---- >> >> My assumption is right? > > Your understanding is correct. Thanks for checking. > >> BTW, why do you want to use "git submodule" to use the >> object dispatch layer? Why don't you install it separately >> or build by externalproject_add() in CMake? >> https://cmake.org/cmake/help/latest/module/ExternalProject.html<https://cmake.org/cmake/help/latest/module/ExternalProject.html><https://cmake.org/cmake/help/latest/module/ExternalProject.html<https://cmake.org/cmake/help/latest/module/ExternalProject.html>> > > After reflecting on your response, we realize that using a git submodule > seems like a less than ideal solution. > > Initially, we were thinking that if code within the apache/arrow repository > were to "depend" on some MATLAB files from the object dispatch layer, that we > would need to "physically" (via vendoring or IP Clearance) or "virtually" > (via git submodule) integrate this code into the apache/arrow source tree. > However, since these files are only needed at build time / run time, this > means that the object dispatch layer code does not necessarily need to be > redistributable along with the rest of the code in the apache/arrow > repository. > > It seems much clearer now that the object dispatch layer can be treated as a > "pure" external "library" dependency, and thus, the code should not need to > be present in the apache/arrow repository. Per your suggestion, at build / > install time, it should be possible to copy any required MATLAB files or C++ > header files to appropriate locations, so that they can be used by CMake and > MATLAB. > > Using externalproject_add() is a great idea and seems more "in-model" than > repeatedly bumping the version of a vendored copy of the object dispatch > layer source or using a git submodule. > > To summarize, it sounds like a reasonable path forward would be to: > > 1. Develop the object dispatch layer in an external repository underneath the > MathWorks GitHub organization, with a 2-Clause BSD license. > 2. Use externalproject_add() to fetch and build the source code dynamically. > > Once the object dispatch layer is available on GitHub, I will follow up on > this email thread with a link to the repository so that anyone in the > community can track development progress, as well as contribute to the > framework, if they are interested. > > If anyone has any objections to this approach, please let us know. > > Thank you! > > Kevin Gurney > > ________________________________ > From: Sutou Kouhei <k...@clear-code.com> > Sent: Friday, June 10, 2022 4:13 AM > To: dev@arrow.apache.org <dev@arrow.apache.org> > Cc: Kevin Gurney <kgur...@mathworks.com>; Fiona La <fion...@mathworks.com>; > Jeremy Hughes <jhug...@mathworks.com>; Nick Haddad <nhad...@mathworks.com> > Subject: Re: [MATLAB] Integrating a framework for connecting MATLAB and C++ > objects using MEX > > Hi, > >> 1. A developer would author a custom MEX function that >> uses C++ "building blocks" (i.e. classes and header files) >> from the object dispatch layer "framework". They would >> link their custom MEX function against a helper shared >> library that is built from the source code of the object >> dispatch layer and provides the symbols/implementation for >> the aforementioned C++ "building blocks". > > How do we install the object dispatch layer to use it in > apache/arrow? I assumed that something like the following: > > ---- > $ git clone > https://github.com/mathworks/object-dispatch-layer.git<https://github.com/mathworks/object-dispatch-layer.git><https://github.com/mathworks/object-dispatch-layer.git<https://github.com/mathworks/object-dispatch-layer.git>> > $ cd object-dispatch-layer > $ cmake -S . -B build ... > $ cmake --build build > $ cmake --install build > $ git clone > https://github.com/apache/arrow.git<https://github.com/apache/arrow.git><https://github.com/apache/arrow.git<https://github.com/apache/arrow.git>> > $ cd apache/matlab > $ cmake -S . -B build # This find installed object-dispatch-layer > $ cmake --build build > $ cmake --install build > ---- > > My assumption is right? > >> Essentially, for a developer to use the object dispatch >> layer, they will need to author a fair amount of custom >> code which makes use of both MATLAB and C++ "building >> blocks" from the "framework". > > I think that this is a normal library usage. For example, > our S3 filesystem module implementation in C++ has about > 2500 lines and uses classes provides by AWS SDK C++: > https://github.com/apache/arrow/blob/master/cpp/src/arrow/filesystem/s3fs.cc<https://github.com/apache/arrow/blob/master/cpp/src/arrow/filesystem/s3fs.cc><https://github.com/apache/arrow/blob/master/cpp/src/arrow/filesystem/s3fs.cc<https://github.com/apache/arrow/blob/master/cpp/src/arrow/filesystem/s3fs.cc>> > > >> If we had to go through the IP Clearance Process, would >> that mean we would need to repeatedly clear the code every >> time we wanted to sync up the git submodule with the >> latest source code from the external repository? It seems >> like this would quickly become impractical since we >> anticipate the need to iterate frequently on the object >> dispatch layer early on. > > If the object dispatch layer doesn't depend on Apache Arrow > and is a general purpose framework, we can vendor it without > IP clearance. > e.g.: > https://github.com/apache/arrow/tree/master/cpp/src/arrow/vendored/xxhash<https://github.com/apache/arrow/tree/master/cpp/src/arrow/vendored/xxhash><https://github.com/apache/arrow/tree/master/cpp/src/arrow/vendored/xxhash<https://github.com/apache/arrow/tree/master/cpp/src/arrow/vendored/xxhash>> > > BTW, why do you want to use "git submodule" to use the > object dispatch layer? Why don't you install it separately > or build by externalproject_add() in CMake? > https://cmake.org/cmake/help/latest/module/ExternalProject.html<https://cmake.org/cmake/help/latest/module/ExternalProject.html><https://cmake.org/cmake/help/latest/module/ExternalProject.html<https://cmake.org/cmake/help/latest/module/ExternalProject.html>> > > > Thanks, > -- > kou > > In > <byapr05mb648755bee2e0caccb0f407d9ae...@byapr05mb6487.namprd05.prod.outlook.com> > "Re: [MATLAB] Integrating a framework for connecting MATLAB and C++ objects > using MEX" on Wed, 8 Jun 2022 15:59:13 +0000, > Kevin Gurney <kgur...@mathworks.com> wrote: > >> Hi Kou, >> >> --- >> >> Note: I am replying to your email as a forward from Fiona (Cc'd) since your >> original email was accidentally blocked by my email client). >> >> --- >> >> The way that we expected the object dispatch layer to be used by client code >> is as follows: >> >> 1. A developer would author a custom MEX function that uses C++ "building >> blocks" (i.e. classes and header files) from the object dispatch layer >> "framework". They would link their custom MEX function against a helper >> shared library that is built from the source code of the object dispatch >> layer and provides the symbols/implementation for the aforementioned C++ >> "building blocks". >> >> 2. The object dispatch layer expects the compiled MEX function to have a >> specific name and be available on the MATLAB Search Path [1] so that it can >> be used by the MATLAB side of the object dispatch layer. >> >> 3. Once the MEX function is available on the MATLAB Search Path, client >> MATLAB code can use a set of MATLAB "building blocks" (i.e. classes), which >> are part of the object dispatch layer "framework", to connect a MATLAB class >> with a corresponding C++ class. >> >> Essentially, for a developer to use the object dispatch layer, they will >> need to author a fair amount of custom code which makes use of both MATLAB >> and C++ "building blocks" from the "framework". >> >> It's not clear to me whether the steps described above classify as "library >> usage" with regard to the IP Clearance Process. >> >> If we had to go through the IP Clearance Process, would that mean we would >> need to repeatedly clear the code every time we wanted to sync up the git >> submodule with the latest source code from the external repository? It seems >> like this would quickly become impractical since we anticipate the need to >> iterate frequently on the object dispatch layer early on. >> >> It's quite possible that I am not answering your questions completely, so >> please let me know if anything is unclear. My apologies in advance for any >> confusion. >> >> [1] >> https://www.mathworks.com/help/matlab/matlab_env/what-is-the-matlab-search-path.html >> >> Best, >> >> Kevin Gurney >> >> ________________________________ >> From: Fiona La <fion...@mathworks.com> >> Sent: Wednesday, June 8, 2022 11:24 AM >> To: Kevin Gurney <kgur...@mathworks.com> >> Subject: FW: [MATLAB] Integrating a framework for connecting MATLAB and C++ >> objects using MEX >> >> >> >> >> >> >> From: Sutou Kouhei <k...@clear-code.com> >> Date: Tuesday, June 7, 2022 at 8:36 PM >> To: dev@arrow.apache.org <dev@arrow.apache.org> >> Cc: Fiona La <fion...@mathworks.com>, Jeremy Hughes <jhug...@mathworks.com>, >> Nick Haddad <nhad...@mathworks.com> >> Subject: Re: [MATLAB] Integrating a framework for connecting MATLAB and C++ >> objects using MEX >> >> Hi, >> >> Can we use the object dispatch layer as a library? Or should >> we copy (or submodule) the object dispatch layer to >> apache/arrow? >> >> If we can use the object dispatch layer as a library, we can >> just use it as an external library like GoogleTest. We don't >> need IP clearance. You can use any Apache License 2.0 >> compatible license for the object dispatch layer. >> >> Thanks, >> -- >> kou >> >> In >> <mn2pr05mb6496cff053c60e54c133c93cae...@mn2pr05mb6496.namprd05.prod.outlook.com> >> "[MATLAB] Integrating a framework for connecting MATLAB and C++ objects >> using MEX" on Tue, 7 Jun 2022 18:10:43 +0000, >> Kevin Gurney <kgur...@mathworks.com> wrote: >> >>> Hi All, >>> >>> I am reaching out to seek guidance from the community regarding a code >>> integration puzzle. >>> >>> The architecture that we are currently pursuing for the MATLAB interface to >>> Arrow [1] involves dispatching to the Arrow C++ libraries using MEX (a >>> MATLAB facility for calling C/C++ code [2]). A major challenge with this >>> approach has been keeping Arrow C++ objects (e.g. arrow::Array) alive in >>> memory for the appropriate amount of time and making it easy to interface >>> with them from MATLAB. >>> >>> MATLAB has a recommended solution for this problem [3]. However, we've been >>> pursuing a MEX-based solution due to the pervasiveness of MEX and its >>> familiarity to MATLAB users. Our hope is that using MEX will make it easy >>> for others to contribute to the MATLAB interface. >>> >>> To help maintain the connection between MATLAB objects and C++, we've been >>> experimenting with a MEX-based object dispatch layer. The primary goal of >>> this work is to unblock development of the MATLAB interface to Arrow. >>> However, this object dispatch layer is non-trivial and ultimately unrelated >>> to the Arrow project's core mission. Therefore, submitting this code to the >>> Arrow project doesn't seem like the optimal code integration strategy. >>> >>> We’ve been considering the possibility of creating a new open-source >>> repository under the MathWorks GitHub organization [4] to host the object >>> dispatch layer (a side effect of this approach is that it may help >>> encourage reuse of this infrastructure in future open-source MATLAB >>> projects). >>> >>> However, this approach would come with notable tradeoffs: >>> >>> 1. We would need to follow the ASF IP Clearance Process [5] to integrate >>> this code into the Arrow project (it's possible we are mistaken about this). >>> >>> 2. It's not obvious how we should keep the code in sync. Would it be >>> possible to use a git submodule [6] to "symlink" to the external repo? >>> >>> 3. What about licensing? Does the code need to be Apache licensed, or would >>> it be possible to use another Apache-compatible license [7], like BSD? BSD >>> is the default choice for new projects hosted under the MathWorks GitHub >>> organization. >>> >>> Admittedly, we aren't sure what the best path forward is, so we appreciate >>> the community's guidance. We welcome any suggestions. >>> >>> [1] >>> https://github.com/apache/arrow/tree/master/matlab<https://github.com/apache/arrow/tree/master/matlab><https://github.com/apache/arrow/tree/master/matlab<https://github.com/apache/arrow/tree/master/matlab>><https://github.com/apache/arrow/tree/master/matlab<https://github.com/apache/arrow/tree/master/matlab><https://github.com/apache/arrow/tree/master/matlab<https://github.com/apache/arrow/tree/master/matlab>>> >>> [2] https://www.mathworks.com/help/matlab/call-mex-files-1.html >>> [3] >>> https://www.mathworks.com/help/matlab/build-matlab-interface-to-c-library.html >>> [4] >>> https://github.com/mathworks<https://github.com/mathworks><https://github.com/mathworks<https://github.com/mathworks>><https://github.com/mathworks<https://github.com/mathworks><https://github.com/mathworks<https://github.com/mathworks>>> >>> [5] >>> https://incubator.apache.org/ip-clearance/<https://incubator.apache.org/ip-clearance><https://incubator.apache.org/ip-clearance<https://incubator.apache.org/ip-clearance>><https://incubator.apache.org/ip-clearance<https://incubator.apache.org/ip-clearance><https://incubator.apache.org/ip-clearance<https://incubator.apache.org/ip-clearance>>> >>> [6] >>> https://github.blog/2016-02-01-working-with-submodules/<https://github.blog/2016-02-01-working-with-submodules><https://github.blog/2016-02-01-working-with-submodules<https://github.blog/2016-02-01-working-with-submodules>><https://github.blog/2016-02-01-working-with-submodules<https://github.blog/2016-02-01-working-with-submodules><https://github.blog/2016-02-01-working-with-submodules<https://github.blog/2016-02-01-working-with-submodules>>> >>> [7] >>> https://www.apache.org/legal/resolved.html#category-a<https://www.apache.org/legal/resolved.html#category-a><https://www.apache.org/legal/resolved.html#category-a<https://www.apache.org/legal/resolved.html#category-a>><https://www.apache.org/legal/resolved.html#category-a<https://www.apache.org/legal/resolved.html#category-a><https://www.apache.org/legal/resolved.html#category-a<https://www.apache.org/legal/resolved.html#category-a>>> >>> >>> Thank you, >>> >>> Kevin Gurney
