Since we are at the "chicken" stage of the chicken-and-egg problem I don't have straightforward guidance about how to proceed, other than to dig in to either the current Java and / or C++ codebases and helping sort out what needs to be done. It may be beneficial on the mailing list to discuss the incremental steps required to reach working integration tests (I suspect there will be many JIRAs / patches required to get there) -- defining these tasks (perhaps in a shared Google document) and creating the associated JIRAs is a valuable and necessary exercise.
(Personally I've been most interested in "up stack" integration with other projects like Apache Parquet as it relates to native code consumers (e.g. Python libraries).) On the IPC side, you can look at the internal IPC round trip C++ tests: https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-adapter-test.cc On the Java side, an initial task would be to create a testing setup that generates sample data and either sends and receives it through a socket or memory map. There are other questions to analyze: - Schema negotiation - Metadata read / write (see https://github.com/apache/arrow/blob/master/format/Message.fbs) As discussed there are some inconsistencies between the reference implementations that we will need to resolve before this work can proceed to completion. The metadata (schemas and logical types, e.g. what is in Message.fbs) are also in flux and will require a round of iteration. Thanks, Wes On Thu, Jun 9, 2016 at 8:32 AM, Nicole Nemer <nicole.ne...@rms.com> wrote: > Hi Wes. > Would love to help. Just point me the tests that need to be > expanded/written and I will work on that today/tomorrow. > Thanks! > nn > — > Nicole Nemer, PhD > Technical Architect/Dev Manager > > 303-641-3340 > > > > > > > On 6/8/16, 5:47 PM, "Wes McKinney" <wesmck...@gmail.com> wrote: > >>hi Nicki >> >>Micah's patch for #1 is in progress here >>https://github.com/apache/arrow/pull/85 >> >>I believe Steven Phillips is working on a patch toward reconciling the >>Java implementation with the current working version of the spec. We >>need to be able to verify that memory can be passed between Java and >>C++ with full fidelity (using files / memory maps as the exchange >>medium to start); these integration tests will help enable other Arrow >>implementations validate their compatibility as well. It would be >>great to have some additional help here >> >>cheers >>Wes >> >>On Thu, Jun 2, 2016 at 7:10 AM, Nicole Nemer <nicole.ne...@rms.com> wrote: >>> Good Morning Micah, >>> How is 1 please? anything that I can do to help? >>> >>> Anyone with more insight on 2 please? >>> >>> Thanks, >>> nicki >>> ‹ >>> Nicole Nemer, PhD >>> Technical Architect/Dev Manager >>> >>> 303-641-3340 >>> >>> >>> >>> >>> >>> >>> On 5/27/16, 9:51 AM, "Micah Kornfield" <emkornfi...@gmail.com> wrote: >>> >>>>Hi Nicki, >>>>1. I'm currently working on the char/string support for C++. I've >>>>been a little bit backlogged on it. If I don't make substantial >>>>progress this weekend, I'm happy to relinquish the task. >>>> >>>>2. I'll let someone more knowledgable about the java implementation >>>>chime in, but I think the answer is a qualified yes. We were just >>>>talking about trying to make the first integration test that proves >>>>C++/Java compatibility [1] >>>> >>>>3. Yes it is easy to become a contributor. The general workflow is >>>>to chime in on jira item [2] (or someone on the PMC? can make you a >>>>contributor so you can assign a ticket to yourself), and submit a pull >>>>request via github with "ARROW-<JIRA-NUMBER>:" as the start of the >>>>pull request title. In addition to the items mentioned below there is >>>>a pretty substantial backlog of items to work on if you are interested >>>>in contributing generally. >>>> >>>>Thanks, >>>>Micah >>>> >>>>[1] >>>>http://mail-archives.apache.org/mod_mbox/arrow-dev/201605.mbox/%3CCAK7Z5 >>>>T8 >>>>X2OiWfSoQ0S-3vu0D4zgkuAO-SD_Q%3DF2Pu%3D4GhaTFbQ%40mail.gmail.com%3E >>>>[2] >>>>https://issues.apache.org/jira/browse/ARROW/?selectedTab=com.atlassian.j >>>>ir >>>>a.jira-projects-plugin:issues-panel >>>> >>>> >>>> >>>>On Fri, May 27, 2016 at 7:35 AM, Nicole Nemer <nicole.ne...@rms.com> >>>>wrote: >>>>> >>>>> 1. does cpp/ipc support char/string types? If not - when please? >>>>> 2. Is there a java implementation of the ipc feature please? If >>>>>not >>>>>- when please? >>>>> 3. Is it easy to join and help as a contributor? I would love to >>>>>help with these 2 items if they are planned for the near future. >>>>> >>>>> Thanks, >>>>> Nicki >>>>> - >>>>> Nicole Nemer, PhD >>>>> Technical Architect/Dev Manager >>>>> >>>>> >>>>> >>> >
