hi folks, We now have patches up for Java, JS, and Go. How are we doing on the code reviews for getting these in?
Since C# implements the binary protocol, the C# developers might want to look at this before the 0.15.0 release also. Absent integration tests it's difficult to verify the C# library, though Thanks On Thu, Aug 29, 2019 at 8:13 AM Ji Liu <niki...@aliyun.com> wrote: > > Here is the Java implementation > https://github.com/apache/arrow/pull/5229 > > cc @Wes McKinney @emkornfield > > Thanks, > Ji Liu > > ------------------------------------------------------------------ > From:Ji Liu <niki...@aliyun.com.INVALID> > Send Time:2019年8月28日(星期三) 17:34 > To:emkornfield <emkornfi...@gmail.com>; dev <dev@arrow.apache.org> > Cc:Paul Taylor <ptay...@apache.org> > Subject:Re: [RESULT] [VOTE] Alter Arrow binary protocol to address 8-byte > Flatbuffer alignment requirements (2nd vote) > > I could take the Java implementation and will take a close watch on this > issue in the next few days. > > Thanks, > Ji Liu > > > ------------------------------------------------------------------ > From:Micah Kornfield <emkornfi...@gmail.com> > Send Time:2019年8月28日(星期三) 17:14 > To:dev <dev@arrow.apache.org> > Cc:Paul Taylor <ptay...@apache.org> > Subject:Re: [RESULT] [VOTE] Alter Arrow binary protocol to address 8-byte > Flatbuffer alignment requirements (2nd vote) > > I should have integration tests with 0.14.1 generated binaries in the next > few days. I think the one remaining unassigned piece of work in the Java > implementation, i can take that up next if no one else gets to it. > > On Tue, Aug 27, 2019 at 7:19 PM Wes McKinney <wesmck...@gmail.com> wrote: > > > Here's the C++ changes > > > > https://github.com/apache/arrow/pull/5211 > > > > I'm going to create a integration branch where we can merge each patch > > before merging to master > > > > On Fri, Aug 23, 2019 at 9:03 AM Wes McKinney <wesmck...@gmail.com> wrote: > > > > > > It isn't implemented in C++ yet but I will try to get a patch up for > > > that soon (today maybe). I think we should create a branch where we > > > can stack the patches that implement this for each language. > > > > > > On Fri, Aug 23, 2019 at 4:04 AM Paul Taylor <ptaylor.apa...@gmail.com> > > wrote: > > > > > > > > I'll do the JS updates. Is it safe to validate against the Arrow C++ > > > > integration tests? > > > > > > > > > > > > On 8/22/19 7:28 PM, Micah Kornfield wrote: > > > > > I created https://issues.apache.org/jira/browse/ARROW-6313 as a > > tracking > > > > > issue with sub-issues on the development work. So far no-one has > > claimed > > > > > Java and Javascript tasks. > > > > > > > > > > Would it make sense to have a separate dev branch for this work? > > > > > > > > > > Thanks, > > > > > Micah > > > > > > > > > > On Thu, Aug 22, 2019 at 3:24 PM Wes McKinney <wesmck...@gmail.com> > > wrote: > > > > > > > > > >> The vote carries with 4 binding +1 votes and 1 non-binding +1 > > > > >> > > > > >> I'll merge the specification patch later today and we can begin > > > > >> working on implementations so we can get this done for 0.15.0 > > > > >> > > > > >> On Tue, Aug 20, 2019 at 12:30 PM Bryan Cutler <cutl...@gmail.com> > > wrote: > > > > >>> +1 (non-binding) > > > > >>> > > > > >>> On Tue, Aug 20, 2019, 7:43 AM Antoine Pitrou <solip...@pitrou.net> > > > > >> wrote: > > > > >>>> Sorry, had forgotten to send my vote on this. > > > > >>>> > > > > >>>> +1 from me. > > > > >>>> > > > > >>>> Regards > > > > >>>> > > > > >>>> Antoine. > > > > >>>> > > > > >>>> > > > > >>>> On Wed, 14 Aug 2019 17:42:33 -0500 > > > > >>>> Wes McKinney <wesmck...@gmail.com> wrote: > > > > >>>>> hi all, > > > > >>>>> > > > > >>>>> As we've been discussing [1], there is a need to introduce 4 > > bytes of > > > > >>>>> padding into the preamble of the "encapsulated IPC message" > > format to > > > > >>>>> ensure that the Flatbuffers metadata payload begins on an 8-byte > > > > >>>>> aligned memory offset. The alternative to this would be for Arrow > > > > >>>>> implementations where alignment is important (e.g. C or C++) to > > copy > > > > >>>>> the metadata (which is not always small) into memory when it is > > > > >>>>> unaligned. > > > > >>>>> > > > > >>>>> Micah has proposed to address this by adding a > > > > >>>>> 4-byte "continuation" value at the beginning of the payload > > > > >>>>> having the value 0xFFFFFFFF. The reason to do it this way is that > > > > >>>>> old clients will see an invalid length (what is currently the > > > > >>>>> first 4 bytes of the message -- a 32-bit little endian signed > > > > >>>>> integer indicating the metadata length) rather than potentially > > > > >>>>> crashing on a valid length. We also propose to expand the "end of > > > > >>>>> stream" marker used in the stream and file format from 4 to 8 > > > > >>>>> bytes. This has the additional effect of aligning the file footer > > > > >>>>> defined in File.fbs. > > > > >>>>> > > > > >>>>> This would be a backwards incompatible protocol change, so older > > > > >> Arrow > > > > >>>>> libraries would not be able to read these new messages. > > Maintaining > > > > >>>>> forward compatibility (reading data produced by older libraries) > > > > >> would > > > > >>>>> be possible as we can reason that a value other than the > > continuation > > > > >>>>> value was produced by an older library (and then validate the > > > > >>>>> Flatbuffer message of course). Arrow implementations could offer > > a > > > > >>>>> backward compatibility mode for the sake of old readers if they > > > > >> desire > > > > >>>>> (this may also assist with testing). > > > > >>>>> > > > > >>>>> Additionally with this vote, we want to formally approve the > > change > > > > >> to > > > > >>>>> the Arrow "file" format to always write the (new 8-byte) > > > > >> end-of-stream > > > > >>>>> marker, which enables code that processes Arrow streams to safely > > > > >> read > > > > >>>>> the file's internal messages as though they were a normal stream. > > > > >>>>> > > > > >>>>> The PR making these changes to the IPC documentation is here > > > > >>>>> > > > > >>>>> https://github.com/apache/arrow/pull/4951 > > > > >>>>> > > > > >>>>> Please vote to accept these changes. This vote will be open for > > at > > > > >>>>> least 72 hours > > > > >>>>> > > > > >>>>> [ ] +1 Adopt these Arrow protocol changes > > > > >>>>> [ ] +0 > > > > >>>>> [ ] -1 I disagree because... > > > > >>>>> > > > > >>>>> Here is my vote: +1 > > > > >>>>> > > > > >>>>> Thanks, > > > > >>>>> Wes > > > > >>>>> > > > > >>>>> [1]: > > > > >> > > https://lists.apache.org/thread.html/8440be572c49b7b2ffb76b63e6d935ada9efd9c1c2021369b6d27786@%3Cdev.arrow.apache.org%3E > > > > >>>> > > > > >>>> > > > > >>>> > >
