I commented on the JIRA that Wes opened: https://issues.apache.org/jira/browse/ARROW-245 We could just add a field in the metadata and fail on reception of a message if the endianness does not match the platform receiving it.
On Mon, Aug 1, 2016 at 11:16 AM, Sanjay Rao <getsanjay...@live.com> wrote: > Hi Wes, Hi Micah, > I understood what you meant, so point 2. Arrow working with Big Endian > machine to Big Endian shouldn't be an issue right ? > Please confirm. > Thanks,Sanjay > > From: wesmck...@gmail.com > > Date: Mon, 1 Aug 2016 11:07:07 -0700 > > Subject: Re: Is there plan to support BigEndian Systems like SUN SPARC > Hardware ? > > To: dev@arrow.apache.org; emkornfi...@gmail.com > > > > hey Micah, > > > > On Mon, Aug 1, 2016 at 11:02 AM, Micah Kornfield <emkornfi...@gmail.com> > wrote: > > > Hi Wes, > > > The point I was trying to argue from an earlier thread is that the most > > > common cases for relocation are: > > > 1. Little endian machine to little endian machine (most likely same > > > machine) > > > 2. big endian machine to big endian machine (most likely same machine) > > > 3. big endian machine to little endian machine or vice versa > > > > > > The purpose of the metadata would be to make use-cases 1 and 2 possible > > > without byte-swapping. Use case 3 would obviously require byte > swapping > > > but for an initial implementation the code could simply indicate that > it is > > > not supported. > > > > > > This seems less complex to me than actually implementing any sort of > > > byte-swapping logic while still supporting the widest variety of > hardware > > > with the same code for the most common use-cases. > > > > This makes sense. My comments were for the situation that a big endian > > system would be exposing memory to an unknown consumer -- for example, > > if we implemented an RPC wire format for Arrow memory, then in general > > a big endian system would need to send little-endian integers to an > > arbitrary receiver. I'm not sure the best way to provide for easy > > native-endianness support for cases 1/2, but trying to fully solve > > this problem now seems premature until we've established some of these > > tools (so long as we haven't painted ourselves into a corner). > > > > - Wes > > > > > > > > Thanks, > > > Micah > > > > > > P.S. If anybody can provide pointers I'd be interested to understand > which > > > pieces of the java code make assumptions about little-endianness. > > -- Julien
