Seems like valid use cases. I created ARROW-1006 and ARROW-1007 to build support for appends. If you are have other problems with the reader can you open a JIRA or post them here? Thanks
On Thu, May 11, 2017 at 11:28 AM, Katelman, Michael <michael.katel...@cubistsystematic.com> wrote: > I do see a streaming reader and writer, which might what I need. Although, > I'm having some minor problems with the reader. > > -----Original Message----- > From: Katelman, Michael [mailto:michael.katel...@cubistsystematic.com] > Sent: Thursday, May 11, 2017 9:37 > To: dev@arrow.apache.org > Subject: RE: list scope question > > Thanks, Wes. > > I was able to get the setup you outlined working as long as I explicitly > communicate the location of the footer to processes wanting to read the table > (I'm not sure if there's a better way of doing this). One other thing, > though, that I would ultimately like to do is share a table that is > conceptually growing. Is that something that arrow is intended for? > > I see in the code that a lot of the table-related data structures are > immutable and that the footer is written only on close, so perhaps not. But, > any thoughts on a use case like that would be appreciated. > > -Mike > > -----Original Message----- > From: Wes McKinney [mailto:wesmck...@gmail.com] > Sent: Wednesday, May 10, 2017 22:11 > To: dev@arrow.apache.org > Subject: Re: list scope question > > hi Mike, > > I recommend using record batches along with io::MemoryMappedFile. You can > write the table with ipc::FileWriter: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_arrow_blob_master_cpp_src_arrow_ipc_writer.h-23L111&d=DwIFaQ&c=f5Q7ov8zryUUIGT55zpGgw&r=p7uiAfJkXEwbVhZPqB-VxtsgxuGNpO5tGgnMUX3wqrPAIvdxhcKmn9kvZiXDziBQ&m=pWtlkbDLKO7PWTrI6MPiTzAuUgTWYLSRraZBy7KPCYA&s=Ud95KVEwn_XX3LwfEKTLH7nXvKGmOLhYnUu09ZrLyOg&e= > > and then read it with ipc::FileReader. If you use MemoryMappedFile, then the > read will be zero-copy (no memory allocated), so ideal for a multiple-process > shared memory setting. You can use one large record batch or multiple smaller > record batches. Let us know if you run into issues. > > Thanks > Wes > > On Wed, May 10, 2017 at 7:06 PM, Katelman, Michael > <michael.katel...@cubistsystematic.com> wrote: >> Great! >> >> What I'd like to do is create a table that can be shared among multiple >> processes. I see, e.g., this in the comments where RecordBatch is defined: >> >> // A record batch is a simpler and more rigid table data structure >> intended for // use primarily in shared memory IPC. It contains a >> schema (metadata) and a // corresponding sequence of equal-length >> Arrow arrays class ARROW_EXPORT RecordBatch { >> >> But I wasn't entirely sure what the author had in mind. >> >> -Mike >> >> -----Original Message----- >> From: Jason Altekruse [mailto:altekruseja...@gmail.com] >> Sent: Wednesday, May 10, 2017 17:59 >> To: dev@arrow.apache.org >> Subject: Re: list scope question >> >> I think we opted to not create a user list yet, as using arrow is likely >> going to require some willingness to poke around in the library until we can >> fill out the docs and some more complete example usages. So as far as I know >> this is the place to ask, what were you looking to do? >> >> On Wed, May 10, 2017 at 2:28 PM, Katelman, Michael < >> michael.katel...@cubistsystematic.com> wrote: >> >>> Is there a good place for user-oriented arrow questions and discussions? >>> (my apologies in advance if this isn't the appropriate venue) >>> >>> -Mike >>> >>> >>> >>> >>> >>> DISCLAIMER: This e-mail message and any attachments are intended >>> solely for the use of the individual or entity to which it is >>> addressed and may contain information that is confidential or legally >>> privileged. If you are not the intended recipient, you are hereby >>> notified that any dissemination, distribution, copying or other use >>> of this message or its attachments is strictly prohibited. If you >>> have received this message in error, please notify the sender >>> immediately and permanently delete this message and any attachments. >>> >>> >>> >>> >> >> >> >> >> >> DISCLAIMER: This e-mail message and any attachments are intended solely for >> the use of the individual or entity to which it is addressed and may contain >> information that is confidential or legally privileged. If you are not the >> intended recipient, you are hereby notified that any dissemination, >> distribution, copying or other use of this message or its attachments is >> strictly prohibited. If you have received this message in error, please >> notify the sender immediately and permanently delete this message and any >> attachments. >> >> >>
