Can someone remove me from this thread, please. Thanks
Sent from my iPhone > On Aug 18, 2016, at 8:54 PM, Micah Kornfield <emkornfi...@gmail.com> wrote: > > Thanks Julien for organizing the meeting and taking notes. I wrote up some > initial thoughts on shared memory IPC on > https://issues.apache.org/jira/browse/ARROW-263 > > I'll try to flesh out a more concrete spec today/tomorrow. > > -Micah > >> On Thu, Aug 18, 2016 at 10:25 AM, Julien Le Dem <jul...@dremio.com> wrote: >> >> My notes: (I'll schedule another one in 2 weeks but people should feel free >> to do ad-hoc discussion in the meantime) >> >> Attendees and their topic of interest for today: >> - Micah Kornfield: Dictionary encoding, Reusing dictionaries across record >> batches, Shared memory, memory management, releasing memory shared accross >> processes >> - Wes McKinney: Finalize types (Category, ...), File format RPC format, >> IPC >> - Julien Le Dem: finalize metadata (RPC, IPC, File), File format >> implementation, UDF use case >> - Erol: Shared memory across Java and C++ to share large amounts of data >> >> Arrow IPC: >> - Shared memory: >> - current version doesn’t do Schema negotiation yet. >> - all unit tests reading writing out memory with a predefined schema >> and known based address. >> - no dictionary encoding yet. >> - issues to discuss: >> - communicating the base memory address: >> - possibly use RPC for coordination. >> - options for shared memory >> - forking a process: anonymous shared memory implicitly >> - starting a new process. Need to spawn alternate shared memory that >> needs to be cleaned up >> - direct memory mapped system call (communicate file name to >> subprocess). >> - Action (Micah) create a JIRA to sum this up >> >> - Memory management: >> - the process producing the data will allocate the memory and pass it >> read only. It needs to wait for the consumer to be done to release it. >> - one option is memory mapped file (persistent independent of the >> process) >> - each process responsible for its memory. Reader needs to release >> memory. >> - mechanism for handling too much memory allocation. >> - In the case of record batches over RPC this is not an issue (memory is >> copied over). >> >> - RPC transport >> definition of the protocol and how we send message. >> - File transport >> >> - Dictionary encoding: >> - start simple: simple buffer<int> layout >> - enable extension in the future (v2: bit packing?) >> >> - Category type: >> - Semantic difference with Dictionary encoded. >> - TODO(Julien): Add Category type in Parquet? >> >> >>> On Thu, Aug 18, 2016 at 9:39 AM, Julien Le Dem <jul...@dremio.com> wrote: >>> >>> Hi Nicole. >>> Can you try again? >>> I was accepting you but it did not seem to work. >>> Julien >>> >>> On Thu, Aug 18, 2016 at 9:26 AM, Nicole Nemer <nicole.ne...@rms.com> >>> wrote: >>> >>>> I am trying to join and it not letting me inŠ >>>> nn >>>> ‹ >>>> Nicole Nemer, PhD >>>> Technical Architect/Dev Manager >>>> >>>> 303-641-3340 >>>> >>>> >>>> >>>> >>>> >>>> >>>>> On 8/18/16, 10:00 AM, "Julien Le Dem" <jul...@dremio.com> wrote: >>>>> >>>>> And this is starting now. >>>>> https://plus.google.com/hangouts/_/dremio.com/arrow >>>>> >>>>>> On Wed, Aug 17, 2016 at 7:07 PM, Julien Le Dem <jul...@dremio.com> >>>>> wrote: >>>>> >>>>>> Here is the hangout link for tomorrow: >>>>>> https://plus.google.com/hangouts/_/dremio.com/arrow >>>>>> >>>>>> I have also added to a google calendar event everyone who replied to >>>>>> that >>>>>> thread. >>>>>> >>>>>> >>>>>> On Wed, Aug 17, 2016 at 6:12 PM, Wes McKinney <wesmck...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> hi folks, >>>>>>> >>>>>>> Reminder that the Arrow sync is tomorrow morning at 09:00 Pacific >>>>>>> (http://timesched.pocoo.org/?date=2016-08-18&tz=pacific-stan >>>>>>> dard-time!&range=540,600). >>>>>>> I believe Julien will send a public Google hangout link to the >> mailing >>>>>>> list for you all to join. >>>>>>> >>>>>>> Thanks >>>>>>> Wes >>>>>>> >>>>>>> On Tue, Aug 16, 2016 at 11:07 AM, Wes McKinney <wesmck...@gmail.com >>> >>>>>>> wrote: >>>>>>>> +1. If there is demand for an Asia-friendly time we can change >>>> things >>>>>>> up from week to week. >>>>>>>> >>>>>>>>> On Aug 16, 2016, at 10:52 AM, Jacques Nadeau <jacq...@apache.org >>> >>>>>>> wrote: >>>>>>>>> >>>>>>>>> sounds good >>>>>>>>> >>>>>>>>>> On Tue, Aug 16, 2016 at 10:39 AM, Julien Le Dem < >>>> jul...@dremio.com> >>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Based on the feedback I'm proposing Thursday Aug 18 at 4PM UTC >> as >>>>>>> the >>>>>>> first >>>>>>>>>> Arrow sync. >>>>>>>>>> That's: >>>>>>>>>> - 9AM PDT (San Francisco) >>>>>>>>>> - 12PM EDT (New York) >>>>>>>>>> - 5PM CET (London) >>>>>>>>>> - 6PM CEST (Paris, Berlin) >>>>>>>>>> >>>>>>>>>>> On Tue, Aug 9, 2016 at 6:45 AM, Uwe L. Korn <uw...@xhochy.com> >>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> +1 for bi-weekly and europeen friendly times: CET (GMT+1) >>>>>>>>>>> >>>>>>>>>>>> Am 09.08.2016 um 00:39 schrieb Julien Le Dem < >> jul...@dremio.com >>>>> : >>>>>>>>>>>> >>>>>>>>>>>> Also to all who are responding let me know your timezone as >>>> well. >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Aug 8, 2016 at 3:30 PM, Micah Kornfield < >>>>>>> emkornfi...@gmail.com >>>>>>>>>>> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Sounds good to me as well. Biweekly would be preferred. >>>>>>>>>>>>> >>>>>>>>>>>>>> On Monday, August 8, 2016, Wes McKinney < >> wesmck...@gmail.com> >>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> hi Julien -- this sounds like a good idea, also +1 for >>>>>>> bi-weekly. >>>>>>> I >>>>>>>>>>>>>> will do my best to join when possible. So far we've mostly >>>> been >>>>>>>>>>>>>> communicating via pull request, so I think periodic syncs >> will >>>>>>> be >>>>>>>>>>>>>> helpful. >>>>>>>>>>>>>> >>>>>>>>>>>>>> - Wes >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Aug 8, 2016 at 2:45 PM, P. Taylor Goetz < >>>>>>> ptgo...@gmail.com >>>>>>>>>>>>>> <javascript:;>> wrote: >>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> My preference would be for bi-weekly. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -Taylor >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Aug 8, 2016, at 5:25 PM, Julien Le Dem < >>>> jul...@dremio.com >>>>>>>>>>>>>> <javascript:;>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>>> My experience with Parquet is that a regular sync up over >>>>>>> hangout >>>>>>>>>>>>> helps >>>>>>>>>>>>>>>> keeping in touch and staying updated about what everyone >> is >>>>>>> doing. >>>>>>>>>>>>>>>> I was thinking of scheduling it weekly or bi-weekly. >>>>>>>>>>>>>>>> Who would join? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The way it goes is first we do a round table where people >>>>>>> introduce >>>>>>>>>>>>>>>> themselves an list the topics they'd like to talk or hear >>>>>>> about. >>>>>>>>>>>>>>>> That makes the agenda and we go through it. >>>>>>>>>>>>>>>> At the end we send notes to the mailing list with >>>> discussions >>>>>>> and >>>>>>>>>>>>> action >>>>>>>>>>>>>>>> items (for example: open JIRA, comment on JIRA, review PR, >>>>>>> etc). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Julien >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Julien >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Julien >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Julien >>>>> >>>>> >>>>> >>>>> -- >>>>> Julien >>> >>> >>> -- >>> Julien >> >> >> >> -- >> Julien >>
