Re: Apache Arrow at JupyterCon

On a library like Arrow it also is very important to have the less dynamic methods call on the critical paths (get/puts). If it is decided to supports other memory systems, it is important to try to minimize that as much as possible. If there is a single vector class that supports both systems (by

Re: Next Arrow sync call

As a reminder, the Arrow sync is at 16:00 UTC today (12 PM in New York) Here is the hangout link: https://plus.google.com/hangouts/_/calendar/d2VzbWNraW5uQGdtYWlsLmNvbQ.0u7205fhssibdlecevkqtaqlg5?authuser=0 On Wed, Sep 6, 2017 at 12:31 AM, Wes McKinney wrote: > I will send out a Google Hangout l

Re: Next Arrow sync call

I have a conflict but look forward to the recap afterwards. One possible discussion items: ARROW-1463 subtasks assignments, etc. thanks, Jacques On Thu, Sep 7, 2017 at 7:33 AM, Wes McKinney wrote: > As a reminder, the Arrow sync is at 16:00 UTC today (12 PM in New York) > > Here is the hangout

Re: Next Arrow sync call

Thanks Jacques -- maybe we can schedule another sync next week or the week after that works for your schedule since we have a bunch of high-touch work pending. We should set up a standing meeting time every 2 weeks On Thu, Sep 7, 2017 at 10:46 AM, Jacques Nadeau wrote: > I have a conflict but loo

Re: Next Arrow sync call

Question: As an interested party following Arrow development I'd probably listen in, but I also recognize that it may be distracting to have a large group of folks calling in. So is there a general preference to 'keep it small' and focused on active developers? Which would be completely understanda

Re: Apache Arrow at JupyterCon

Our general goal (which hasn't always been succesfully implemented) is what I'd describe as "fractured subclassing". You can see our use of this where ArrowBuf may extend various Netty classes but is interacting directly with memory addresses for all the hot path get/set operations (not delegating

Re: Next Arrow sync call

Agreed. Do you want to discuss a couple proposed times on the call and then we can finalize afterwards on the list? On Thu, Sep 7, 2017 at 7:48 AM, Wes McKinney wrote: > Thanks Jacques -- maybe we can schedule another sync next week or the > week after that works for your schedule since we have

Re: Next Arrow sync call

Brian, you're welcome to join. On Thu, Sep 7, 2017 at 7:49 AM, Brian Wylie wrote: > Question: As an interested party following Arrow development I'd probably > listen in, but I also recognize that it may be distracting to have a large > group of folks calling in. So is there a general preference

Re: Next Arrow sync call

hi Brian, No, everyone is welcome. The sync calls are mostly about reviewing priorities and having higher-bandwidth discussion about recent issues. As far as the governance of the project, no development decisions are made in private, and anything that's discussed on the call will be relayed to th

Re: Next Arrow sync call

Sounds good. Since we have many active developers in Europe and the US, the 9am Pacific / 12 PM Eastern slot is usually pretty good. That's 00:00 in Shanghai, and 01:00 in Tokyo, so can be a little bit challenging for developers in Asia, so a bit earlier might be better. On Thu, Sep 7, 2017 at 10:

Re: Next Arrow sync call

We had 10 people on the call today, thanks all for joining! If we go over 10 people we will need to find an alternative to Google Hangouts - Attendees - Wes (Two Sigma) - 0.7.0 release - Li Jin (Two Sigma) - Java - Other complex types integration tests - Tom Augspurger (Anaconda

Re: Apache Arrow at JupyterCon

Yes, the performance is critical for most of the big data applications that is one of key success factors for both of Arrow and Mnemonic. a performance-oriented engineer might even against fundamental design patterns for performance. so the problem is how can we make their lives easier? from my poi

[jira] [Created] (ARROW-1481) [C++] Expose type casts as generic callable object that can write into pre-allocated memory

Wes McKinney created ARROW-1481: --- Summary: [C++] Expose type casts as generic callable object that can write into pre-allocated memory Key: ARROW-1481 URL: https://issues.apache.org/jira/browse/ARROW-1481

Time for bi-weekly Arrow sync

The sync call today was at 16:00 UTC which which is an hour that works decently well for developers in North/South America and Europe, but it's a bit late for Asia (12am in Shanghai, 1am in Tokyo). It would be good to have a standing biweekly meeting. I would suggest we try the 16:00 time and if w

ARROW-1463 subtask assignments - https://issues.apache.org/jira/browse/ARROW-1463

Hi All, I am wondering if anyone is interested in working on sub-tasks for ARROW-1463. Please feel free to grab the child JIRAs. Thanks, Siddharth

[jira] [Created] (ARROW-1483) [C++] Implement casts between time32 and time64

Wes McKinney created ARROW-1483: --- Summary: [C++] Implement casts between time32 and time64 Key: ARROW-1483 URL: https://issues.apache.org/jira/browse/ARROW-1483 Project: Apache Arrow Issue Type

[jira] [Created] (ARROW-1482) [C++] Implement casts between date32 and date64

Wes McKinney created ARROW-1482: --- Summary: [C++] Implement casts between date32 and date64 Key: ARROW-1482 URL: https://issues.apache.org/jira/browse/ARROW-1482 Project: Apache Arrow Issue Type

[jira] [Created] (ARROW-1484) [C++] Implement (safe and unsafe) casts between timestamps of different units

Wes McKinney created ARROW-1484: --- Summary: [C++] Implement (safe and unsafe) casts between timestamps of different units Key: ARROW-1484 URL: https://issues.apache.org/jira/browse/ARROW-1484 Project: Ap

[jira] [Created] (ARROW-1485) [C++] Implement union-like data type for accommodating kernel arguments which may be scalars or arrays

Wes McKinney created ARROW-1485: --- Summary: [C++] Implement union-like data type for accommodating kernel arguments which may be scalars or arrays Key: ARROW-1485 URL: https://issues.apache.org/jira/browse/ARROW-1485

[jira] [Created] (ARROW-1486) [C++] Decide if arrow::RecordBatch needs to be copyable

Wes McKinney created ARROW-1486: --- Summary: [C++] Decide if arrow::RecordBatch needs to be copyable Key: ARROW-1486 URL: https://issues.apache.org/jira/browse/ARROW-1486 Project: Apache Arrow Is

Travis CI problems

Is anyone else seeing the following failures in Travis CI build? I am seeing these problems for PR https://github.com/apache/arrow/pull/1052 I looked at the raw log and nothing seems to indicate problems w.r.t code changes. travis_time:end:082db3a8:start=1504813701624387520,finish=150481370162812

Re: Time for bi-weekly Arrow sync

Generally yes. However, Unfortunately Thursday's at 16:00 are bad for me. Maybe we do when Parquet does but offset weeks? On Thu, Sep 7, 2017 at 11:28 AM, Wes McKinney wrote: > The sync call today was at 16:00 UTC which which is an hour that works > decently well for developers in North/South Am

Re: Time for bi-weekly Arrow sync

Wednesday is better than Thursday for me too (though you should optimize for more active members than me). Kevin Moore CEO, Quilt Data, Inc. ke...@quiltdata.io | LinkedIn (415) 497-7895 Manage Data like Code quiltdata.com On Thu, Sep 7, 2017 at 5:

Re: Time for bi-weekly Arrow sync

I would be happy to do that time but on Wednesday. So 20th September at 16:00 for the next sync (if we run into a conflict with the Parquet sync we can skip a week). I can set up a calendar invite and add all interested parties. On Thu, Sep 7, 2017 at 8:26 PM, Kevin Moore wrote: > Wednesday is be

[jira] [Created] (ARROW-1487) [C++] Implement casts from List to List, where a cast function is defined from any A to B

Wes McKinney created ARROW-1487: --- Summary: [C++] Implement casts from List to List, where a cast function is defined from any A to B Key: ARROW-1487 URL: https://issues.apache.org/jira/browse/ARROW-1487

Re: Travis CI problems

That's from the JDK9 build which was only added on August 17 (https://github.com/apache/arrow/commit/4ef7c898bb82cd3513e0ad3d80730e29ebaeb60e#diff-93f725a07423fe1c889f448b33d21f46). The flakiness started sometime in the last 3 days. If Laurent or someone with Java background could investigate the

[jira] [Created] (ARROW-1488) [C++] Implement ArrayBuilder::Finish in terms of internal::ArrayData

Wes McKinney created ARROW-1488: --- Summary: [C++] Implement ArrayBuilder::Finish in terms of internal::ArrayData Key: ARROW-1488 URL: https://issues.apache.org/jira/browse/ARROW-1488 Project: Apache Arro

[jira] [Created] (ARROW-1489) [C++] Add casting option to set unsafe casts to null rather than some garbage value

Wes McKinney created ARROW-1489: --- Summary: [C++] Add casting option to set unsafe casts to null rather than some garbage value Key: ARROW-1489 URL: https://issues.apache.org/jira/browse/ARROW-1489 Proje

[jira] [Created] (ARROW-1490) [Java] Allow Travis CI failures for JDK9 for now

Wes McKinney created ARROW-1490: --- Summary: [Java] Allow Travis CI failures for JDK9 for now Key: ARROW-1490 URL: https://issues.apache.org/jira/browse/ARROW-1490 Project: Apache Arrow Issue Typ

Re: Travis CI problems

I submitted a patch to allow failures in this entry in the build matrix for the time being https://github.com/apache/arrow/pull/1064 On Thu, Sep 7, 2017 at 9:08 PM, Wes McKinney wrote: > That's from the JDK9 build which was only added on August 17 > (https://github.com/apache/arrow/commit/4ef7c8

Re: Travis CI problems

Thanks a lot. I will take a stab at finding the root cause of these failures. On Sep 7, 2017 6:57 PM, "Wes McKinney" wrote: > I submitted a patch to allow failures in this entry in the build > matrix for the time being > > https://github.com/apache/arrow/pull/1064 > > On Thu, Sep 7, 2017 at 9:08

[DISCUSS] Publishing Arrow development artifacts more frequently for alpha stage components

hi folks, As we expand the scope of the Arrow project, we will acquire more components and code in new languages. Some of these components will be at a different stage of maturity, e.g. lacking integration tests or other important features that we've been working on in the main reference implement

Re: Time for bi-weekly Arrow sync

Great, thanks for coordinating! On Thu, Sep 7, 2017 at 6:03 PM, Wes McKinney wrote: > I would be happy to do that time but on Wednesday. So 20th September > at 16:00 for the next sync (if we run into a conflict with the Parquet > sync we can skip a week). I can set up a calendar invite and add a

[jira] [Created] (ARROW-1491) [C++] Add casting implementations from strings to numbers or boolean

Wes McKinney created ARROW-1491: --- Summary: [C++] Add casting implementations from strings to numbers or boolean Key: ARROW-1491 URL: https://issues.apache.org/jira/browse/ARROW-1491 Project: Apache Arro

[jira] [Created] (ARROW-1492) [C++] Type casting function suite

Wes McKinney created ARROW-1492: --- Summary: [C++] Type casting function suite Key: ARROW-1492 URL: https://issues.apache.org/jira/browse/ARROW-1492 Project: Apache Arrow Issue Type: New Feature

[jira] [Created] (ARROW-1493) [C++] Flush the output stream at the end of each PrettyPrint function

Wes McKinney created ARROW-1493: --- Summary: [C++] Flush the output stream at the end of each PrettyPrint function Key: ARROW-1493 URL: https://issues.apache.org/jira/browse/ARROW-1493 Project: Apache Arr

[jira] [Created] (ARROW-1494) [C++] Document that shared_ptr returned by RecordBatch::column needs to be retained

Wes McKinney created ARROW-1494: --- Summary: [C++] Document that shared_ptr returned by RecordBatch::column needs to be retained Key: ARROW-1494 URL: https://issues.apache.org/jira/browse/ARROW-1494 Proje

Re: Time for bi-weekly Arrow sync

Wednesdays at 16:00 UTC is fine for me. Thanks for organising! On Fri, Sep 8, 2017, at 04:19 AM, Jacques Nadeau wrote: > Great, thanks for coordinating! > > On Thu, Sep 7, 2017 at 6:03 PM, Wes McKinney wrote: > > > I would be happy to do that time but on Wednesday. So 20th September > > at 16:0