Re: [Discuss] Support an alternative memory layout for varchar/varbinary vectors

2019-07-14 Thread Fan Liya
@Wes McKinney, Thanks a lot for your comments and effort. The JIRA looks good. I will track it. Best, Liya Fan On Fri, Jul 12, 2019 at 10:31 PM Wes McKinney wrote: > hi Liya -- yes, it seems reasonable to defer the conversion from your > pointer-based extension representation to a proper VarCh

Re: Gandiva

2019-07-14 Thread Andy Grove
I hadn't planned on discussing this here for another week or two, but I am working on a new PoC right now (https://github.com/andygrove/ballista) where I have taken the Gandiva protobuf/gRPC definition and started adding logical query plan messages to it, to enable distributed queries (using the Ru

Re: Gandiva

2019-07-14 Thread Wes McKinney
I must apologize for my C++-centrist view of the world -- note that we do have a query engine project in the codebase already, in Rust https://github.com/apache/arrow/tree/master/rust/datafusion Andy has developed a SQL front end for DataFusion that I think is partially contained in https://githu

Re: Gandiva

2019-07-14 Thread Wes McKinney
hi Doug, Gandiva has a much narrower scope of functionality than what you're looking for. It generates optimized computational "kernels" that can be used inside of query engine runtime. Let's consider a SQL query like: SELECT YEAR(a), MONTH(b), DAY(c), log(d - 1) FROM table WHERE e < 5 & f > 10

Gandiva

2019-07-14 Thread Doug Friedman
Hello- I have a feeling this question will have a painfully simple answer so I apologize in advance: I am interested in the Gandiva portion of the apache arrow project, as a SQL-like interface to arrow data. I browsed the source code and I see a lot relating to building and evaluating expression

[jira] [Created] (ARROW-5949) [Rust] Implement DictionaryArray

2019-07-14 Thread David Atienza (JIRA)
David Atienza created ARROW-5949: Summary: [Rust] Implement DictionaryArray Key: ARROW-5949 URL: https://issues.apache.org/jira/browse/ARROW-5949 Project: Apache Arrow Issue Type: New Feature

[jira] [Created] (ARROW-5948) [Rust] [DataFusion] create_logical_plan should not call optimizer

2019-07-14 Thread Andy Grove (JIRA)
Andy Grove created ARROW-5948: - Summary: [Rust] [DataFusion] create_logical_plan should not call optimizer Key: ARROW-5948 URL: https://issues.apache.org/jira/browse/ARROW-5948 Project: Apache Arrow

[jira] [Created] (ARROW-5947) [Rust] [DataFusion] Remove serde_json dependency

2019-07-14 Thread Andy Grove (JIRA)
Andy Grove created ARROW-5947: - Summary: [Rust] [DataFusion] Remove serde_json dependency Key: ARROW-5947 URL: https://issues.apache.org/jira/browse/ARROW-5947 Project: Apache Arrow Issue Type: I