Re: Peak memory usage for pyarrow.parquet.read_table

2018-06-10 Thread Wes McKinney
> * There seems to be some interaction between > `parquet::internal::RecordReader` and `arrow::PoolBuffer` or > `arrow::DefaultMemoryPool`. `RecordReader` request an allocation to hold the > entire column in memory without compression/encoding even though Arrow > supports dictionary encoding (a

Re: Proposed Arrow Graph representations

2018-06-10 Thread Wes McKinney
hi folks, I have glanced at the Flatbuffers file with the proposed graph schemas. IP / licensing problems aside, I don't know enough about graph representations to have the context to judge whether this is the correct approach. My initial reaction is that the file is very long and without a great

[jira] [Created] (ARROW-2694) [Python] ArrayValue string conversion returns the representation instead of the converted python object string

2018-06-10 Thread Florian Jetter (JIRA)
Florian Jetter created ARROW-2694: - Summary: [Python] ArrayValue string conversion returns the representation instead of the converted python object string Key: ARROW-2694 URL: https://issues.apache.org/jira/brows