[Java] Memory Allocation Tips

2020-04-20 Thread Razvan Chitu
Hi, Does the Arrow community have any tips / recommendations / best practices on how to manage Arrow memory in Java? Is there a way to rely on the GC exclusively (i.e. is there support for heap-only allocation)? Best, Razvan

[jira] [Created] (ARROW-6899) to_pandas() not implemented on list

2019-10-16 Thread Razvan Chitu (Jira)
Razvan Chitu created ARROW-6899: --- Summary: to_pandas() not implemented on list Key: ARROW-6899 URL: https://issues.apache.org/jira/browse/ARROW-6899 Project: Apache Arrow Issue Type: Bug

Re: IPC Tensor + Indices

2019-07-12 Thread Razvan Chitu
x? means? > > On Fri, Jul 12, 2019 at 11:17 AM Razvan Chitu > wrote: > > > > Hi, > > > > Does the IPC format currently support streaming a tensor together with a > > row and a column index? If not, are there any plans for this to be > > supporte

IPC Tensor + Indices

2019-07-12 Thread Razvan Chitu
Hi, Does the IPC format currently support streaming a tensor together with a row and a column index? If not, are there any plans for this to be supported? It'd be quite a useful for matrices that could have 10s of thousands of either rows, columns or both. For my use case I am currently representi

Re: Java OutOfMemoryException!

2019-03-24 Thread Razvan Chitu
Hi Tanveer, The stack trace seems to indicate that you you've breached the limit of the allocator used by the ArrowStreamReader, so that's where I'd look first. The limit is usually set when constructing an allocator (e.g. new RootAllocator(myLimit)) or when getting a child allocator (e.g. rootAll

Memory mapped files in Java

2019-03-19 Thread Razvan Chitu
Hi, I was looking for a way to interact with memory mapped Arrow files in Java and I found this thread: http://mail-archives.apache.org/mod_mbox/arrow-dev/201709.mbox/%3CCAOgX8szfO-F=ccsqcggucqfzqkgu2wy+pihztbv1gkat4eq...@mail.gmail.com%3E . Are there any updates on the status of an implementation