Re: [EXTERNAL] Re: .NET support for Arrow

2020-07-12 Thread anthony . abate
Wes, >I find that people have many reasons for not contributing to >an existing open source project, so I want to make sure I know what >yours are, whether one of: For the record, I have contributed to this project: both PRs and Jiras for various 'bugs' as I found them - some (most?) made their w

Re: [EXTERNAL] Re: .NET support for Arrow

2020-07-12 Thread anthony . abate
Wes, I thought Arrow was (or at least includes) an open standard for interoperability? There are even specific 'implementation guidelines' regarding supporting parts or all of the specification. It appears that fragmentation is already a problem (ie private forks) Where I work, we don't trust th

Re: [EXTERNAL] Re: .NET support for Arrow

2020-07-12 Thread anthony . abate
I am in the same position as Adam - We don't use the official apache arrow library any more either and have been using an old fork with our own (probably the same) bug fixes. Personally, I have somewhat given up on the Apache .Net library... I have an alternative C# arrow library that I have writt

Re: optimal way to store historical data

2020-06-26 Thread anthony . abate
Also, let me clarify so there is no confusion - There should be no problem creating static / read only arrow data files with a 'date to batch' index in the manner i described. The problem I am referring to only becomes an issue if you need to append a new batch on a daily basis -Anthony On Fri

Re: optimal way to store historical data

2020-06-26 Thread anthony . abate
+1 to this.. There is a logical way to do this now - If you create a batch per day you can maintain a separate arrow file (an index) to map the date to batch.. We do this for indexing via other keys, and I can say it works well for 'large' files - 25gb+. I think unfortunately, doing this via the c

[jira] [Created] (ARROW-9035) 8 vs 64 byte alignment

2020-06-04 Thread Anthony Abate (Jira)
Anthony Abate created ARROW-9035: Summary: 8 vs 64 byte alignment Key: ARROW-9035 URL: https://issues.apache.org/jira/browse/ARROW-9035 Project: Apache Arrow Issue Type: Bug

[jira] [Created] (ARROW-7511) [C#] - Batch / Data Size Can't Exceed 2 gigs

2020-01-07 Thread Anthony Abate (Jira)
Anthony Abate created ARROW-7511: Summary: [C#] - Batch / Data Size Can't Exceed 2 gigs Key: ARROW-7511 URL: https://issues.apache.org/jira/browse/ARROW-7511 Project: Apache Arrow Issue

[jira] [Created] (ARROW-7509) Turn on Checked mode for debug builds

2020-01-07 Thread Anthony Abate (Jira)
Anthony Abate created ARROW-7509: Summary: Turn on Checked mode for debug builds Key: ARROW-7509 URL: https://issues.apache.org/jira/browse/ARROW-7509 Project: Apache Arrow Issue Type

[jira] [Created] (ARROW-7508) DateTime Reading is Broken

2020-01-07 Thread Anthony Abate (Jira)
Anthony Abate created ARROW-7508: Summary: DateTime Reading is Broken Key: ARROW-7508 URL: https://issues.apache.org/jira/browse/ARROW-7508 Project: Apache Arrow Issue Type: Bug

[jira] [Created] (ARROW-7157) RecordBatchFileReader - Crashes RStudio

2019-11-13 Thread Anthony Abate (Jira)
Anthony Abate created ARROW-7157: Summary: RecordBatchFileReader - Crashes RStudio Key: ARROW-7157 URL: https://issues.apache.org/jira/browse/ARROW-7157 Project: Apache Arrow Issue Type: Bug

[jira] [Created] (ARROW-7156) [R] [C++] get_batch - failes for large batches

2019-11-13 Thread Anthony Abate (Jira)
Anthony Abate created ARROW-7156: Summary: [R] [C++] get_batch - failes for large batches Key: ARROW-7156 URL: https://issues.apache.org/jira/browse/ARROW-7156 Project: Apache Arrow Issue

[jira] [Created] (ARROW-7040) StringArrayBuilder.AppendRange - Crashes

2019-10-31 Thread Anthony Abate (Jira)
Anthony Abate created ARROW-7040: Summary: StringArrayBuilder.AppendRange - Crashes Key: ARROW-7040 URL: https://issues.apache.org/jira/browse/ARROW-7040 Project: Apache Arrow Issue Type

[jira] [Created] (ARROW-6830) Question / Feature Request- Select Subset of Columns in read_arrow

2019-10-09 Thread Anthony Abate (Jira)
Anthony Abate created ARROW-6830: Summary: Question / Feature Request- Select Subset of Columns in read_arrow Key: ARROW-6830 URL: https://issues.apache.org/jira/browse/ARROW-6830 Project: Apache

[jira] [Created] (ARROW-6682) Arrow Hangs on Large Files (10-12gb)

2019-09-24 Thread Anthony Abate (Jira)
Anthony Abate created ARROW-6682: Summary: Arrow Hangs on Large Files (10-12gb) Key: ARROW-6682 URL: https://issues.apache.org/jira/browse/ARROW-6682 Project: Apache Arrow Issue Type: Bug

[jira] [Created] (ARROW-6681) [C# -> R] - Record Batches in reverse order?

2019-09-24 Thread Anthony Abate (Jira)
Anthony Abate created ARROW-6681: Summary: [C# -> R] - Record Batches in reverse order? Key: ARROW-6681 URL: https://issues.apache.org/jira/browse/ARROW-6681 Project: Apache Arrow Issue T