Hello,
Re: other projects, I'd like to point out the approach that we've taken
on GeoMesa, a geospatial project that I work on. We model geometries in
Arrow similarly to the GeoJSON spec[1], as lists of pairs of
coordinates. We used FixedSizeList vectors of size 2 to represent each
coordinate
Hello,
I think the issue is that current master is version 4.0.0-SNAPSHOT now,
but your PR is 3.0.0-SNAPSHOT:
https://github.com/apache/arrow/blob/master/java/format/pom.xml#L18
Thanks,
Emilio
On 2/26/21 4:58 AM, Fan Liya wrote:
Dear all,
In a recent PR [1], I have created a new sub-modul
From my time working on the arrow writers, I think that would be fine.
You could do the same thing with the dictionary blocks, as well.
As an implementation idea, it might be cleaner to add some callback
hooks, i.e. onRecordBlockWritten(), and then implement that in the
FileWriter instead of h
Emilio Lahr-Vivaz created ARROW-2500:
Summary: [Java] IPC Writers/readers are not always setting
validity bits correctly
Key: ARROW-2500
URL: https://issues.apache.org/jira/browse/ARROW-2500
Hi Atul,
You should be able to use the overloaded 'set' method that takes a
NullableVarCharHolder:
https://github.com/apache/arrow/blob/master/java/vector/src/main/java/org/apache/arrow/vector/VarCharVector.java#L237
Thanks,
Emilio
On 04/10/2018 05:23 PM, Atul Dambalkar wrote:
Hi,
I wante
This has probably changed with the Java code refactor, but I've posted
some answers inline, to the best of my understanding.
Thanks,
Emilio
On 12/16/2017 12:17 PM, Animesh Trivedi wrote:
Thanks Wes for you help.
Based upon some code reading, I managed to code-up a basic working example.
The
ttps://github.com/apache/arrow/blob/master/format/IPC.md that message
sizes are expected to be a multiple of 8. We should also take a look
at the File format implementation to ensure that padding is inserted
after the magic number at the start of the file
- Wes
On Tue, Aug 8, 2017 at 1:32 PM, E
distributed fashion, and then > concatenating them in the streaming format.
Can you show the code for this?
On Tue, Aug 8, 2017 at 12:35 PM, Emilio Lahr-Vivaz
wrote:
So I think the issue is that we are serializing record batches in a
distributed fashion, and then concatenating them in
Emilio Lahr-Vivaz created ARROW-1340:
Summary: [Java] NullableMapVector field doesn't maintain metadata
Key: ARROW-1340
URL: https://issues.apache.org/jira/browse/ARROW-1340
Project: Apache
.
Thanks,
Emilio
On 08/08/2017 09:18 AM, Emilio Lahr-Vivaz wrote:
Hi Wes,
You're right, I just realized that. I think the alignment issue might
be in some unrelated code, actually. From what I can tell the the
arrow writers are aligning buffers correctly; if not I'll open a bug.
Thank
is a bug. Could you clarify?
- Wes
On Tue, Aug 8, 2017 at 8:52 AM, Emilio Lahr-Vivaz wrote:
After looking at it further, I think only the buffers themselves need to be
aligned, not the metadata and/or schema. Would there be any problem with
changing the alignment to 64 bytes then?
Thanks,
Emili
After looking at it further, I think only the buffers themselves need to
be aligned, not the metadata and/or schema. Would there be any problem
with changing the alignment to 64 bytes then?
Thanks,
Emilio
On 08/08/2017 08:08 AM, Emilio Lahr-Vivaz wrote:
I'm looking into buffer alignme
I'm looking into buffer alignment in the java writer classes. Currently
some files written with the java streaming writer can't be read due to
the javascript TypedArray's restriction that the start offset of the
array must be a multiple of the data size of the array type (i.e.
Int32Vectors must
I'm also curious. For reference, I've been using a global RootAllocator
instance shared among all my classes. For unit tests, I've been creating
one per test to ensure that things are being closed correctly.
Thanks,
Emilio
On 06/27/2017 08:15 AM, Li Jin wrote:
Still wondering if anyone can g
Congrats on the release! Is there a time frame for java artifacts being
available on maven central?
Thanks,
Emilio
On 05/23/2017 01:06 PM, Wes McKinney wrote:
The Apache Arrow community is pleased to announce the 0.4.0 release. It
includes 77 resolved issues ([1]) since the 0.3.0 release.
Th
Emilio Lahr-Vivaz created ARROW-1015:
Summary: [Java] Implement schema-level metadata
Key: ARROW-1015
URL: https://issues.apache.org/jira/browse/ARROW-1015
Project: Apache Arrow
Issue
Emilio Lahr-Vivaz created ARROW-999:
---
Summary: [Java] Minor types don't account for nullable FieldType
flag
Key: ARROW-999
URL: https://issues.apache.org/jira/browse/ARROW-999
Project: Apache
Emilio Lahr-Vivaz created ARROW-997:
---
Summary: [Java] Implement transfer in FixedSizeListVector
Key: ARROW-997
URL: https://issues.apache.org/jira/browse/ARROW-997
Project: Apache Arrow
Emilio Lahr-Vivaz created ARROW-886:
---
Summary: VariableLengthVectors don't reAlloc offsets
Key: ARROW-886
URL: https://issues.apache.org/jira/browse/ARROW-886
Project: Apache Arrow
Emilio Lahr-Vivaz created ARROW-815:
---
Summary: [Java] Allow for expanding underlying buffer size after
allocation
Key: ARROW-815
URL: https://issues.apache.org/jira/browse/ARROW-815
Project: Apache
Emilio Lahr-Vivaz created ARROW-691:
---
Summary: [Java] Encode dictionary Int type in message format
Key: ARROW-691
URL: https://issues.apache.org/jira/browse/ARROW-691
Project: Apache Arrow
Oops, sorry I mean to attend but forgot. Re: dictionary encoding
support, I haven't had time to work on it recently, but I should have
some next week. I've partially coded some of the changes but it's not in
a working state to push at the moment.
Thanks,
Emilio
On 03/02/2017 02:28 PM, Julien
[
https://issues.apache.org/jira/browse/ARROW-542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860292#comment-15860292
]
Emilio Lahr-Vivaz commented on ARROW-542:
-
Another blocker I'm hitting
[
https://issues.apache.org/jira/browse/ARROW-542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15859863#comment-15859863
]
Emilio Lahr-Vivaz commented on ARROW-542:
-
It's getting a little co
[
https://issues.apache.org/jira/browse/ARROW-542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15858342#comment-15858342
]
Emilio Lahr-Vivaz commented on ARROW-542:
-
Ah, makes sense thanks.
>
[
https://issues.apache.org/jira/browse/ARROW-542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15858295#comment-15858295
]
Emilio Lahr-Vivaz commented on ARROW-542:
-
[~wesmckinn] I'm looking
Emilio Lahr-Vivaz created ARROW-542:
---
Summary: [Java] Implement dictionaries in stream/file encoding
Key: ARROW-542
URL: https://issues.apache.org/jira/browse/ARROW-542
Project: Apache Arrow
he
initial C++ implementation in
https://github.com/apache/arrow/commit/74685f386307171a90a9f97316e25b
7f39cdd0a1#diff-708b00b9a46568e0fac8dcc1ac5f2749
If you need help feel free to ping us on the mailing list or JIRA.
best
Wes
On Tue, Jan 24, 2017 at 12:39 PM, Emilio Lahr-Vivaz
wrote:
H
Hello,
I'm interested in Java dictionary encoding
(https://issues.apache.org/jira/browse/ARROW-366). Can I pick that up
and start working on it? Anything I need to do first?
Thanks,
Emilio Lahr-Vivaz
29 matches
Mail list logo
