Hello all,
I've raised a JIRA ticket (https://issues.apache.org/jira/browse/ARROW-15705)
for this, but I'm still uncertain on my reading of the spec so I thought
I'd ask here to confirm I've understood it correctly.
I believe that child arrays should always be the same length as the struct
array?
Can someone clarify whether the spec is clear about the behavior?
On Feb 18, 2022 at 07:23:19, Alfie Mountfield wrote:
> Hello all,
> I've raised a JIRA ticket (
> https://issues.apache.org/jira/browse/ARROW-15705)
> for this, but I'm still uncertain on my reading of the spec so I thought
> I'd
Hi Dominik,
That is my understanding - if it exists, the length of the validity must
equal the length of each field. Otherwise, it would be difficult to iterate
over the fields and validity together, since we would not have enough rows
in the fields for the validity.
I think that this is broader
My read of the spec for structs [1] is that there is no requirement to have
a value in child arrays where there are nulls, which suggests the
implementation conforms to the spec here.
The example emphasizes this by showing the VarBinary column data as
"joemark" as opposed to something like "joemar
Isn't field-0 representing ["joe", None, None, "mark"]? validity is
"1001" and offsets [0,3,3,7]. My reading is that the values buffer is
"joemark" because we do not represent values in null slots.
Best,
Jorge
On Fri, Feb 18, 2022 at 7:07 PM Phillip Cloud wrote:
> My read of the spec for s
On Fri, Feb 18, 2022 at 1:14 PM Jorge Cardoso Leitão <
jorgecarlei...@gmail.com> wrote:
> Isn't field-0 representing ["joe", None, None, "mark"]? validity is
> "1001" and offsets [0,3,3,7]. My reading is that the values buffer is
> "joemark" because we do not represent values in null slots.
>
Le 18/02/2022 à 19:29, Phillip Cloud a écrit :
The description underneath the example says:
While a struct does not have physical storage for each of its semantic
slots
(i.e. each scalar C-like struct), an entire struct slot can be set to
null via the validity bitmap.
To me this suggests
>
> It is definitely required according to my understanding, and to how the
> C++ implementation works. The validation functions in the C++
> implementation also check for this (if a child buffer is too small for
> the number of values advertised by the parent, it is an error).
+1.
I think the w
I think I'm confused by where this appended value lives. Is it only a
logical value or does the value show up in memory?
For example, appending another null to the name field is only going to
change the validity map, offsets array and length and there will not be any
changes the values buffer.
The
Le 18/02/2022 à 20:01, Phillip Cloud a écrit :
I think I'm confused by where this appended value lives. Is it only a
logical value or does the value show up in memory?
The logical value is null. The appended value is only a physical value
that shows up in memory but doesn't have any bearing
On Fri, Feb 18, 2022 at 2:06 PM Antoine Pitrou wrote:
>
> Le 18/02/2022 à 20:01, Phillip Cloud a écrit :
> > I think I'm confused by where this appended value lives. Is it only a
> > logical value or does the value show up in memory?
>
> The logical value is null. The appended value is only a ph
A change in the length of an array is equivalent to a change in at least
one of its buffers (i.e. length is always physical).
* Primitive arrays (i32, i64, etc): the arrays' length is equal to the
length of the buffer divided by the size of the type. E.g. buffer.len() = 8
and i32 <=> length = 2)
*
Le 18/02/2022 à 20:26, Phillip Cloud a écrit :
On Fri, Feb 18, 2022 at 2:06 PM Antoine Pitrou wrote:
Le 18/02/2022 à 20:01, Phillip Cloud a écrit :
I think I'm confused by where this appended value lives. Is it only a
logical value or does the value show up in memory?
The logical value is
On Fri, Feb 18, 2022 at 2:32 PM Antoine Pitrou wrote:
>
> Le 18/02/2022 à 20:26, Phillip Cloud a écrit :
> > On Fri, Feb 18, 2022 at 2:06 PM Antoine Pitrou
> wrote:
> >
> >> Le 18/02/2022 à 20:01, Phillip Cloud a écrit :
> >>> I think I'm confused by where this appended value lives. Is it only a
Le 18/02/2022 à 21:32, Phillip Cloud a écrit :
I am really struggling to see how anything I've said is inconsistent with
the spec or what you are saying here.
To recap what I've said:
1. Appending a null sentinel to the values buffer isn't _required_ unless
the type requires it.
Ex: "joemark
I wasn't able to able to run the entire process, so I downloaded a few
artifacts from the nightly java-jars and pointed the script there to see
the output:
dev/release/06-java-upload.sh 7.0.0 10
deploy:deploy-file -Durl=
https://repository.apache.org/service/local/staging/deploy/maven2
-Dreposito
On Fri, Feb 18, 2022 at 3:44 PM Antoine Pitrou wrote:
>
> Le 18/02/2022 à 21:32, Phillip Cloud a écrit :
> >
> > I am really struggling to see how anything I've said is inconsistent with
> > the spec or what you are saying here.
> >
> > To recap what I've said:
> >
> > 1. Appending a null sentine
>
> Ok, then perhaps you might have some thoughts on the original question: is
> the JavaScript implementation currently incorrect?
I think whether it is a bug or not depends on the contract of the builder.
If the contract is that the builder assumes users will ensure equal lengths
of all the chi
18 matches
Mail list logo
