hi there,
I am working on the deserialization support for the Go backend.
at this point, I have (I think) primitive and binary/string arrays working
with a simple Arrow file I created like so:
import pyarrow as pa
data = [
pa.array([1, 2, 3, None, 5], type="i4"),
pa.array(['foo', 'bar', 'baz', None, "quux"]),
pa.array([1, 2, None, 4, 5], type="f4"),
pa.array([True, None, False, True, False])
]
batch = pa.RecordBatch.from_arrays(data, ['f0', 'f1', 'f2', "f3"])
sink = pa.BufferOutputStream()
writer = pa.RecordBatchFileWriter(sink, batch.schema)
for i in range(5):
writer.write_batch(batch)
writer.close()
buf = sink.getvalue()
f = open("out.dat", "wb")
f.write(buf.to_pybytes())
f.close()
and, as I said, I can now successfully read that back from Go.
but I was wondering what's the recommanded way to test for this kind of
round-trip/cross-language thing.
I tried to play a bit with "integration/integration_test.py" but it fails
with:
##########################################################
C++ producing, C++ consuming
##########################################################
==========================================================
Testing file /home/binet/work/gonum/src/
github.com/apache/arrow/integration/data/struct_example.json
==========================================================
-- Creating binary inputs
/home/binet/work/gonum/src/
github.com/apache/arrow/integration/../cpp/build/latest/arrow-json-integration-test
--integration --arrow=testdata/75bc5ca6_struct_example.json_as_file
--json=/home/binet/work/gonum/src/
github.com/apache/arrow/integration/data/struct_example.json
--mode=JSON_TO_ARROW
Command failed: /home/binet/work/gonum/src/
github.com/apache/arrow/integration/../cpp/build/latest/arrow-json-integration-test
--integration --arrow=testdata/75bc5ca6_struct_example.json_as_file
--json=/home/binet/work/gonum/src/
github.com/apache/arrow/integration/data/struct_example.json
--mode=JSON_TO_ARROW
With output:
--------------
Found schema: struct_nullable: struct<f1: int32, f2: string>
--------------
==========================================================
Testing file /home/binet/work/gonum/src/
github.com/apache/arrow/integration/data/simple.json
==========================================================
-- Creating binary inputs
/home/binet/work/gonum/src/
github.com/apache/arrow/integration/../cpp/build/latest/arrow-json-integration-test
--integration --arrow=testdata/b0d388ed_simple.json_as_file
--json=/home/binet/work/gonum/src/
github.com/apache/arrow/integration/data/simple.json --mode=JSON_TO_ARROW
Command failed: /home/binet/work/gonum/src/
github.com/apache/arrow/integration/../cpp/build/latest/arrow-json-integration-test
--integration --arrow=testdata/b0d388ed_simple.json_as_file
--json=/home/binet/work/gonum/src/
github.com/apache/arrow/integration/data/simple.json --mode=JSON_TO_ARROW
With output:
--------------
Found schema: foo: int32
bar: double
baz: string
--------------
==========================================================
Testing file testdata/generated_primitive.json
==========================================================
-- Creating binary inputs
/home/binet/work/gonum/src/
github.com/apache/arrow/integration/../cpp/build/latest/arrow-json-integration-test
--integration --arrow=testdata/8eacb124_generated_primitive.json_as_file
--json=testdata/generated_primitive.json --mode=JSON_TO_ARROW
Command failed: /home/binet/work/gonum/src/
github.com/apache/arrow/integration/../cpp/build/latest/arrow-json-integration-test
--integration --arrow=testdata/8eacb124_generated_primitive.json_as_file
--json=testdata/generated_primitive.json --mode=JSON_TO_ARROW
With output:
--------------
Found schema: bool_nullable: bool
bool_nonnullable: bool not null
int8_nullable: int8
int8_nonnullable: int8 not null
int16_nullable: int16
int16_nonnullable: int16 not null
int32_nullable: int32
int32_nonnullable: int32 not null
int64_nullable: int64
int64_nonnullable: int64 not null
uint8_nullable: uint8
uint8_nonnullable: uint8 not null
uint16_nullable: uint16
uint16_nonnullable: uint16 not null
uint32_nullable: uint32
uint32_nonnullable: uint32 not null
uint64_nullable: uint64
uint64_nonnullable: uint64 not null
float32_nullable: float
float32_nonnullable: float not null
float64_nullable: double
float64_nonnullable: double not null
binary_nullable: binary
binary_nonnullable: binary not null
utf8_nullable: string
utf8_nonnullable: string not null
fixedsizebinary_19_nullable: fixed_size_binary[19]
fixedsizebinary_19_nonnullable: fixed_size_binary[19] not null
fixedsizebinary_120_nullable: fixed_size_binary[120]
fixedsizebinary_120_nonnullable: fixed_size_binary[120] not null
--------------
is this supposed to work?
are there reference files already available somewhere?
cheers,
-s
