hi Sebastien, The integration tests indeed should work (they are run in the for release verification script [1]), so something is either wrong with your C++ build or your environment if integration_test.py fails. It would be great to get Go into the integration tests to have proof that the implementation is compatible with the others (C++, Java, JS)
- Wes [1]: https://github.com/apache/arrow/blob/master/dev/release/verify-release-candidate.sh#L386 On Wed, Apr 3, 2019 at 12:22 PM Sebastien Binet <bi...@cern.ch> wrote: > > hi there, > > I am working on the deserialization support for the Go backend. > at this point, I have (I think) primitive and binary/string arrays working > with a simple Arrow file I created like so: > > import pyarrow as pa > data = [ > pa.array([1, 2, 3, None, 5], type="i4"), > pa.array(['foo', 'bar', 'baz', None, "quux"]), > pa.array([1, 2, None, 4, 5], type="f4"), > pa.array([True, None, False, True, False]) > ] > > batch = pa.RecordBatch.from_arrays(data, ['f0', 'f1', 'f2', "f3"]) > sink = pa.BufferOutputStream() > writer = pa.RecordBatchFileWriter(sink, batch.schema) > > for i in range(5): > writer.write_batch(batch) > writer.close() > > buf = sink.getvalue() > f = open("out.dat", "wb") > f.write(buf.to_pybytes()) > f.close() > > and, as I said, I can now successfully read that back from Go. > but I was wondering what's the recommanded way to test for this kind of > round-trip/cross-language thing. > > I tried to play a bit with "integration/integration_test.py" but it fails > with: > > ########################################################## > C++ producing, C++ consuming > ########################################################## > ========================================================== > Testing file /home/binet/work/gonum/src/ > github.com/apache/arrow/integration/data/struct_example.json > ========================================================== > -- Creating binary inputs > /home/binet/work/gonum/src/ > github.com/apache/arrow/integration/../cpp/build/latest/arrow-json-integration-test > --integration --arrow=testdata/75bc5ca6_struct_example.json_as_file > --json=/home/binet/work/gonum/src/ > github.com/apache/arrow/integration/data/struct_example.json > --mode=JSON_TO_ARROW > Command failed: /home/binet/work/gonum/src/ > github.com/apache/arrow/integration/../cpp/build/latest/arrow-json-integration-test > --integration --arrow=testdata/75bc5ca6_struct_example.json_as_file > --json=/home/binet/work/gonum/src/ > github.com/apache/arrow/integration/data/struct_example.json > --mode=JSON_TO_ARROW > With output: > -------------- > Found schema: struct_nullable: struct<f1: int32, f2: string> > > -------------- > ========================================================== > Testing file /home/binet/work/gonum/src/ > github.com/apache/arrow/integration/data/simple.json > ========================================================== > -- Creating binary inputs > /home/binet/work/gonum/src/ > github.com/apache/arrow/integration/../cpp/build/latest/arrow-json-integration-test > --integration --arrow=testdata/b0d388ed_simple.json_as_file > --json=/home/binet/work/gonum/src/ > github.com/apache/arrow/integration/data/simple.json --mode=JSON_TO_ARROW > Command failed: /home/binet/work/gonum/src/ > github.com/apache/arrow/integration/../cpp/build/latest/arrow-json-integration-test > --integration --arrow=testdata/b0d388ed_simple.json_as_file > --json=/home/binet/work/gonum/src/ > github.com/apache/arrow/integration/data/simple.json --mode=JSON_TO_ARROW > With output: > -------------- > Found schema: foo: int32 > bar: double > baz: string > > -------------- > ========================================================== > Testing file testdata/generated_primitive.json > ========================================================== > -- Creating binary inputs > /home/binet/work/gonum/src/ > github.com/apache/arrow/integration/../cpp/build/latest/arrow-json-integration-test > --integration --arrow=testdata/8eacb124_generated_primitive.json_as_file > --json=testdata/generated_primitive.json --mode=JSON_TO_ARROW > Command failed: /home/binet/work/gonum/src/ > github.com/apache/arrow/integration/../cpp/build/latest/arrow-json-integration-test > --integration --arrow=testdata/8eacb124_generated_primitive.json_as_file > --json=testdata/generated_primitive.json --mode=JSON_TO_ARROW > With output: > -------------- > Found schema: bool_nullable: bool > bool_nonnullable: bool not null > int8_nullable: int8 > int8_nonnullable: int8 not null > int16_nullable: int16 > int16_nonnullable: int16 not null > int32_nullable: int32 > int32_nonnullable: int32 not null > int64_nullable: int64 > int64_nonnullable: int64 not null > uint8_nullable: uint8 > uint8_nonnullable: uint8 not null > uint16_nullable: uint16 > uint16_nonnullable: uint16 not null > uint32_nullable: uint32 > uint32_nonnullable: uint32 not null > uint64_nullable: uint64 > uint64_nonnullable: uint64 not null > float32_nullable: float > float32_nonnullable: float not null > float64_nullable: double > float64_nonnullable: double not null > binary_nullable: binary > binary_nonnullable: binary not null > utf8_nullable: string > utf8_nonnullable: string not null > fixedsizebinary_19_nullable: fixed_size_binary[19] > fixedsizebinary_19_nonnullable: fixed_size_binary[19] not null > fixedsizebinary_120_nullable: fixed_size_binary[120] > fixedsizebinary_120_nonnullable: fixed_size_binary[120] not null > > -------------- > > is this supposed to work? > are there reference files already available somewhere? > > cheers, > -s
