I’m trying to understand how to parse a Buffer into a Schema, but using
using pdb with Python and reading the TS/Python/C++ Arrow source hasn’t
really cleared much up for me. Nor has studying
https://arrow.apache.org/docs/ipc.html
Here’s are the steps of what I’ve tried (the code is Julia, but only
because I’m trying to do this natively, rather than wrap the Arrow C code):
# Thrift API method returning a struct (sm_buf, sm_size, df_buf, df_size)
(works as expected)
julia> tdf = sql_execute_df(conn, "select * from flights_2008_7m limit
1000", 0, 0, 1000)
MapD.TDataFrame(UInt8[0xba, 0x58, 0x1b, 0x3d], 93856, UInt8[0xab, 0xd7,
0x7e, 0x50], 188880)
# Wrap shared memory into julia array, based on handle and size (works as
expected)
julia> sm_buf = MapD.load_buffer(tdf.sm_handle, tdf.sm_size) #wrapper using
shmget/shmat
93856-element Array{UInt8,1}:
0x2c
0x16
0x00
0x00
0x14
0x00
0x00
0x00
0x00
0x00
⋮
0x20
0x74
0x6f
0x20
0x4d
0x66
0x72
0x00
0x00
At this point, walking through an similar Python process, I know that
sm_buf represents
- type: Schema
- metadata length: 5676
- body_length: 0
Where I’m confused is how to proceed.
I am getting metadata_length by reinterpreting the first 4-bytes as Int32.
julia> mlen = reinterpret(Int32, sm_buf[1:4])[1]
5676
I then assumed that I could start at byte 5 and take the next `mlen-1`
bytes:
julia> metadata = sm_buf[5:5+mlen-1]
5676-element Array{UInt8,1}:
0x14
0x00
0x00
0x00
0x00
0x00
0x00
0x00
0x0c
0x00
⋮
0x79
0x65
0x61
0x72
0x00
0x00
0x00
0x00
0x00
Am I on the right track here? I *think* that my `metadata` variable above
is a FlatBuffer, but how do I know what its structure is? Additionally,
what am I supposed to do with all of the bytes that haven’t been read from
`sm_buf` yet? `sm_buf` is 93856 bytes and I’ve only read the first 4 bytes
+ metadata length, leaving some 88,000 bytes not processed yet.
Any help would be greatly appreciated here. Please note that I’m not asking
for julia coding help, but rather what the Arrow bytes actually mean/their
structure and how to process them further.
Thanks,
Randy Zwitch
