Thanks Daniel, that seems on the right track.
The driver api itself is a C library, without source code.
>From the driver api docs:
/**
* get_column() fills the supplied buffer with the value fetched for the
* specified column. *For A_BINARY and A_STRING * data types,*
* * value->buffer will point to an internal buffer associated with the
result set.*
* The content of the pointer buffer should not be relied on or altered
* as it will change when a new row is fetched or when the result set
* object is freed. Users should copy the data out of those pointers into
their
* own buffers. The length field indicates the number of valid characters
that
* buffer->buffer points to. The data returned in buffer->buffer is not
* null-terminated. This function fetches all of the returned value from the
server.
* For example, if the column contains
* a 2GB blob, this function will attempt to allocate enough memory to hold
that value.
* If this is not desired, get_data() should be used instead.
*
* returns 1 on success or 0 for failure. A failure can happen if any of the
parameters is invalid
* or if there is not enough memory to retrieve the full value from the
server.
*/
get_column( a_stmt * stmt, u32 col_index, a_data_value * value );
Here's an outline of a C implementation. This is what I want to replace
with Go:
typedef struct a_data_value
{
char * buffer; //allocated by caller except for binary & string type
size_t buffer_size; //set by caller except for binary & string type
size_t * length; //The number of valid bytes in the buffer. Must be
less than buffer_size
a_data_type type;
bool * is_null;
} a_data_value;
char * blob;
size_t size;
a_data_value value;
err = get_column( stmt_handle, column_index, &value )
char * buffer = (char *)emalloc( *(value.length) );
memcpy( buffer, val.buffer, *(value.length) );
size = *(val.length)
An attempt in Go, occasionally panics:
value := new(C.a_sqlany_data_value)
C.sqlany_get_column(r.stmt.ptr, col_index, value)
blob := C.GoBytes(unsafe.Pointer(value.buffer), C.int(value.length))
Another attempt, allocating value in c memory, here the call to free
occasionally panics:
value := (*C.a_data_value)(C.malloc(C.sizeof_a_data_value))
defer C.free(unsafe.Pointer(value))
C.sqlany_get_column(r.stmt.ptr, col_index, value)
blob := C.GoBytes(unsafe.Pointer(value.buffer), C.int(value.length))
Assigning the value directly appears to panic more frequently:
blob = (*[1 << 30]byte)(unsafe.Pointer(r.value.buffer))[:*value.length:*
value.length]
Note:
- null checks and get_column error check aren't shown here.
- each query returns many rows each with a single binary column value up
to around 10Mb
Any suggestions appreciated!
Thanks,
Mark
On Monday, 13 February 2017 12:51:28 UTC+13, Daniel Theophanes wrote:
>
> It is difficult to be certain, but I would put my bets on your sql
> anywhere driver may not be handling the c data lifetimes correctly, or
> might not be keeping a reference. It is difficult to tell without the
> driver code.
>
> Thanks, -Daniel
>
> On Saturday, February 11, 2017 at 7:27:11 PM UTC-8, Mark Crook wrote:
>>
>> Hello,
>>
>> I'm seeing a panic in go version go1.8rc3 linux/amd64 while trying to
>> write a database driver using cgo. The go code runs in a docker container,
>> the database in another.
>>
>> Generally it works reliably, but occasionally, during repeated retrieval
>> of binary data (photos approx 100K to 2M), the code panics.
>>
>> I haven't isolated a reproducible example, but I think it's related to
>> this code:
>>
>>
>> value := new(C.a_sqlany_data_value)
>>
>> //load the column value binary data into value.buffer
>>
>> C.sqlany_get_column(r.stmt.ptr, C.sacapi_u32(i), value)
>>
>>
>> //copy the value buffer from C to Go
>>
>> val = C.GoBytes(unsafe.Pointer(value.buffer), C.int(value.length))
>>
>>
>> I haven't seen the panic if the last line is commented out.
>>
>> Below is a stack trace. Perhaps it panics when a garbage collection is
>> triggered while waiting for C.GoBytes to return?
>>
>> Any suggestions?
>>
>> Thank you,
>> Mark
>>
>>
>>
>> runtime: unexpected return pc for runtime.sigpanic called from
>> 0xc420e1a000
>> fatal error: unknown caller pc
>>
>> runtime stack:
>> runtime.throw(0x59a90e, 0x11)
>> /go/src/runtime/panic.go:596 +0x95
>> runtime.gentraceback(0xffffffffffffffff, 0xc42002a5d8, 0x0, 0xc4200c49c0,
>> 0x0, 0x0, 0x7fffffff, 0x7fb675ad5cb8, 0x0, 0x0, ...)
>> /go/src/runtime/traceback.go:317 +0x13ea
>> runtime.scanstack(0xc4200c49c0, 0xc420021828)
>> /go/src/runtime/mgcmark.go:842 +0x265
>> runtime.newstack(0x0)
>> /go/src/runtime/stack.go:1064 +0x41e
>> runtime.morestack()
>> /go/src/runtime/asm_amd64.s:398 +0x86
>>
>> goroutine 11 [GC worker (idle) (scan)]:
>> runtime.assertE2I2(0x56c220, 0x569e20, 0x841d20, 0xc42002a6b0,
>> 0xc4200c49e0, 0xc42002a640)
>> /go/src/runtime/iface.go:289 +0xb2 fp=0xc42002a5e0 sp=0xc42002a5d8
>> runtime.preprintpanics(0xc42002a6b0)
>> /go/src/runtime/panic.go:389 +0x9f fp=0xc42002a650 sp=0xc42002a5e0
>> panic(0x569e20, 0x841d20)
>> /go/src/runtime/panic.go:528 +0x1ab fp=0xc42002a6e8 sp=0xc42002a650
>> runtime.panicmem()
>> /go/src/runtime/panic.go:63 +0x5e fp=0xc42002a708 sp=0xc42002a6e8
>> runtime.sigpanic()
>> /go/src/runtime/signal_unix.go:290 +0x29f fp=0xc42002a758 sp=0xc42002a708
>> created by runtime.gcBgMarkStartWorkers
>> /go/src/runtime/mgc.go:1410 +0x98
>>
>> goroutine 1 [chan receive]:
>> testing.(*T).Run(0xc420072820, 0x598622, 0x8, 0x5a3718, 0xc420051d01)
>> /go/src/testing/testing.go:698 +0x2f4
>> testing.runTests.func1(0xc420072820)
>> /go/src/testing/testing.go:881 +0x67
>> testing.tRunner(0xc420072820, 0xc420051de0)
>> /go/src/testing/testing.go:657 +0x96
>> testing.runTests(0xc42000cb60, 0x8461e0, 0x4, 0x4, 0xc420051ef8)
>> /go/src/testing/testing.go:887 +0x2c1
>> testing.(*M).Run(0xc420051f20, 0xc420051f20)
>> /go/src/testing/testing.go:822 +0xfc
>> main.main()
>> github.com/mdcnz/sqlanywhere/_test/_testmain.go:48 +0xf7
>>
>> goroutine 17 [syscall, 6 minutes, locked to thread]:
>> runtime.goexit()
>> /go/src/runtime/asm_amd64.s:2197 +0x1
>>
>> goroutine 231 [GC assist wait]:
>> database/sql.convertAssign(0x5517a0, 0xc4200c20a0, 0x55a040,
>> 0xc42000c0e0, 0x85b5c0, 0x4535c0)
>> /go/src/database/sql/convert.go:161 +0x1a22
>> database/sql.(*Rows).Scan(0xc42001c6c0, 0xc420041f68, 0x1, 0x1, 0x0, 0x0)
>> /go/src/database/sql/sql.go:2342 +0xca
>> github.com/mdcnz/sqlanywhere.TestBlob(0xc4200728f0)
>> /root/gome/src/github.com/mdcnz/sqlanywhere/test.go:26 +0x2a5
>> testing.tRunner(0xc4200728f0, 0x5a3718)
>> /go/src/testing/testing.go:657 +0x96
>> created by testing.(*T).Run
>> /go/src/testing/testing.go:697 +0x2ca
>>
>> goroutine 233 [chan receive]:
>> database/sql.(*Rows).awaitDone(0xc42001c6c0, 0x84b240, 0xc42047e080)
>> /go/src/database/sql/sql.go:2091 +0x54
>> created by database/sql.(*Rows).initContextClose
>> /go/src/database/sql/sql.go:2086 +0x92
>>
>> goroutine 232 [chan receive]:
>> database/sql.(*DB).connectionOpener(0xc42008c210)
>> /go/src/database/sql/sql.go:835 +0x4a
>> created by database/sql.Open
>> /go/src/database/sql/sql.go:580 +0x1c8
>> exit status 2
>>
>>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
