Keith, I think you've cracked it. It seems very simple in retrospect.
Thanks.
On Tuesday, September 24, 2019 at 1:40:02 AM UTC+2, Keith Randall wrote:
>
> In the runtime we use structs like these, but with unsafe.Pointer data
> fields (runtime.stringStruct and runtime.slice). They are much safer to use
> than reflect's types with uintptr Data fields. Unfortunately we can't
> change reflect's types because of the Go 1 compatibility guarantee.
>
> You can do the same thing the runtime does. Something like this would
> work, and hopefully catch any future changes which would break the
> implementation:
>
> type stringHeader struct {
> data unsafe.Pointer
> len int
> }
> type sliceHeader struct {
> data unsafe.Pointer
> len int
> cap int
> }
>
> func init() {
> // Check to make sure string header is what reflect thinks it is.
> // They should be the same except for the type of the Data field.
> if unsafe.Sizeof(stringHeader{}) != unsafe.Sizeof(reflect.StringHeader{}) {
> panic("string layout has changed")
> }
> x := stringHeader{}
> y := reflect.StringHeader{}
> x.data = unsafe.Pointer(y.Data)
> y.Data = uintptr(x.data)
> x.len = y.Len
>
> // same for slice
> }
>
>
>
> On Monday, September 23, 2019 at 9:38:27 AM UTC-7, Robert Engels wrote:
>>
>> As someone that has worked with a lot of similar libraries in the HFT
>> space - things like UnsafeString or FastString in Java I would caution
>> against doing this in Go - especially as proposed here. Taking an immutable
>> object like string and making it mutable by accident is a recipe for
>> disaster. You are almost always better mapping a struct with accessors and
>> letting Go escape analysis perform the work on the stack and keep the
>> safety.
>>
>>
>>
>> On Sep 23, 2019, at 10:09 AM, Francis <francis...@gmail.com> wrote:
>>
>> So I think the current state of unsafe conversions of string <-> []byte
>> is roughly
>>
>> 1. Use the reflect Slice/StringHeader struct. These structs give you
>> clear fields to set and read from. If the runtime representation of a
>> string or []byte ever changes then these structs should change to reflect
>> this (they have a non-backwards compatibility carve out in the comments).
>> But this also means that you run into all these exotic problems because
>> these two structs have a `uintpr` an `unsafe.Pointer` so for a short time
>> the GC won't realise you are reading/writing a pointer. This makes correct
>> use of these structs very difficult.
>> 2. You can just cast between these two types going through
>> `unsafe.Pointer` on the way. This works, because these two types have
>> almost identical layouts. We don't use any uintptr at all and so the GC
>> probably won't get confused. But, if the representations of string or
>> []byte ever change then you code breaks silently, and could have very
>> weird/hard to track down problems.
>>
>> So I don't think `neither is safer than the other` is quite the right
>> description in this context. They both have problems, so they are both
>> not-perfect. But their problems are quite distinct. At the least if we
>> choose one over the other we can describe clearly which set of problems we
>> want to have.
>>
>> My hope was that someone had thought through these problems and could
>> indicate the right way to do it.
>>
>> On a related note. I was trying to track down where the
>> Slice/StringHeader was first introduced. It was a long time ago
>>
>> <Rob Pike> (10 years ago) 29e6eb21ec (HEAD)
>>
>> make a description of the slice header public
>>
>> R=rsc
>> DELTA=18 (3 added, 0 deleted, 15 changed)
>> OCL=31086
>> CL=31094
>>
>> Although I couldn't open that CL in gerrit (I assume user-error). From
>> reading the commit I think the intention was for these header structs to be
>> used for this or similar things. But the data was represented as a uintptr
>> and a comment explicitly states that these structs are of no use without
>> `unsafe.Pointer`. I have seen roughly three other CL which try to change
>> the data field to `unsafe.Pointer` but are rejected because they change the
>> reflect packages API.
>>
>> There is also this issue
>>
>> https://github.com/golang/go/issues/19367
>>
>> Which proposes that Slice/StringHeader be moved/duplicated in unsafe and
>> use `unsafe.Pointer`. As far as I can tell once we have this then all the
>> subtle problems disappear and lovingly crafted examples like
>>
>> https://github.com/m3db/m3x/blob/master/unsafe/string.go#L62
>>
>> just become the right way to do it.
>>
>> Until then maybe we should just rely on the structural similarities
>> between the two types and cast between them. This seems especially
>> appealing as Jan pointed out above that at least one of the hypothetical
>> problems isn't hypothetical at all.
>>
>>
>> On Monday, 23 September 2019 12:43:34 UTC+2, kortschak wrote:
>>>
>>> Any particular reason for that? Neither is safer than the other and
>>> it's not clear to me that you can actually achieve the goal of having a
>>> compile-time check for the correctness of this type of conversion.
>>>
>>> On Mon, 2019-09-23 at 02:36 -0700, fra...@adeven.com wrote:
>>> > But this relies on a string's representation being the same as, but a
>>> > bit smaller thabn, a []byte. I would prefer to use
>>> > the Slice/StringHeader.
>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golan...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/422ca2bd-d6c8-4ebe-9578-8dd3cd8317e9%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/golang-nuts/422ca2bd-d6c8-4ebe-9578-8dd3cd8317e9%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/00bdea81-4a2e-45ab-9775-c398cbea7e2a%40googlegroups.com.
