Sure, I'll share my code and what I'm trying to do. Thank you all for the
help so far. My program reads the sql table's metadata to determine the
type and length of each column in the table. These values are encoded as
varint of unsigned bigendian integers. I already validated the expected
values match the tables's actual data type/size.
package main
import (
"encoding/binary"
"fmt"
)
func main() {
// SQLite format 3, sample DB file record header
//Expected: 7 23 27 27 1 199
// |-------| |-------| |-------| |-------| |------|
|----------------|
inputs := []byte{0x07, 0x17, 0x1b, 0x1b, 0x01, 0x81, 0x47}
offset := 0
for remaining := len(inputs); remaining > 0; {
d, n := binary.Uvarint(inputs[offset:])
if n <= 0 {
break
}
remaining -= n
offset += n
fmt.Println(d, n)
// Actual output
// 7 1
// 23 1
// 27 1
// 27 1
// 1 1
// 9089 2
}
}
I now see why I get the 9089 figure after looking at Uvarint source code
(https://cs.opensource.google/go/go/+/refs/tags/go1.25.1:src/encoding/binary/varint.go):
func Uvarint(buf []byte) (uint64, int) {
var x uint64
var s uint
for i, b := range buf {
if i == MaxVarintLen64 {
// Catch byte reads past MaxVarintLen64.
// See issue https://golang.org/issues/41185
return 0, -(i + 1) // overflow
}
if b < 0x80 {
if i == MaxVarintLen64-1 && b > 1 {
return 0, -(i + 1) // overflow
}
return x | uint64(b)<<s, i + 1
}
x |= uint64(b&0x7f) << s
s += 7
}
return 0, 0
}
Here I see the bits after the first byte are left-shifted by 7 before
concatenating and left-padding.
My solution so far has been to create custom uvarint function that performs
the left-shift before the concat, preserving the byte order.
func Uvarint(buf []byte) (uint64, int) {
var x uint64
var s uint
for i, b := range buf {
if i == MaxVarintLen64 {
// Catch byte reads past MaxVarintLen64.
// See issue https://golang.org/issues/41185
return 0, -(i + 1) // overflow
}
if b < 0x80 {
if i == MaxVarintLen64-1 && b > 1 {
return 0, -(i + 1) // overflow
}
x <<= s
return x | uint64(b), i + 1
}
x <<= s
x |= uint64(b&0x7f)
s += 7
}
return 0, 0
}
I would prefer to use the go library's functions if at all possible rather
than make my own but so far I haven't found alternatives or even
discussions on this topic. If anything's unclear let me know. Cheers.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion visit
https://groups.google.com/d/msgid/golang-nuts/84756c8b-a319-4ab8-8bfb-01375e2e5e88n%40googlegroups.com.
