Sure, I'll share my code and what I'm trying to do. Thank you all for the help so far. My program reads the sql table's metadata to determine the type and length of each column in the table. These values are encoded as varint of unsigned bigendian integers. I already validated the expected values match the tables's actual data type/size.
package main
import (
"encoding/binary"
"fmt"
)
func main() {
// SQLite format 3, sample DB file record header
//Expected: 7 23 27 27 1 199
// |-------| |-------| |-------| |-------| |------| |----------------|
inputs := []byte{0x07, 0x17, 0x1b, 0x1b, 0x01, 0x81, 0x47}
offset := 0
for remaining := len(inputs); remaining > 0; {
d, n := binary.Uvarint(inputs[offset:])
if n <= 0 {
break
}
remaining -= n
offset += n
fmt.Println(d, n)
// Actual output
// 7 1
// 23 1
// 27 1
// 27 1
// 1 1
// 9089 2
}
}
I now see why I get the 9089 figure after looking at Uvarint source code (https://cs.opensource.google/go/go/+/refs/tags/go1.25.1:src/encoding/binary/varint.go):
func Uvarint(buf []byte) (uint64, int) {
var x uint64
var s uint
for i, b := range buf {
if i == MaxVarintLen64 {
// Catch byte reads past MaxVarintLen64.
// See issue https://golang.org/issues/41185
return 0, -(i + 1) // overflow
}
if b < 0x80 {
if i == MaxVarintLen64-1 && b > 1 {
return 0, -(i + 1) // overflow
}
return x | uint64(b)<<s, i + 1
}
x |= uint64(b&0x7f) << s
s += 7
}
return 0, 0
}
Here I see the bits after the first byte are left-shifted by 7 before concatenating and left-padding.
My solution so far has been to create custom uvarint function that performs the left-shift before the concat, preserving the byte order.
func Uvarint(buf []byte) (uint64, int) {
var x uint64
var s uint
for i, b := range buf {
if i == MaxVarintLen64 {
// Catch byte reads past MaxVarintLen64.
// See issue https://golang.org/issues/41185
return 0, -(i + 1) // overflow
}
if b < 0x80 {
if i == MaxVarintLen64-1 && b > 1 {
return 0, -(i + 1) // overflow
}
x <<= s return x | uint64(b), i + 1
}
x <<= s x |= uint64(b&0x7f)
s += 7
}
return 0, 0
}
I would prefer to use the go library's functions if at all possible rather than make my own but so far I haven't found alternatives or even discussions on this topic. If anything's unclear let me know. Cheers.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
[email protected].
To view this discussion visit
https://groups.google.com/d/msgid/golang-nuts/84756c8b-a319-4ab8-8bfb-01375e2e5e88n%40googlegroups.com.
