So in short, you are saying that the byte sequence 0x81, 0x47 written by SQLite decodes by binary.Uvarint to 9089, but you wanted it to decode to 199.
What this means is: the encoding that SQLite has chosen to use is *not* the varint as defined by protobuf (and implemented by the Go standard library). And therefore, you do indeed need to write your own custom decoding function. The SQLite file format is defined here: https://www.sqlite.org/fileformat.html *A variable-length integer or "varint" is a static Huffman encoding of 64-bit twos-complement integers that uses less space for small positive values. A varint is between 1 and 9 bytes in length. The varint consists of either zero or more bytes which have the high-order bit set followed by a single byte with the high-order bit clear, or nine bytes, whichever is shorter. The lower seven bits of each of the first eight bytes and all 8 bits of the ninth byte are used to reconstruct the 64-bit twos-complement integer. Varints are big-endian: bits taken from the earlier byte of the varint are more significant than bits taken from the later bytes.* And for protobuf, see: https://protobuf.dev/programming-guides/encoding/#varints On Saturday, 4 October 2025 at 01:31:25 UTC+1 R. Men wrote: > Sure, I'll share my code and what I'm trying to do. Thank you all for the > help so far. My program reads the sql table's metadata to determine the > type and length of each column in the table. These values are encoded as > varint of unsigned bigendian integers. I already validated the expected > values match the tables's actual data type/size. > > package main > > import ( > "encoding/binary" > "fmt" > ) > > func main() { > // SQLite format 3, sample DB file record header > //Expected: 7 23 27 27 1 199 > // |-------| |-------| |-------| |-------| |------| > |----------------| > inputs := []byte{0x07, 0x17, 0x1b, 0x1b, 0x01, 0x81, 0x47} > offset := 0 > for remaining := len(inputs); remaining > 0; { > d, n := binary.Uvarint(inputs[offset:]) > if n <= 0 { > break > } > > remaining -= n > offset += n > fmt.Println(d, n) > > // Actual output > // 7 1 > // 23 1 > // 27 1 > // 27 1 > // 1 1 > // 9089 2 > } > } > > I now see why I get the 9089 figure after looking at Uvarint source code ( > https://cs.opensource.google/go/go/+/refs/tags/go1.25.1:src/encoding/binary/varint.go > ): > > func Uvarint(buf []byte) (uint64, int) { > var x uint64 > var s uint > for i, b := range buf { > if i == MaxVarintLen64 { > // Catch byte reads past MaxVarintLen64. > // See issue https://golang.org/issues/41185 > return 0, -(i + 1) // overflow > } > if b < 0x80 { > if i == MaxVarintLen64-1 && b > 1 { > return 0, -(i + 1) // overflow > } > return x | uint64(b)<<s, i + 1 > } > x |= uint64(b&0x7f) << s > s += 7 > } > return 0, 0 > } > > Here I see the bits after the first byte are left-shifted by 7 before > concatenating and left-padding. > My solution so far has been to create custom uvarint function that > performs the left-shift before the concat, preserving the byte order. > > func Uvarint(buf []byte) (uint64, int) { > var x uint64 > var s uint > for i, b := range buf { > if i == MaxVarintLen64 { > // Catch byte reads past MaxVarintLen64. > // See issue https://golang.org/issues/41185 > return 0, -(i + 1) // overflow > } > if b < 0x80 { > if i == MaxVarintLen64-1 && b > 1 { > return 0, -(i + 1) // overflow > } > x <<= s > return x | uint64(b), i + 1 > } > x <<= s > x |= uint64(b&0x7f) > s += 7 > } > return 0, 0 > } > > I would prefer to use the go library's functions if at all possible rather > than make my own but so far I haven't found alternatives or even > discussions on this topic. If anything's unclear let me know. Cheers. > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/e2f4bb5e-81cf-48c7-844d-93788f271440n%40googlegroups.com.
