Hi
I hope this is the right forum for posting this kind of question:
Debugging application performance, I came across JSON stream parsing
causing a large number of mem allocations. Here's a snippet of code, which
exemplifies my findings:
var (
noSpace = `{"Key":"StringValue"}`
oneSpace = `{"Key":"StringValue"} `
twoSpace = `{"Key":"StringValue" } `
threeSpace = `{"Key": "StringValue" } `
fourSpace = `{"Key" : "StringValue" } `
fiveSpace = `{ "Key" : "StringValue" } `
)
func TryJSONStreamParsing(b *testing.B, jsonMsg string) {
for i := 0; i < b.N; i++ {
dec := json.NewDecoder(bytes.NewReader([]byte(jsonMsg)))
t, err := dec.Token()
if err != nil {
b.Fatal(err)
}
if t != json.Delim('{') {
b.Fatal("No {")
}
for dec.More() {
t, err = dec.Token()
if err != nil {
b.Fatal(err)
}
if t == nil {
b.Fatal("no t")
}
}
}
}
func BenchmarkJSONNoSpace(b *testing.B) {
TryJSONStreamParsing(b, noSpace)
}
func BenchmarkJSONOneSpace(b *testing.B) {
TryJSONStreamParsing(b, oneSpace)
}
func BenchmarkJSONTwoSpace(b *testing.B) {
TryJSONStreamParsing(b, twoSpace)
}
func BenchmarkJSONThreeSpace(b *testing.B) {
TryJSONStreamParsing(b, threeSpace)
}
func BenchmarkJSONFourSpace(b *testing.B) {
TryJSONStreamParsing(b, fourSpace)
}
func BenchmarkJSONFiveSpace(b *testing.B) {
TryJSONStreamParsing(b, fiveSpace)
}
Here's an example output for running that:
BenchmarkJSONNoSpace-4 500000 3166 ns/op 1304
B/op 24 allocs/op
BenchmarkJSONOneSpace-4 500000 2995 ns/op 1304 B/op
24 allocs/op
BenchmarkJSONTwoSpace-4 500000 2423 ns/op 1200 B/op
18 allocs/op
BenchmarkJSONThreeSpace-4 1000000 2533 ns/op 1200 B/op
18 allocs/op
BenchmarkJSONFourSpace-4 1000000 2145 ns/op 1104
B/op 12 allocs/op
BenchmarkJSONFiveSpace-4 1000000 2069 ns/op 1104
B/op 12 allocs/op
Depending on where the whitespace is put in the JSON message, there are
very different count of memory allocs.
When comparing memory profiles of the worst case vs. the best case...
Top 7 items in worst case:
2255615 27.06% 27.06% 3074826 36.89% encoding/json.(*scanner).error
2064414 24.77% 51.83% 6761536 81.12% encoding/json.(*Decoder).Token
1573541 18.88% 70.71% 8335077 100% .....TryJSONStreamParsing
1081360 12.97% 83.68% 1081360 12.97%
encoding/json.(*decodeState).literalStore
819211 9.83% 93.51% 819211 9.83% strconv.quoteWith
540936 6.49% 100% 540936 6.49% encoding/json.(*Decoder).refill
0 0% 100% 4156186 49.86% encoding/json.(*Decoder).Decode
Top 7 items in best case:
3932219 41.93% 41.93% 6762044 72.10% encoding/json.(*Decoder).Token
2616008 27.89% 69.82% 9378052 100% .....TryJSONStreamParsing
1835036 19.57% 89.39% 1835036 19.57%
encoding/json.(*decodeState).literalStore
994789 10.61% 100% 994789 10.61% encoding/json.(*Decoder).refill
0 0% 100% 1835036 19.57% encoding/json.(*Decoder).Decode
0 0% 100% 994789 10.61% encoding/json.(*Decoder).peek
... it seems that the more compact JSON causes the Decoder to generate
errors (which are not surfaced and which causes allocations). While the
JSON with some extra spaces causes a cleaner parse.
Does anyone know why the JSON parser consider missing whitespace after
values for and error?
Or am I using the json.Decoder incorrectly?
Cheers
Tonny
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
