Hi P
ragya,
While Robert Engles is probably correct in identifying your bottleneck, if it does turn out to be slow JSON parsing here are a few things
you can look at.
1. You mention you have to use a map because of response keys not being fixed. Be aware that you
do not need to create a struct to match the full JSON. You can easily just create a flyweight struct which is a subset if for your use-case containing only the specific parts you need,
for example.
2. If you optionally need to parse portions of the JSON, but not always, you can use
json.RawMessage to capture the properties you don't always need to parse, and then
parse them only when you need to.
3. You can also declare an
.UnmarshalJSON() method on a struct you are passing to
json.Unmarshal(), or on any struct that is a property of your top level object, and then be
fully in control of parsing the data, meaning you could combine with (a)
json.RawMessage property(s) to part just the non-fixed keys as a map, and only for use-cases when you need to.
4. And finally, if you are willing to venture out from the Go standard library, there are numerous open-source packages that claim to be much faster than the standard library. Usually you want to stick with the Go standard library so other Go developers will be familiar with it and to minimize dependencies that could potentially introduce a bug or security hole. However, if you really need better performance it might be worth the added dependency.
While I cannot validate the performance claims of any of these packages, I can provide you with this list of packages that claim better JSON unmarshalling performance:
- https://github.com/valyala/fastjson
- https://github.com/bytedance/sonic
- https://github.com/buger/jsonparser
- https://github.com/mailru/easyjson
- https://github.com/sugawarayuuta/sonnet
- https://github.com/json-iterator/go
- https://github.com/segmentio/encoding
- https://github.com/goccy/go-json
- https://github.com/simdjson/simdjson
Benchmarks from authors of some of the packages
(so take with a grain of salt):- https://github.com/sugawarayuuta/benchmark
- https://blog.min.io/simdjson-go-parsing-gigabyes-of-json-per-second-in-go/
And benchmarks from someone who is not an author of one of those packages:
- https://kokizzu.blogspot.com/2022/12/map-to-struct-and-struct-to-map-golang.html
Hope this helps.
-Mike
On Friday, March 8, 2024 at 4:15:25 PM UTC-5 Robert Engels wrote:
It is highly unlikely that the Go marshaling is the cause. I’m guessing you are probably not using a buffered output stream.
PHP is written in C but depending on what you are doing it is either a script or calling out to another process (afaik)
Hi,
"I am facing an issue in my Go code. I converted my read API from PHP to Go, but the response time of Go API is higher than PHP. We are returning the response in JSON format, and the response contains around 30k keys, which is taking too much time in JSON marshaling. We are adding all the response in map format as we can't use struct because response keys are not fixed and vary from user to user.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/7345c383-1529-489c-b156-72601bd80530n%40googlegroups.com.
