The problem is that when encoding, even with json.Encoder, the entire object is marshaled, into memory, before it is written to the io.Writer. My proposal allows writing the JSON output immediately, rather than waiting for the entire process to complete successfully first.
The same problem occurs in reverse--when reading a large JSON response, you cannot begin processing the result until the entire result is received. To anchor these abstract concepts to real life, let me offer an example of each where this is quite painful: When writing a CouchDB document, it may contain an arbitrary amount of data, possibly even including BASE64-encoded attachments. For some extreme cases, these documents may be multiple mb. Dozens or hundreds of kb is not at all unusual. A typical use case may have 10k of normal JSON, with an additional 200k of, say, an image. The current JSON implementation buffers this entire payload, ensures there are no marshaling errors, then writes to the `io.Writer`. My proposal would allow writing immediately, with no need to buffer 100s of kb of JSON. In the case of CouchDB, the reverse may actually be more harmful (and I've already gone to some lengths to mitigate the worst of it, using json.Decoder's tokenizer API): A typical query returns multiple documents (which, again, may be up to hundreds of kb each). With the existing implementation, one must read the entire resultset from the network, before parsing the first document. My proposal would make it possible to begin reading the individual JSON documents (and indeed, even individual parts of said document), without waiting for the entire result to be buffered. On Friday, August 9, 2019 at 10:10:23 PM UTC+2, Robert Engels wrote: > > I'm sorry, maybe I didn't understand your original concern. There is an > example of doing stream based processing in the json package (using > Decoder). > > How is this not sufficient? > > The only problem I see with it is that it doesn't allow error > correction/continuation, but in the modern world that seems rather rare (or > very difficult to do well). > > -----Original Message----- > From: Jonathan Hall > Sent: Aug 9, 2019 11:00 AM > To: golang-nuts > Subject: Re: [go-nuts] RFC for opt-in streaming support in encoding/json > package > > An interesting observation. > > Although in a sense, we already have the decoding half of that low-level > streaming API exposed by way of the `json.Decoder` type. > > Would you be suggesting that a separate, stream-based API makes sense even > within the stdlib? > > I'm not really sure what that separate API would look like, or do > differently than my proposal (I'm open to new ideas, though). > > Given that the "Go way" of handling streams is with io.Reader/io.Writer > (as opposed to events, for example), and the internal implementation of > `json/encoding` is already so close to that, I wonder if the APIs would end > up looking very much the same, anyway. > > Jonathan > > > On Friday, August 9, 2019 at 5:53:41 PM UTC+2, Robert Engels wrote: >> >> In other environments (e.g. Java), the streaming processors are distinct >> from the instance oriented - usually for good reason as the APIs are very >> different - the former is usually event based, and the latter returns >> realized instances as a whole or an error. The streaming processors can >> often skip ill-formed entities, and/or have them manipulated during >> decoding. >> >> -----Original Message----- >> From: Jonathan Hall >> Sent: Aug 9, 2019 10:38 AM >> To: golang-nuts >> Subject: Re: [go-nuts] RFC for opt-in streaming support in encoding/json >> package >> >> Thanks for the reply. My responses inline below. >> >> On Friday, August 9, 2019 at 5:14:52 PM UTC+2, burak serdar wrote: >>> >>> On Fri, Aug 9, 2019 at 8:53 AM Jonathan Hall <fli...@flimzy.com> wrote: >>> > >>> > Can you say more? Better in which way? >>> >>> Better in the way that it wouldn't change existing code. >> >> >> That doesn't seem like a benefit, in its own right >> >> I understand the desire not to just change code for its own sake, or add >> extra features nobody needs. But people have been asking for these types of >> features for several years. This doesn't seem like a frivolous code change >> to me. >> >> >>> Also, I think >>> the use cases for existing and proposed json encoders/decoders are >>> different enough to justify a separate implementation. >> >> >> I don't think I agree with this. >> >> The proposal deals with a subset of current use cases, but not, strictly >> speaking, a _different set_ of use cases. And the number of commentators on >> the issues linked above, I think lends weight to the idea that the use >> cases this proposal addresses are not insignificant, or fundamentally >> "different". >> >> If I were to fork the standard `encoding/json` library, and add my >> proposed functionality, the code would still be 95% the same. Standard >> reasons for sharing code apply, as far as I can tell. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golan...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/golang-nuts/df688733-a2e9-4bc8-aa7b-09267827007a%40googlegroups.com >> >> <https://groups.google.com/d/msgid/golang-nuts/df688733-a2e9-4bc8-aa7b-09267827007a%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> >> >> >> -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golan...@googlegroups.com <javascript:>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/718e2b9e-39e2-44f2-9308-8c94e31afbff%40googlegroups.com > > <https://groups.google.com/d/msgid/golang-nuts/718e2b9e-39e2-44f2-9308-8c94e31afbff%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > > > > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/10b996d3-f46c-4798-b9af-958183ec82f7%40googlegroups.com.
