Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-10-01 Thread Greg Ewing via Python-list
On 2/10/24 12:26 pm, avi.e.gr...@gmail.com wrote: The real problem is how the JSON is set up. If you take umpteen data structures and wrap them all in something like a list, then it may be a tad hard to stream as you may not necessarily be examining the contents till the list finishes gigabytes l

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-10-01 Thread Dan Sommers via Python-list
On 2024-10-01 at 23:03:01 +0200, Left Right wrote: > > If I recognize the first digit, then I *can* hand that over to an > > external function to accumulate the digits that follow. > > And what is that external function going to do with this information? > The point is you didn't parse anything

RE: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-10-01 Thread AVI GROSS via Python-list
This discussion has become less useful. E can all agree that in Computer Science, real infinities are avoided, and frankly, need not be taken seriously in any serious program. You can store all kinds of infinities quite compactly as in a transcendental number you can derive to as many decimal poi

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-10-01 Thread Greg Ewing via Python-list
On 2/10/24 10:03 am, Left Right wrote: Consider also an interesting consequence of SCSI not being able to have infinite words: this means, besides other things that fsync() is nonsense! :) If you aren't familiar with the concept: UNIX filesystem API suggests that it's possible to destage arbitrar

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-10-01 Thread Greg Ewing via Python-list
On 1/10/24 8:34 am, Left Right wrote: You probably forgot that it has to be _streaming_. Suppose you parse the first digit: can you hand this information over to an external function to process the parsed data? -- No! because you don't know the magnitude yet. By that definition of "streaming",

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-10-01 Thread Left Right via Python-list
> If I recognize the first digit, then I *can* hand that over to an > external function to accumulate the digits that follow. And what is that external function going to do with this information? The point is you didn't parse anything if you just sent the digit. You just delegated the parsing furt

[RELEASE] Python 3.13.0rc3 and 3.12.7 released.

2024-10-01 Thread Thomas Wouters via Python-list
This is not the release you’re looking for… (unless you’re looking for 3.12.7.) Because no plan survives contact with reality, instead of the actual Python 3.13.0 release we have a new Python 3.13 release candidate today. Python 3.13.0rc3 rolls back the incremental cyclic garbage collector (GC),

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-10-01 Thread Dan Sommers via Python-list
On 2024-09-30 at 21:34:07 +0200, Regarding "Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API," Left Right via Python-list wrote: > > What am I missing? Handwavingly, start with the first digit, and as > > long as the next character is a digit, multipliy th

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-10-01 Thread Dan Sommers via Python-list
On 2024-09-30 at 18:48:02 -0700, Keith Thompson via Python-list wrote: > 2qdxy4rzwzuui...@potatochowder.com writes: > [...] > > In Common Lisp, you can write integers as #nnR[digits], where nn is the > > decimal representation of the base (possibly without a leading zero), > > the # and the R are

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-10-01 Thread Left Right via Python-list
> What am I missing? Handwavingly, start with the first digit, and as > long as the next character is a digit, multipliy the accumulated result > by 10 (or the appropriate base) and add the next value. Oh, and handle > scientific notation as a special case, and perhaps fail spectacularly > instea

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-10-01 Thread Keith Thompson via Python-list
2qdxy4rzwzuui...@potatochowder.com writes: [...] > In Common Lisp, you can write integers as #nnR[digits], where nn is the > decimal representation of the base (possibly without a leading zero), > the # and the R are literal characters, and the digits are written in > the intended base. So the inp