Figured it out and cleaned it up. It turns out that I was using
with-handlers oddly, but reading further though the documentation it works
as expected. Here's a new version (generalized to any input-port):
(define (with-gunzip thunk)
(define-values (pipe-from pipe-to) (make-pipe))
(with-handlers ([exn:fail?
(λ (err)
(close-output-port pipe-to)
(close-input-port pipe-from)
(error 'with-gunzip (exn-message err)))])
(gunzip-through-ports (current-input-port) pipe-to)
(close-output-port pipe-to)
(parameterize ([current-input-port pipe-from])
(thunk))
(close-input-port pipe-from)))
If anyone's interested in a more in depth write up / source code for this
and with-gzip:
- writeup: http://blog.jverkamp.com/2013/08/06/adventures-in-racket-gzip/
- source:
https://github.com/jpverkamp/small-projects/tree/master/blog/with-gzip.rkt
On Mon, Aug 5, 2013 at 5:36 PM, JP Verkamp <rac...@jverkamp.com> wrote:
> Thanks! make-pipe isn't something that I've had to use otherwise, so I
> missed the optional parameter. That does certainly seem to help.
>
> Here's my first take of with-input-from-gzipped-file:
>
> (define (with-input-from-gzipped-file filename thunk #:buffer-size
> [buffer-size #f])
> (call-with-input-file filename
> (lambda (file-from)
> (define-values (pipe-from pipe-to) (make-pipe buffer-size))
>
> (thread
> (λ ()
> (gunzip-through-ports file-from pipe-to)
> (close-output-port pipe-to)))
>
> (current-input-port pipe-from)
> (thunk)
> (close-input-port pipe-from))))
>
> The main thing missing is that there's no error handling (where the pipe
> should still be closed). At the very least, if I try to call this on a
> non-gzipped file, it breaks on the gunzip-through-ports line.
> Theoretically, some variation of with-handlers should work (error should
> raise an exn:fail?, yes?), but it doesn't seem to be helping.
>
> Any help with that?
>
> Alternatively, I've now found this:
> http://planet.racket-lang.org/display.ss?package=gzip.plt&owner=soegaard
>
> It seems to do exactly what I need, albeit without the call-with-* forms,
> but that's easy enough to wrap. With some very basic testing, it does seem
> to be buffering though, although it is a bit slower than the above. Not
> enough to cause trouble though.
>
>
> On Mon, Aug 5, 2013 at 4:51 PM, Ryan Culpepper <ry...@ccs.neu.edu> wrote:
>
>> On 08/05/2013 04:29 PM, JP Verkamp wrote:
>>
>>> Is there a nice / idiomatic way to work with gzipped data in a streaming
>>> manner (to avoid loading the rather large files into memory at once). So
>>> far as I can tell, my code isn't doing that. It hangs for a while on the
>>> call to gunzip-through-ports, long enough to uncompress the entire file,
>>> then reads are pretty quick afterwords.
>>>
>>> Here's what I have thus far:
>>>
>>> #lang racket
>>>
>>> (require file/gunzip)
>>>
>>> (define-values (pipe-from pipe-to) (make-pipe))
>>> (with-input-from-file "test.rkt.gz"
>>> (lambda ()
>>> (gunzip-through-ports (current-input-port) pipe-to)
>>> (for ([line (in-lines pipe-from)])
>>> (displayln line))))
>>>
>>
>> You should probably 1) limit the size of the pipe (to stop it from
>> inflating the whole file at once) and 2) put the gunzip-through-ports call
>> in a separate thread. The gunzip thread will block when the pipe is full;
>> when your program reads some data out of the pipe, the gunzip thread will
>> be able to make some more progress. Something like this:
>>
>> (define-values (pipe-from pipe-to) (make-pipe 4000))
>> (with-input-from-file "test.rkt.gz"
>> (lambda ()
>> (thread
>>
>> (lambda ()
>> (gunzip-through-ports (current-input-port) pipe-to)
>> (close-output-port pipe-to)))
>>
>> (for ([line (in-lines pipe-from)])
>> (displayln line))))
>>
>> As an additional problem, that code doesn't actually work.
>>> in-lines seems to be waiting for an eof-object? that
>>> gunzip-through-ports isn't sending. Am I missing something? It ends up
>>> just hanging after reading and printing the file.
>>>
>>
>> The docs don't say anything about closing the port, so you'll probably
>> have to do that yourself. In the code above, I added a call to
>> close-output-port.
>>
>> Ryan
>>
>>
>
____________________
Racket Users list:
http://lists.racket-lang.org/users
