Looks like this issue has been dead for quite some time. I'd like to resurrect it :) Any chance configurable window sizes for flate might be on the horizon?
On Tuesday, February 10, 2015 at 7:24:25 PM UTC-5, Rob Napier wrote: > > Looks like one was already open: https://github.com/golang/go/issues/3155 > > I've added some comments. > > -Rob > > On Tuesday, February 10, 2015 at 4:49:16 PM UTC-5, Ivan Krasin wrote: >> >> Hi Rob, >> >> while I don't remember all the details, I would not be surprised if the >> code had some assumptions about the window size to be large enough. >> >> I have time to look at that, if the Go team agrees it's something we want >> to have in the standard library. I would also recommend to file an issue >> for tracking, but my recommendation is completely optional. >> >> krasin >> >> On Tue, Feb 10, 2015 at 1:36 PM, Rob Napier <robn...@gmail.com> wrote: >> >>> (Copying Ivan Krasin because I believe this is his code originally.) >>> >>> This has turned out to be a bit harder than I hoped. I've been running >>> into trouble with deflate when the window size is smaller than the maximum >>> match length (258). >>> >>> If flate.logWindowSize == 8 (which is the minimum allowed by spec), then >>> the window will be 256 bytes and flate.deflate() will crash when the window >>> is adjusted because the index will be negative: >>> >>> panic: runtime error: index out of range >>> >>> goroutine 1 [running]: >>> ztest/compress/flate.(*compressor).deflate(0x2081c8900) >>> /Users/rnapier/go/src/ztest/compress/flate/deflate.go:235 +0xff9 >>> >>> I've posted example code to demonstrate: >>> https://github.com/rnapier/ztest >>> >>> I'm still digging into how to address this, consulting the equivalent >>> zlib code. Simply reducing maxMatchLength to 128 seems to fix it, but zlib >>> doesn't require this hack, so I'm thinking that's the wrong direction. >>> >>> Insights welcome. >>> >>> Thanks, >>> -Rob >>> >>> On Friday, January 9, 2015 at 12:37:14 AM UTC-5, Rob Napier wrote: >>>> >>>> Got it. I'll put together a CL for review. Thanks. >>>> >>>> -Rob >>>> >>>> On Thu, Jan 8, 2015 at 10:58 PM, Brad Fitzpatrick <brad...@golang.org> >>>> wrote: >>>> >>>>> Go in general doesn't force you to play by the rules. Just document >>>>> the rules in English and it's their fault if there's a race. A public >>>>> field >>>>> on the Writer struct seems fine to me. >>>>> >>>>> >>>>> On Thu, Jan 8, 2015 at 2:13 PM, Rob Napier <robn...@gmail.com> wrote: >>>>> >>>>>> Totally agree. I'm embarrassed to say that I looked at "Writer" and >>>>>> my brain substituted "io.Writer". Yes, the sensible way to manage it >>>>>> would >>>>>> be via flate.Writer and zlib.Writer. >>>>>> >>>>>> I assume these should accessed through functions like SetWindow() >>>>>> rather than directly modifying public fields? That way Write() could >>>>>> enforce locking them, and zlib could pass along updates to the >>>>>> underlying >>>>>> flate.Writer. >>>>>> >>>>>> Do you have a particular data structure that would be a good model? >>>>>> >>>>>> -Rob >>>>>> >>>>>> On Thu, Jan 8, 2015 at 1:05 PM, Brad Fitzpatrick <brad...@golang.org> >>>>>> wrote: >>>>>> >>>>>>> I think I'd prefer if NewWriter returned a *Writer with a few public >>>>>>> fields you could tweak before your first call to Write, at which time >>>>>>> the >>>>>>> Writer is initialized and you can't tweak them anymore. We use that >>>>>>> pattern >>>>>>> sometimes, and I wish we would've used it here. But I think we still >>>>>>> could >>>>>>> retroactively, keeping the old constructors. >>>>>>> >>>>>>> >>>>>>> On Thu, Jan 8, 2015 at 7:41 AM, Rob Napier <robn...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Following up on https://groups.google.com/ >>>>>>>> forum/#!searchin/golang-nuts/zlib/golang-nuts/s7xpW4pBTPk/ >>>>>>>> QT5mUkbrl1QJ, I'm also using this spdy library and would like to >>>>>>>> shrink the zlib window size. The headers we're sending are tiny and >>>>>>>> will >>>>>>>> benefit little from compression, so the 32k buffer per connection is >>>>>>>> overly >>>>>>>> expensive when dealing with 30,000 connections. >>>>>>>> >>>>>>>> Looking at the zlib code, the window size appears to be hardcoded >>>>>>>> as "default" (2^15). >>>>>>>> >>>>>>>> From src/compress/zlib/writer.go >>>>>>>> <https://go.googlesource.com/go/+/release-branch.go1.4/src/compress/zlib/writer.go> >>>>>>>> : >>>>>>>> >>>>>>>> >>>>>>>> 1. // ZLIB has a two-byte header (as documented in RFC 1950). >>>>>>>> 2. // The first four bits is the CINFO (compression info), >>>>>>>> which is 7 for the default deflate window size. >>>>>>>> 3. // The next four bits is the CM (compression method), which >>>>>>>> is 8 for deflate. >>>>>>>> 4. z.scratch[0] = 0x78 >>>>>>>> >>>>>>>> >>>>>>>> It also appears to be hardcoded in compress/flate. Making this >>>>>>>> configurable seems possible, but not trivial. The actual change inside >>>>>>>> of >>>>>>>> deflate.go would be pretty easy, but the API doesn't provide any hooks >>>>>>>> for >>>>>>>> more configuration. It looks like it would need a >>>>>>>> NewWriterDictWindow() function (and maybe a NewWriterWindow() >>>>>>>> function). At the gzip level, it would require >>>>>>>> NewWriterLevelDictWindow() (which is quite a mouthful, but >>>>>>>> consistent). >>>>>>>> >>>>>>>> Is there a better way to configure this window? >>>>>>>> >>>>>>> >> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
