Presumably you are also aware of patents 5,341,457 and 5,627,938, which Lucent has been seeking to enforce against Dolby AC-3. As your encoder appears to use Ehmer's tone masking techniques, which are also cited in the AC-3 standard definition, that litigation may be of interest -- particularly as Dolby obtained in April a summary judgment of non-infringement and there is a real possibility that Lucent's patents will be invalidated altogether if the remainder of the case goes to trial as scheduled in September.
Although your psy.c is a bit opaque and I have given these Lucent patents only the briefest of glances, I would say that the '457 patent disclosure resembles the Vorbis encoder more closely than anything else I have seen in the literature. No suggestion that you infringe is implied; I'm just trying to get a handle on how psy.c works and the patent database is the best hook into the primary literature that I am currently holding. I haven't identified which, if any, patents cover Dolby AC-3 qua AC-3 (that's a Dolby trademark, as is Dolby Digital 5.1; the generic name is "ATSC Standard A/52"). There seems to be a relevant patent pool in DVD space (in which Dolby participates, to the extent of receiving a small royalty on AC-3 encoded DVDs, but may not have contributed any patents AFAICT). There is an interesting list in Appendix A of http://contracts.onecle.com/intervideo/dolby.lic.1999.03.04.shtml but I haven't ground through it and probably won't. The "cease and desist" letter at http://www4.netbsd.org/Letters/20010803-dolby.html looks to me to be "actual notice" of nothing whatsoever. Given that Dolby approaches "violators" even more prejudicially than Thomson does (or used to?), somehow I suspect that if Dolby had a leg to stand on other than their trademarks then there would be some record of their citing a specific patent against A52dec and/or FFmpeg (which remain on SourceForge). IANAL, TINLA, YMMV. I have, however, tracked down the principal DTS patent (#5,956,674) claimed against VideoLan's libdca; but it has such a thicket of claims that I cannot begin to say what might or might not infringe it other than an implementation of DTS itself. I doubt they would bother you, though; their format is wildly different (mixed VQ and ADPCM, specialized subframes to massage transients away, "signal-to-mask ratios"), and it has the general air of a tweak of a hack to a kludge. So, by reputation, does AC-3; so unless Dolby holds something pretty general (which would surprise me), it also seems unlikely to threaten Vorbis unless you know something I don't. Cheers, - Michael (IANAL, TINLA)