On 23/08/2012, at 5:23 PM, Alan M. Carroll <a...@network-geographics.com> wrote:
> Since this has been a topic for a while, I will just throw out an idea to see > how fast you guys can shoot it down. > > A cache object is stored as a series of fragments. If we subdivided each > fragment in to "chunks", we could have 64 chunks / fragment and represent > them with a bitmap in a single uint64_t. A set bit would indicate valid data > in that chunk. Partial content would be written only in chunk units and only > for chunks that are complete in the data. For the default size of 1M > fragments, each chunk would be 16K which seems a reasonable value. The > bitmaps would be stored along with the fragment offset table in the alternate > info header. This would keep it out of the directory while making it > available when serving because the alternate data is loaded before that > point. Range validity checks could also be done without additional disk I/O > because you can't detect if a range is valid for an object before the > alternate is determined. We could only serve if the request range was > completely covered, or generate a synthetic range request to cover parts that > were not already in the cache. > > This would mean that files less than one fragment would not have partial > content cached but I think that's acceptable as the advantages of partial > caching are only for larger objects. If 1 of N fragments is cached, can we currently serve ranges out of that? J
