Thank you very much for the clarifications! I think now I have a better view on the current cache structure and your proposal.
Current flow for a range request: 1. Find the first Doc for the cached object. 2. The first Doc contains the alternate vector and the fragment table (of course the fragment table belongs to the entries in the alternate and you said you will do this change soon) 3. Find the best matching alternate entry and use that as a representation of the cached object in the next steps. 4. Read the Doc corresponding to the best matching alternate. 5. Compute the needed fragments for serving the requested range. 6. Read the needed fragments and serve the response. In the current state if an object is cached that means its whole data is available from the cache. With your proposal: 1. Find the first Doc for the cached object. 2. The first Doc contains the alternate vector. 3. Find the best matching alternate entry and use that as a representation of the cached object in the next steps. 4. Read the Doc corresponding to the best matching alternate.[with your change we will have available here the fragment table I presume] 5. Compute the needed fragments and verify that all data is available (check corresponding chunk validity info that should be located in the fragment table) 6. Read the needed fragments and serve the response if everything is available. Considering that your proposal fully supports partially cached objects without any performance hit my opinion is that it would be a good representation for our needs. The changes to the cache structure are also kept to a minimum. Who should also validate these changes before going to the next step which I think would be the strategy of what and when to cache? PS: is there a branch already available with your change to move the fragment table from the first Doc to the alternate entries? Best regards, Bogdan Graur On Thu, Sep 6, 2012 at 4:15 AM, Alan M. Carroll <a...@network-geographics.com > wrote: > Wednesday, September 5, 2012, 4:16:42 PM, you wrote: > > > Are all Doc structures of the same size? > > Mostly, but you don't need their actual size, only the size of the content > per Doc instance, which is implicit in the fragment offset table. Given a > range request we can walk the fragment offset table (which was read with > the First/Earliest Doc) until we find the fragment which contains the > starting byte of the range and being loading fragments with that one. > > > So the fragments table entry will be augmented with a new uint64_t field > > which holds the chunk validity information. > > Right. > > > But after computing all the necessary "Doc" structures for satisfying a > > range request they still have to be read from disk (not the content, just > > the first part containing the fragments table) to check for chunk > validity. > > No. The fragment offset table will be moved in to the alternate header, > which is stored in First Doc. Once that is read you have all of the > fragment offset and chunk validity data in memory. The entire table, with > all offsets and validity bitmaps, is a single chunk of data in the > alternate header. > > Again, this is just a proposal to stimulate discussion, it is *not* a > committed plan of action. > > > Only when we know we have the complete range valid we may serve the range > > from the cache. > > Yes, and that can be computed from in memory data as soon as the alternate > is selected. > >
