Thank you Alan. We will take a deeper look and see where it goes. Best regards, Ori
On Wed, Feb 1, 2017 at 4:35 PM, Alan Carroll <solidwallofcode@yahoo-inc. com.invalid> wrote: > I think it would be possible, it's something that been discussed before. > You would also need to replace CacheVC which is used as the interface > between the cache the virtual I/O system inside ATS. I haven't investigated > that to any serious extent so it's hard to say the overall difficulty. You > would have replace the entire cache subsystem, the current internals are > not very compatible with this. > > On Monday, January 30, 2017 6:52 AM, Ori Finkelman <o...@qwilt.com> wrote: > > *Writing to cache* > We need to be able to write huge files and be able to receive them in > chunks (ranges) as they are received over separated requests. When writing > a chunk of a file we pre-allocate sequential space for the next chunks. > Small chunks are buffered and written to disk only when enough is > aggregated across separate requests. This also inherently solves the http > ranges problem as space is always reserved for ranges which were not yet > acquired. > > *Reading from cache* > When reading an object we use read-ahead into memory to optimize on IOPS as > in most cases the next chunks of the objects will be requested within a few > seconds. > > *Evacuation* > Evacuation should be done based on popularity (for example LRU), and works > for the whole object together, not per chunk. > The cyclone-buffer is not ideal for such a use case as the evacuation > policy is different and we can't pay for the the copy of large objects > every time the write cursor pass them. > > > Since objects are very large and writing out is done in very large chunks, > file system overhead is negligible and therefore an architecture relying on > Linux file system may have benefits, such as the ability to read objects > from FS rather than via HTTP for debug and analysis. > It is also possible to implement something which is based on the existing > cache architecture with the required adaptations. > > We are trying to evaluate if implementing such a new cache storage mode is > possible ? is it reasonable ? were there previous thoughts in such > directions ? > I could find in the wiki reference for both large files and ranges > requests, but no conclusions. > > In a fast glimpse, It does look like it would not be very hard to replace > the existing *CacheProcessor *and *CacheVConnection *with different ones > which will implement a different cache storage. > > Thank you in advance for any feedback. > > -- > > *Ori Finkelman*Qwilt | Work: +972-72-2221647 | Mobile: +972-52-3832189 | > o...@qwilt.com > -- *Ori Finkelman*Qwilt | Work: +972-72-2221647 <072-222-1647> | Mobile: +972-52-3832189 <052-383-2189> | o...@qwilt.com
