Animesh, I am working through it now. I will do my best to be done by Friday, but it is a massive patch to digest (25 pages in Review Board).
Thanks, -John On May 22, 2013, at 3:01 PM, Animesh Chaturvedi <animesh.chaturv...@citrix.com> wrote: > > >> -----Original Message----- >> From: John Burwell [mailto:jburw...@basho.com] >> Sent: Tuesday, May 21, 2013 8:51 AM >> To: dev@cloudstack.apache.org >> Subject: Re: [MERGE]object_store branch into master >> >> Edison, >> >> Thanks, I will start going through it today. Based on other $dayjob >> responsibilities, it may take me a couple of days. >> >> Thanks, >> -John > [Animesh>] John we are just a few days away from 4.2 feature freeze, can you > provide your comments by Friday 5/24. I would like all feature threads to > be resolved sooner so that we don't have last minute rush. >> >> On May 20, 2013, at 6:15 PM, Edison Su <edison...@citrix.com> wrote: >> >>> >>> >>>> -----Original Message----- >>>> From: Edison Su [mailto:edison...@citrix.com] >>>> Sent: Monday, May 20, 2013 2:30 PM >>>> To: dev@cloudstack.apache.org >>>> Subject: RE: [MERGE]object_store branch into master >>>> >>>> >>>> >>>>> -----Original Message----- >>>>> From: John Burwell [mailto:jburw...@basho.com] >>>>> Sent: Monday, May 20, 2013 12:56 PM >>>>> To: dev@cloudstack.apache.org >>>>> Subject: Re: [MERGE]object_store branch into master >>>>> >>>>> All, >>>>> >>>>> Since this change is so large, it makes reviewing and commenting in >>>>> detail extremely difficult. Would it be possible to push this patch >>>>> through Review Board to ease comprehension and promote a >>>>> conversation >>>> about this patch? >>>> >>>> We can try to push it into Review Board. >>> >>> The review board url is: https://reviews.apache.org/r/11277/, 25 pages... >>> >>>> >>>>> >>>>> Reading through the FS, I have the following questions regarding the >>>>> operation of the NFS cache: >>>>> >>>>> What happens if/when the disk space of the NFS cache is exhausted? >>>>> What are the sizing recommendations/guidelines for it? >>>>> What strategy is used to age files out of the NFS cache? >>>> As usual, admin can have multiple NFS secondary storages, admin can >>>> also add multiple NFS cache storages. The NFS cache storage capacity >>>> plan will be the same as NFS secondary storage. >>>> If there multiple NFS cache storages, the current strategy will >>>> randomly choose one of them. Currently, no clean up/aging out >>>> strategy implemented yet But the situation can be improved: most of >>>> cached object can be deleted after accessed once. Take template as >>>> example, if zone wide storage is used, put template on cache storage >>>> has little value, as once the template is downloaded into primary >>>> storage, suddenly all the hypervisor host can access it. >>>> I think the simple LRU algorithm to delete cached objects should be >> enough. >>>> It can be added later, the cache storage has its own pom project, >>>> it's place to add more intelligence. >>>> >>>>> If two processes, process1 and process2, are both using a template, >>>>> templateA, will both processes reference the same file in the NFS >>>>> cache? If >>>> It's possible, that one template can be downloaded into cache storage >>>> twice, in case of concurrent accessed by two processes. The current >>>> implementation is that, if two processes want to download the same >>>> template from s3 into one primary storage at the same time, there is >>>> only one template will be downloaded into cache storage. While, if >>>> two processes want to download the same template into different >>>> primary storage, the template will be cached twice. >>>>> they are reading from the same file and process1 finishes before >>>>> process2, will process1 attempt to delete process2? >>>> >>>> There is no way to delete while read, as each cached object has its >>>> own state machine. If it's accessed by one process, the state will be >>>> changed to "Copying", you can't delete an object when it's in "Copying" >> state. >>>> >>>>> If a file transfer from the NFS cache to the object store fails, >>>>> what is the recovery/retry strategy? What durability guarantees >>>>> will CloudStack supply when a snapshot, template, or ISO is in the >>>>> cache, but can't be written to the object store? >>>> >>>> The error handling of cache storage shouldn't be different than >>>> without cache storage. For example, directly backup snapshot from >>>> primary storage to S3, without cache storage. If backup failed, then >>>> the whole process failed, user needs to do it again through >>>> cloudstack API. So in cache storage case, if push object from cache >>>> storage to s3 failed, then the whole backup process failed. >>>> >>>>> What will be the migration strategy for the objects contained in S3 >>>>> buckets/Swift containers from pre-4.2.0 instances? Currently, >>>>> CloudStack tracks a mapping between these objects and templates/ISOs >>>>> in the template_switt_ref and template_s3_ref table. >>>> >>>> We need to migrate DB from existing template_s3_ref to >>>> template_store_ref, and put all the s3 information into image_store >>>> and image_store_details tables. >>>> >>>>> >>>>> Finally, does the S3 implementation use multi-part upload to >>>>> transfer files to the object store? If not, the implementation will >>>>> be limited to storing files no larger than 5GB in size. >>>> Oh, this is something we don't know yet. We haven't try to upload a >>>> template which is large than 5GB, so haven't met this issue. >>>> Could you help to hack it up?:) >>>> >>>>> >>>>> Thanks, >>>>> -John >>>>> >>>>> On May 20, 2013, at 1:52 PM, Chip Childers >>>>> <chip.child...@sungard.com> >>>>> wrote: >>>>> >>>>>> On Fri, May 17, 2013 at 08:19:57AM -0400, David Nalley wrote: >>>>>>> On Fri, May 17, 2013 at 4:11 AM, Edison Su <edison...@citrix.com> >>>> wrote: >>>>>>>> Hi all, >>>>>>>> Min and I worked on object_store branch during the last one >>>>>>>> and half >>>>> month. We made a lot of refactor on the storage code, mostly related >>>>> to secondary storage, but also on the general storage framework. The >>>>> following goals are made: >>>>>>>> >>>>>>>> 1. An unified storage framework. Both secondary >>>>> storages(nfs/s3/swift etc) and primary storages will share the same >>>>> plugin model, the same interface. Add any other new storages into >>>>> cloudstack will much easier and straightforward. >>>>>>>> >>>>>>>> 2. The storage interface between mgt server and resource is >>>> unified, >>>>> currently there are only 5 commands send out by mgt server: >> copycommand/createobjectcommand/deletecommand/attachcommand/de >>>>> ttachcommand, and each storage vendor can decode/encode all the >>>>> entities(volume/snapshot/storage pool/ template etc) by its own. >>>>>>>> >>>>>>>> 3. NFS secondary storage is not explicitly depended on by other >>>>> components. For example, when registering template into S3, template >>>>> will be write into S3 directly, instead of storing into nfs >>>>> secondary storage, then push to S3. If s3 is used as secondary >>>>> storage, then nfs storage will be used as cache storage, but from >>>>> other components point of view, cache storage is invisible. So, it's >>>>> possible to make nfs storage as optional if s3 is used for certain >> hypervisors. >>>>>>>> The detailed FS is at >> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+Backu >>>>>>>> p+Object+Store+Plugin+Framework >>>>>>>> The test we did: >>>>>>>> >>>>>>>> 1. We modified marvin to use new storage api >>>>>>>> >>>>>>>> 2. Test_volume and test_vm_life_cycle, test_template under >>>> smoke >>>>> test folder are executed against xenserver/kvm/vmware and devcloud, >>>>> some of them are failed, it's partly due to bugs introduced by our >>>>> code, partly master branch itself has issue(e.g. resizevolume >>>>> doesn't work). We want to fix these issues after merging into master. >>>>>>>> >>>>>>>> The basic follow does work: create user vm, attach/detach volume, >>>>> register template, create template from volume/snapshot, take >>>>> snapshot, create volume from snapshot. >>>>>>>> It's a huge change, around 60k LOC patch, to review the code, you >>>>>>>> can >>>>> try: git diff master..object_store, will show all the diff. >>>>>>>> Comments/feedback are welcome. Thanks. >>>>>>> >>>>>>> >>>>>>> Given the amount of change, can we get at least a BVT run against >>>>>>> your branch done before merge? >>>>>>> >>>>>>> --David >>>>>> >>>>>> +1 to BVT please. >
