Edison, I did not say that Swift can't handle files greater than 5 GB. With S3, you must use multi-part uploads in order to upload a file greater than 5GB. Therefore, Swift's S3 compatibility layer must be compatible with the S3 multipart upload API in order to work with CloudStack. Please see my previous remarks regarding the differences in approach between the two APIs for more a detailed explanation regarding the incompatibilities and the difficulties for Swift to support S3-style multi-part uploads.
Thanks, -John On Jul 9, 2013, at 4:56 PM, Edison Su <edison...@citrix.com> wrote: > > >> -----Original Message----- >> From: John Burwell [mailto:jburw...@basho.com] >> Sent: Tuesday, July 09, 2013 1:31 PM >> To: dev@cloudstack.apache.org >> Cc: 'Chip Childers' >> Subject: Re: Swift in 4.2 is broken, anybody wants it to be supported in 4.2? >> >> Edison, >> >> TL;DR The shorter path is to re-implement/fix the Swift driver. >> >> Multipart upload would need to implemented in Swift, not in CloudStack. >> Therefore, such a change would need to be accepted and released by the >> OpenStack project before the 4.2.0 release. We would also be stranding any >> of our current users who cannot or will not upgrade their Swift instances. >> >> Knowing what it took to implement it in Riak CS, multi-part upload was a lot >> of work to implement in a Dynamo-based system. The S3 has the following >> three-phase process: >> >> 1. Initiate a Upload: Declare the number of parts and their size >> 2. Submit each part per the definition in Step 1 (e.g. 50 parts = 50 >> HTTP PUTs) >> 3. Complete the multi-part upload: Declare that all parts have been >> uploaded which causes the object to become available >> >> In contrast, Swift uses HTTP chunking to solve the same problem with one >> API call. In addition to providing all of the reliability guarantees of the >> S3 API, >> an implementor of S3 multipart uploads will have to provide a way to >> translate the 3-phase model into the single call model used by Swift. > > Seems swift client can support upload files more than 5GB: > http://fedoraproject.org/wiki/QA:Testcase_Swift_Upload_Large_File > > Ok, so we should just re-implement/fix the existing swift driver. > >> >> Thanks, >> -John >> >> On Jul 9, 2013, at 4:15 PM, Edison Su <edison...@citrix.com> wrote: >> >>> Could they(swiftstack) help us, or guide us on how to implement multi-part >> upload? >>> >>>> -----Original Message----- >>>> From: Sebastien Goasguen [mailto:run...@gmail.com] >>>> Sent: Tuesday, July 09, 2013 1:07 PM >>>> To: dev@cloudstack.apache.org >>>> Cc: 'Chip Childers' >>>> Subject: Re: Swift in 4.2 is broken, anybody wants it to be supported in >>>> 4.2? >>>> >>>> if swift does not work anymore in 4.0 or 4.1 maybe be should inform >>>> swiftstack: >>>> http://swiftstack.com/cloudstack/ >>>> >>>> >>>> On Jul 9, 2013, at 3:57 PM, John Burwell <jburw...@basho.com> wrote: >>>> >>>>> Edison, >>>>> >>>>> Swift does not support S3 multi-part uploads [1] which CloudStack >>>>> must use >>>> in order to store files larger than 5 GB. Therefore, using the >>>> Swift's S3 compatibility layer is not a viable workaround. >>>>> >>>>> Thanks, >>>>> -John >>>>> >>>>> [1]: https://wiki.openstack.org/wiki/Swift/APIFeatureComparison >>>>> >>>>> On Jul 9, 2013, at 2:12 PM, Edison Su <edison...@citrix.com> wrote: >>>>> >>>>>> >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Chip Childers [mailto:chip.child...@sungard.com] >>>>>>> Sent: Monday, July 08, 2013 1:26 PM >>>>>>> To: Edison Su >>>>>>> Cc: <dev@cloudstack.apache.org> >>>>>>> Subject: Re: Swift in 4.2 is broken, anybody wants it to be >>>>>>> supported in >>>> 4.2? >>>>>>> >>>>>>> On Mon, Jul 08, 2013 at 05:15:19PM +0000, Edison Su wrote: >>>>>>>> >>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Chip Childers [mailto:chip.child...@sungard.com] >>>>>>>>> Sent: Monday, July 08, 2013 6:46 AM >>>>>>>>> To: <dev@cloudstack.apache.org>; Edison Su >>>>>>>>> Subject: Re: Swift in 4.2 is broken, anybody wants it to be >>>>>>>>> supported in >>>>>>> 4.2? >>>>>>>>> >>>>>>>>> On Mon, Jul 8, 2013 at 9:22 AM, David Nalley <da...@gnsa.us> >> wrote: >>>>>>>>>> On Wed, Jul 3, 2013 at 5:29 PM, Edison Su >>>>>>>>>> <edison...@citrix.com> >>>>>>> wrote: >>>>>>>>>>> Due to object store refactor, Swift is broken. The reason, is >>>>>>>>>>> that, we only >>>>>>>>> have S3 test environment in our lab, so only S3 is tested for now. >>>>>>>>>>> Before adding the feature back, I'd better ask from, the >>>>>>>>>>> community, do >>>>>>>>> we want to support Swift? If so, which version of Swift? This >>>>>>>>> will take some efforts to support Swift, are there any >>>>>>>>> volunteers can help the >>>>>>> integration? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Whats the bug ID for this? >>>>>>>>>> Unplanned/Unannounced deprecation of a feature is a blocker >> IMO. >>>>>>>>>> It engenders a bad relationship with our users, and strands >>>>>>>>>> them on previous versions with no good migration/upgrade path. >>>>>>>>>> >>>>>>>>>> --David >>>>>>>>>> >>>>>>>>> >>>>>>>>> Edison, How broken is it? Is it shorter to fix or revert the >>>>>>>>> object store changes? >>>>>>>> It's not working at all. Not sure, revert object store will >>>>>>>> change it or not, as >>>>>>> this feature is not tested by QA for a long time. >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> So any idea what the effort of fixing it looks like? I mean, just >>>>>>> because it >>>>>> >>>>>> If it's ok to use S3 api talking to swift, then there is zero >>>>>> effort to support >>>> Swift. >>>>>> But who will make the decision? >>>>>> >>>>>>> wasn't tested in the last couple of releases doesn't necessarily >>>>>>> mean that it wasn't working. As Sudha mentioned, it wasn't tested >>>>>>> only because of a lack of change that triggered the expected need >>>>>>> to perform regression testing of that feature. >>>>>>> >>>>>>> I believe that this was an honest mistake, but we need to figure >>>>>>> out what to do. I'm -1 on us saying "we'll drop Swift support". >>>>>>> If necessary, I'd say that we need to roll back the object-store >>>>>>> branch merge... I don't want to see that happen though. That's >>>>>>> why I'm asking >>>> about effort to fix it. >>>>>>> >>>>>>> -chip >>>>> >>> >
