Min, (I apologize for my belated reply -- I lost track of this draft in the chaos of the last couple of days.)
Upon further review, I think I feel into the confusion between management server and ssvm. This code is executing on the management server side, correct? Based on my "corrected" understanding is correct, I would like to amend my thoughts. Namely, I would like to see the driver operations pushed out to the SSVM where we can use the stream. As I think about it, the management server should not need to interact with the driver. Simply yard up the DataStore attributes + details map and other extract parameters, and send them to the SSVM. Using this information, the S3 driver could open a stream to write the template out to the bucket/directory. I recognize it changes the protocol between the management server and SSVM, but it simply both sides of the operation by allowing the DataStore information to be treated opaquely until it is consumed by the driver to execute the write operation. I also recognize that we may a little late in the cycle to address it for 4.2, and it may need to be part of the 4.3 enhancements. Thanks, -John On Jun 18, 2013, at 3:55 PM, Min Chen <min.c...@citrix.com> wrote: > John, > In that case, how do we keep backward compatibility of extractTemplate > api, which requires a URL in the response? > > Thanks > -min > > On 6/18/13 11:53 AM, "John Burwell" <jburw...@basho.com> wrote: > >> Min, >> >> Looking through the code, I think we can simplify driver operation and >> increase robustness by changing ImageStoreDriver#createEntityExtractUrl() >> : String to ImageStoreDriver#readEntity(Š) : InputStream. My first >> concern with the current implementation is that it circumvents any >> connection pooling/resource management underlying client libraries >> provide. I/O streams provide a higher-level abstraction that allows >> drivers to provide the orchestration components with actual resources >> rather String references. Second, the current interface seems to appears >> to assume that an http/https URL will be returned. With I/O streams, we >> can support any client library capable of using the standard I/O >> framework -- enabling us to support other protocols for downloading >> templates in the future (e.g. RBD, local filesystem, NBD, etc). >> >> Thanks, >> -John >> >> On Jun 18, 2013, at 1:11 PM, Min Chen <min.c...@citrix.com> wrote: >> >>> A new version of using generatePresignedUrl in S3ImageStoreDriverImpl is >>> checked into object_store. >>> >>> THanks >>> -min >>> >>> On 6/18/13 8:29 AM, "Min Chen" <min.c...@citrix.com> wrote: >>> >>>> Yes, current code is in S3ImageStoreDriverImpl.createEntityExtractUrl, >>>> which has a security issue mentioned in CLOUDSTACK-3030. I am going to >>>> change it to use generatePresignedUrl api from AWS S3 api. >>>> >>>> Thanks >>>> -min >>>> >>>> From: John Burwell <jburw...@basho.com<mailto:jburw...@basho.com>> >>>> Date: Tuesday, June 18, 2013 8:07 AM >>>> To: Min Chen <min.c...@citrix.com<mailto:min.c...@citrix.com>> >>>> Cc: Thomas O'Dowd <tpod...@cloudian.com<mailto:tpod...@cloudian.com>>, >>>> "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>" >>>> <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>> >>>> Subject: Re: Query String Request Authentication(QSRA) support by S3 >>>> providers >>>> >>>> Min, >>>> >>>> Is the code checked into the object_store branch? If so, which lines >>>> in >>>> S3TemplateDownloader? >>>> >>>> Thanks, >>>> -John >>>> >>>> On Jun 18, 2013, at 12:39 AM, Min Chen >>>> <min.c...@citrix.com<mailto:min.c...@citrix.com>> wrote: >>>> >>>> Hi John, >>>> >>>> This is regarding extractTemplate api, where for extractable template, >>>> users can click "Download Template" button from UI to get a http url to >>>> download the template already stored at S3 without providing S3 >>>> credentials. In 4.1, we don't have this issue, since the URL returned >>>> is >>>> the public web server location hosted in ssvm, and in 4.2, we are >>>> returning URL pointing to s3 object. Without setting ACL to the S3 >>>> object, user cannot directly click the URL returned from >>>> extractTemplate >>>> api to download the template without providing credentials. By reading >>>> the AWS SDK doc today, I ran across the following API that I may be >>>> able >>>> to use for this purpose: >>>> >>>> >>>> >>>> URL<http://java.sun.com/j2se/1.5.0/docs/api/java/net/URL.html?is-externa >>>> l= >>>> true> >>>> >>>> generatePresignedUrl<http://docs.aws.amazon.com/AWSJavaSDK/latest/javado >>>> c/ >>>> >>>> com/amazonaws/services/s3/AmazonS3Client.html#generatePresignedUrl%28jav >>>> a. >>>> >>>> lang.String,%20java.lang.String,%20java.util.Date,%20com.amazonaws.HttpM >>>> et >>>> >>>> hod%29>(String<http://java.sun.com/j2se/1.5.0/docs/api/java/lang/String. >>>> ht >>>> ml?is-external=true> bucketName, >>>> >>>> String<http://java.sun.com/j2se/1.5.0/docs/api/java/lang/String.html?is- >>>> ex >>>> ternal=true> key, >>>> >>>> Date<http://java.sun.com/j2se/1.5.0/docs/api/java/util/Date.html?is-exte >>>> rn >>>> al=true> expiration, >>>> >>>> HttpMethod<http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amaz >>>> on >>>> aws/HttpMethod.html> method) >>>> Returns a pre-signed URL for accessing an Amazon S3 resource. >>>> >>>> This is along the same line as QSRA mentioned by Tom, by wrapped in >>>> AmazonS3Client for easy consumption. By using this method, I think >>>> that I >>>> don't need to change ACL of S3 object to open a security hole. >>>> >>>> Thanks >>>> -min >>>> >>>> From: John Burwell <jburw...@basho.com<mailto:jburw...@basho.com>> >>>> Date: Monday, June 17, 2013 7:38 PM >>>> To: Min Chen <min.c...@citrix.com<mailto:min.c...@citrix.com>> >>>> Cc: Thomas O'Dowd <tpod...@cloudian.com<mailto:tpod...@cloudian.com>>, >>>> "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>" >>>> <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>> >>>> Subject: Re: Query String Request Authentication(QSRA) support by S3 >>>> providers >>>> >>>> Min, >>>> >>>> Why are we mucking with ACLs at all? The best security practice would >>>> be >>>> to create a bucket for CloudStack's use and assign it a dedicated >>>> access >>>> key and secret key pair with read/write access only to that bucket. >>>> Requiring an administrative account to an object store opens an >>>> unnecessarily large attack surface. Therefore, as implemented in 4.1, >>>> we >>>> should defer bucket creation, ACL assignment, and credential creation >>>> to >>>> the administrator/operator. >>>> >>>> Thanks, >>>> -John >>>> >>>> On Jun 17, 2013, at 1:15 PM, Min Chen >>>> <min.c...@citrix.com<mailto:min.c...@citrix.com>> wrote: >>>> >>>> Tom filed a very good bug for ACL setting change on S3 object when >>>> users >>>> issue extractTemplate API >>>> (https://issues.apache.org/jira/browse/CLOUDSTACK-3030), and his >>>> recommendation of using Query String Request Authentication (QSRA) >>>> alternative sounds like a right approach to fix this bug. Before >>>> implementing it, I would like to confirm if QSRA should be supported by >>>> all S3 providers if they claim that they are AWS s3 compatible. If so, >>>> we >>>> will make this assumption in our code. Based on Tom, Cloudian is >>>> supporting it. How about RiakCS, John? >>>> >>>> Thanks >>>> -min >>>> >>>> >>> >> >
