Hi Min, Can you check this bug? I'm trying to test this feature for Amazon but having no luck getting the Download template link/button to appear.
https://issues.apache.org/jira/browse/CLOUDSTACK-3220 Thanks, Tom. On Fri, 2013-06-21 at 17:21 +0000, Min Chen wrote: > John, > > For S3, the api call createEntityExtractUrl is done on management server > side; while for NFS secondary storage, if the implementation of > createEntityExtractUrl will involve some code be executed in ssvm to copy > template from the install location to a public accessible web server > location. > I don't quite understand some of your comments below. This API is not > used to write any information to S3 bucket/directory. This is used for > object already existed on S3, and we just provide a URL for user to > download a template from S3, just like how Amazon provided user a way to > user to extract a S3 object through generatePresignedUrl. We can discuss > more on this on collaboration conference. > > Thanks > -min > > > > On 6/21/13 7:25 AM, "John Burwell" <jburw...@basho.com> wrote: > > >Min, > > > >(I apologize for my belated reply -- I lost track of this draft in the > >chaos of the last couple of days.) > > > >Upon further review, I think I feel into the confusion between management > >server and ssvm. This code is executing on the management server side, > >correct? Based on my "corrected" understanding is correct, I would like > >to amend my thoughts. Namely, I would like to see the driver operations > >pushed out to the SSVM where we can use the stream. As I think about it, > >the management server should not need to interact with the driver. > >Simply yard up the DataStore attributes + details map and other extract > >parameters, and send them to the SSVM. Using this information, the S3 > >driver could open a stream to write the template out to the > >bucket/directory. I recognize it changes the protocol between the > >management server and SSVM, but it simply both sides of the operation by > >allowing the DataStore information to be treated opaquely until it is > >consumed by the driver to execute the write operation. I also recognize > >that we may a little late in the cycle to address it for 4.2, and it may > >need to be part of the 4.3 enhancements. > > > >Thanks, > >-John > > > >On Jun 18, 2013, at 3:55 PM, Min Chen <min.c...@citrix.com> wrote: > > > >> John, > >> In that case, how do we keep backward compatibility of extractTemplate > >> api, which requires a URL in the response? > >> > >> Thanks > >> -min > >> > >> On 6/18/13 11:53 AM, "John Burwell" <jburw...@basho.com> wrote: > >> > >>> Min, > >>> > >>> Looking through the code, I think we can simplify driver operation and > >>> increase robustness by changing > >>>ImageStoreDriver#createEntityExtractUrl() > >>> : String to ImageStoreDriver#readEntity(Š) : InputStream. My first > >>> concern with the current implementation is that it circumvents any > >>> connection pooling/resource management underlying client libraries > >>> provide. I/O streams provide a higher-level abstraction that allows > >>> drivers to provide the orchestration components with actual resources > >>> rather String references. Second, the current interface seems to > >>>appears > >>> to assume that an http/https URL will be returned. With I/O streams, > >>>we > >>> can support any client library capable of using the standard I/O > >>> framework -- enabling us to support other protocols for downloading > >>> templates in the future (e.g. RBD, local filesystem, NBD, etc). > >>> > >>> Thanks, > >>> -John > >>> > >>> On Jun 18, 2013, at 1:11 PM, Min Chen <min.c...@citrix.com> wrote: > >>> > >>>> A new version of using generatePresignedUrl in S3ImageStoreDriverImpl > >>>>is > >>>> checked into object_store. > >>>> > >>>> THanks > >>>> -min > >>>> > >>>> On 6/18/13 8:29 AM, "Min Chen" <min.c...@citrix.com> wrote: > >>>> > >>>>> Yes, current code is in > >>>>>S3ImageStoreDriverImpl.createEntityExtractUrl, > >>>>> which has a security issue mentioned in CLOUDSTACK-3030. I am going > >>>>>to > >>>>> change it to use generatePresignedUrl api from AWS S3 api. > >>>>> > >>>>> Thanks > >>>>> -min > >>>>> > >>>>> From: John Burwell <jburw...@basho.com<mailto:jburw...@basho.com>> > >>>>> Date: Tuesday, June 18, 2013 8:07 AM > >>>>> To: Min Chen <min.c...@citrix.com<mailto:min.c...@citrix.com>> > >>>>> Cc: Thomas O'Dowd > >>>>><tpod...@cloudian.com<mailto:tpod...@cloudian.com>>, > >>>>> "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>" > >>>>> <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>> > >>>>> Subject: Re: Query String Request Authentication(QSRA) support by S3 > >>>>> providers > >>>>> > >>>>> Min, > >>>>> > >>>>> Is the code checked into the object_store branch? If so, which lines > >>>>> in > >>>>> S3TemplateDownloader? > >>>>> > >>>>> Thanks, > >>>>> -John > >>>>> > >>>>> On Jun 18, 2013, at 12:39 AM, Min Chen > >>>>> <min.c...@citrix.com<mailto:min.c...@citrix.com>> wrote: > >>>>> > >>>>> Hi John, > >>>>> > >>>>> This is regarding extractTemplate api, where for extractable > >>>>>template, > >>>>> users can click "Download Template" button from UI to get a http url > >>>>>to > >>>>> download the template already stored at S3 without providing S3 > >>>>> credentials. In 4.1, we don't have this issue, since the URL returned > >>>>> is > >>>>> the public web server location hosted in ssvm, and in 4.2, we are > >>>>> returning URL pointing to s3 object. Without setting ACL to the S3 > >>>>> object, user cannot directly click the URL returned from > >>>>> extractTemplate > >>>>> api to download the template without providing credentials. By > >>>>>reading > >>>>> the AWS SDK doc today, I ran across the following API that I may be > >>>>> able > >>>>> to use for this purpose: > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>URL<http://java.sun.com/j2se/1.5.0/docs/api/java/net/URL.html?is-exter > >>>>>na > >>>>> l= > >>>>> true> > >>>>> > >>>>> > >>>>>generatePresignedUrl<http://docs.aws.amazon.com/AWSJavaSDK/latest/java > >>>>>do > >>>>> c/ > >>>>> > >>>>> > >>>>>com/amazonaws/services/s3/AmazonS3Client.html#generatePresignedUrl%28j > >>>>>av > >>>>> a. > >>>>> > >>>>> > >>>>>lang.String,%20java.lang.String,%20java.util.Date,%20com.amazonaws.Htt > >>>>>pM > >>>>> et > >>>>> > >>>>> > >>>>>hod%29>(String<http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Strin > >>>>>g. > >>>>> ht > >>>>> ml?is-external=true> bucketName, > >>>>> > >>>>> > >>>>>String<http://java.sun.com/j2se/1.5.0/docs/api/java/lang/String.html?i > >>>>>s- > >>>>> ex > >>>>> ternal=true> key, > >>>>> > >>>>> > >>>>>Date<http://java.sun.com/j2se/1.5.0/docs/api/java/util/Date.html?is-ex > >>>>>te > >>>>> rn > >>>>> al=true> expiration, > >>>>> > >>>>> > >>>>>HttpMethod<http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/am > >>>>>az > >>>>> on > >>>>> aws/HttpMethod.html> method) > >>>>> Returns a pre-signed URL for accessing an Amazon S3 resource. > >>>>> > >>>>> This is along the same line as QSRA mentioned by Tom, by wrapped in > >>>>> AmazonS3Client for easy consumption. By using this method, I think > >>>>> that I > >>>>> don't need to change ACL of S3 object to open a security hole. > >>>>> > >>>>> Thanks > >>>>> -min > >>>>> > >>>>> From: John Burwell <jburw...@basho.com<mailto:jburw...@basho.com>> > >>>>> Date: Monday, June 17, 2013 7:38 PM > >>>>> To: Min Chen <min.c...@citrix.com<mailto:min.c...@citrix.com>> > >>>>> Cc: Thomas O'Dowd > >>>>><tpod...@cloudian.com<mailto:tpod...@cloudian.com>>, > >>>>> "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>" > >>>>> <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>> > >>>>> Subject: Re: Query String Request Authentication(QSRA) support by S3 > >>>>> providers > >>>>> > >>>>> Min, > >>>>> > >>>>> Why are we mucking with ACLs at all? The best security practice > >>>>>would > >>>>> be > >>>>> to create a bucket for CloudStack's use and assign it a dedicated > >>>>> access > >>>>> key and secret key pair with read/write access only to that bucket. > >>>>> Requiring an administrative account to an object store opens an > >>>>> unnecessarily large attack surface. Therefore, as implemented in > >>>>>4.1, > >>>>> we > >>>>> should defer bucket creation, ACL assignment, and credential creation > >>>>> to > >>>>> the administrator/operator. > >>>>> > >>>>> Thanks, > >>>>> -John > >>>>> > >>>>> On Jun 17, 2013, at 1:15 PM, Min Chen > >>>>> <min.c...@citrix.com<mailto:min.c...@citrix.com>> wrote: > >>>>> > >>>>> Tom filed a very good bug for ACL setting change on S3 object when > >>>>> users > >>>>> issue extractTemplate API > >>>>> (https://issues.apache.org/jira/browse/CLOUDSTACK-3030), and his > >>>>> recommendation of using Query String Request Authentication (QSRA) > >>>>> alternative sounds like a right approach to fix this bug. Before > >>>>> implementing it, I would like to confirm if QSRA should be supported > >>>>>by > >>>>> all S3 providers if they claim that they are AWS s3 compatible. If > >>>>>so, > >>>>> we > >>>>> will make this assumption in our code. Based on Tom, Cloudian is > >>>>> supporting it. How about RiakCS, John? > >>>>> > >>>>> Thanks > >>>>> -min > >>>>> > >>>>> > >>>> > >>> > >> > > > -- Cloudian KK - http://www.cloudian.com/get-started.html Fancy 100TB of full featured S3 Storage? Checkout the Cloudian® Community Edition!
