Excellent. The link is there now. Thank you Min. I verified that bug and
closed it.
However - now that I can finally click the download link... I ran into
the issue that the link doesn't work on AWS or Cloudian. Please see this
bug for details (latest 4.2 updates included in my test).
https://issues.apache.org/jira/browse/CLOUDSTACK-3341
Tom.
On Tue, 2013-07-02 at 22:54 +0000, Min Chen wrote:
> Hi Tom,
> I investigated this issue through the db dump you provided in the bug,
> this is an issue with our db view template_view creation script, and it
> has been fixed in resolving
> another bug (https://issues.apache.org/jira/browse/CLOUDSTACK-3314). I
> have verified the fix using your db dump on my local setup. Please check
> out latest 4.2 or master code to try again.
>
> Thanks
> -min
>
> On 7/2/13 2:18 PM, "Min Chen" <min.c...@citrix.com> wrote:
>
> >Tom, this seems like an issue with entry stored in our DB. I will take a
> >look at this bug and update you. Just to clarify, this symptom only
> >happens when you register these templates to Amazon S3, not for Cloudian
> >or RiakCS S3, right?
> >
> >Thanks
> >-min
> >
> >On 7/1/13 7:27 PM, "Thomas O'Dowd" <tpod...@cloudian.com> wrote:
> >
> >>Yes thanks Jessica. I re-opened the bug again. I know its not a gui
> >>problem per-say in that the template is not ready to show the download
> >>link. However, it never becomes ready is the actual problem. What sets
> >>the "isready" property to true? As far as I can see, the objects in the
> >>S3 stores (AWS or Cloudian) are complete and from my perspective "ready"
> >>to download/use. It sounds like a bug when registering the template.
> >>
> >>Tom.
> >>
> >>On Mon, 2013-07-01 at 18:54 +0000, Jessica Wang wrote:
> >>> Thomas,
> >>>
> >>> I checked the data you provided.
> >>>
> >>> The reason that the 2 templates("MyTiny", "AnotherTiny") have no
> >>>download button is because they are not ready
> >>> (i.e. their "isready" property is false).
> >>>
> >>> Download button is only available when "isready" property is true.
> >>>
> >>> Jessica
> >>>
> >>> -----Original Message-----
> >>> From: Thomas O'Dowd [mailto:tpod...@cloudian.com]
> >>> Sent: Thursday, June 27, 2013 8:04 PM
> >>> To: Min Chen
> >>> Cc: dev@cloudstack.apache.org; Jessica Wang
> >>> Subject: Re: Query String Request Authentication(QSRA) support by S3
> >>>providers
> >>>
> >>> Hi Min/Jessica,
> >>>
> >>> I attached an image to that issue to show what what my browser is
> >>> showing.
> >>>
> >>> https://issues.apache.org/jira/browse/CLOUDSTACK-3220
> >>>
> >>> Tom.
> >>>
> >>> On Fri, 2013-06-28 at 09:45 +0900, Thomas O'Dowd wrote:
> >>> > Hi Min,
> >>> >
> >>> > Yes. I'll try it again today to check again but when I added Amazon
> >>>S3
> >>> > as the S3 secondary storage and uploaded a template, I was not shown
> >>>the
> >>> > "download template" link. However - for Cloudian S3, I am shown it so
> >>> > I'm wondering why.
> >>> >
> >>> > Tom.
> >>> >
> >>> > On Fri, 2013-06-28 at 00:26 +0000, Min Chen wrote:
> >>> > > Hi Tom,
> >>> > >
> >>> > > Are you saying that you cannot see a Download Template button
> >>> > > from
> >>>UI
> >>> > > when Amazon S3 is added as secondary storage? I only tested with
> >>>RiakCS
> >>> > > and Cloudian, so didn't see this issue. But I am CC Jessica her to
> >>>confirm
> >>> > > what special handling is done in UI to enable/disable a button from
> >>>UI.
> >>> > >
> >>> > > Thanks
> >>> > > -min
> >>> > >
> >>> > > On 6/27/13 5:23 PM, "Thomas O'Dowd" <tpod...@cloudian.com> wrote:
> >>> > >
> >>> > > >Hi Min,
> >>> > > >
> >>> > > >Can you check this bug? I'm trying to test this feature for Amazon
> >>>but
> >>> > > >having no luck getting the Download template link/button to
> >>>appear.
> >>> > > >
> >>> > > >https://issues.apache.org/jira/browse/CLOUDSTACK-3220
> >>> > > >
> >>> > > >Thanks,
> >>> > > >
> >>> > > >Tom.
> >>> > > >
> >>> > > >On Fri, 2013-06-21 at 17:21 +0000, Min Chen wrote:
> >>> > > >> John,
> >>> > > >>
> >>> > > >> For S3, the api call createEntityExtractUrl is done on
> >>>management
> >>> > > >>server
> >>> > > >> side; while for NFS secondary storage, if the implementation of
> >>> > > >> createEntityExtractUrl will involve some code be executed in
> >>>ssvm to
> >>> > > >>copy
> >>> > > >> template from the install location to a public accessible web
> >>>server
> >>> > > >> location.
> >>> > > >> I don't quite understand some of your comments below. This API
> >>>is not
> >>> > > >> used to write any information to S3 bucket/directory. This is
> >>>used for
> >>> > > >> object already existed on S3, and we just provide a URL for user
> >>>to
> >>> > > >> download a template from S3, just like how Amazon provided user
> >>>a way to
> >>> > > >> user to extract a S3 object through generatePresignedUrl. We can
> >>>discuss
> >>> > > >> more on this on collaboration conference.
> >>> > > >>
> >>> > > >> Thanks
> >>> > > >> -min
> >>> > > >>
> >>> > > >>
> >>> > > >>
> >>> > > >> On 6/21/13 7:25 AM, "John Burwell" <jburw...@basho.com> wrote:
> >>> > > >>
> >>> > > >> >Min,
> >>> > > >> >
> >>> > > >> >(I apologize for my belated reply -- I lost track of this draft
> >>>in the
> >>> > > >> >chaos of the last couple of days.)
> >>> > > >> >
> >>> > > >> >Upon further review, I think I feel into the confusion between
> >>> > > >>management
> >>> > > >> >server and ssvm. This code is executing on the management
> >>>server side,
> >>> > > >> >correct? Based on my "corrected" understanding is correct, I
> >>>would
> >>> > > >>like
> >>> > > >> >to amend my thoughts. Namely, I would like to see the driver
> >>> > > >>operations
> >>> > > >> >pushed out to the SSVM where we can use the stream. As I think
> >>>about
> >>> > > >>it,
> >>> > > >> >the management server should not need to interact with the
> >>>driver.
> >>> > > >> >Simply yard up the DataStore attributes + details map and other
> >>>extract
> >>> > > >> >parameters, and send them to the SSVM. Using this information,
> >>>the S3
> >>> > > >> >driver could open a stream to write the template out to the
> >>> > > >> >bucket/directory. I recognize it changes the protocol between
> >>>the
> >>> > > >> >management server and SSVM, but it simply both sides of the
> >>>operation
> >>> > > >>by
> >>> > > >> >allowing the DataStore information to be treated opaquely until
> >>>it is
> >>> > > >> >consumed by the driver to execute the write operation. I also
> >>> > > >>recognize
> >>> > > >> >that we may a little late in the cycle to address it for 4.2,
> >>>and it
> >>> > > >>may
> >>> > > >> >need to be part of the 4.3 enhancements.
> >>> > > >> >
> >>> > > >> >Thanks,
> >>> > > >> >-John
> >>> > > >> >
> >>> > > >> >On Jun 18, 2013, at 3:55 PM, Min Chen <min.c...@citrix.com>
> >>>wrote:
> >>> > > >> >
> >>> > > >> >> John,
> >>> > > >> >> In that case, how do we keep backward compatibility of
> >>> > > >>extractTemplate
> >>> > > >> >> api, which requires a URL in the response?
> >>> > > >> >>
> >>> > > >> >> Thanks
> >>> > > >> >> -min
> >>> > > >> >>
> >>> > > >> >> On 6/18/13 11:53 AM, "John Burwell" <jburw...@basho.com>
> >>>wrote:
> >>> > > >> >>
> >>> > > >> >>> Min,
> >>> > > >> >>>
> >>> > > >> >>> Looking through the code, I think we can simplify driver
> >>>operation
> >>> > > >>and
> >>> > > >> >>> increase robustness by changing
> >>> > > >> >>>ImageStoreDriver#createEntityExtractUrl()
> >>> > > >> >>> : String to ImageStoreDriver#readEntity(Š) : InputStream.
> >>>My first
> >>> > > >> >>> concern with the current implementation is that it
> >>>circumvents any
> >>> > > >> >>> connection pooling/resource management underlying client
> >>>libraries
> >>> > > >> >>> provide. I/O streams provide a higher-level abstraction
> >>>that allows
> >>> > > >> >>> drivers to provide the orchestration components with actual
> >>> > > >>resources
> >>> > > >> >>> rather String references. Second, the current interface
> >>>seems to
> >>> > > >> >>>appears
> >>> > > >> >>> to assume that an http/https URL will be returned. With I/O
> >>> > > >>streams,
> >>> > > >> >>>we
> >>> > > >> >>> can support any client library capable of using the standard
> >>>I/O
> >>> > > >> >>> framework -- enabling us to support other protocols for
> >>>downloading
> >>> > > >> >>> templates in the future (e.g. RBD, local filesystem, NBD,
> >>>etc).
> >>> > > >> >>>
> >>> > > >> >>> Thanks,
> >>> > > >> >>> -John
> >>> > > >> >>>
> >>> > > >> >>> On Jun 18, 2013, at 1:11 PM, Min Chen <min.c...@citrix.com>
> >>>wrote:
> >>> > > >> >>>
> >>> > > >> >>>> A new version of using generatePresignedUrl in
> >>> > > >>S3ImageStoreDriverImpl
> >>> > > >> >>>>is
> >>> > > >> >>>> checked into object_store.
> >>> > > >> >>>>
> >>> > > >> >>>> THanks
> >>> > > >> >>>> -min
> >>> > > >> >>>>
> >>> > > >> >>>> On 6/18/13 8:29 AM, "Min Chen" <min.c...@citrix.com> wrote:
> >>> > > >> >>>>
> >>> > > >> >>>>> Yes, current code is in
> >>> > > >> >>>>>S3ImageStoreDriverImpl.createEntityExtractUrl,
> >>> > > >> >>>>> which has a security issue mentioned in CLOUDSTACK-3030. I
> >>>am
> >>> > > >>going
> >>> > > >> >>>>>to
> >>> > > >> >>>>> change it to use generatePresignedUrl api from AWS S3 api.
> >>> > > >> >>>>>
> >>> > > >> >>>>> Thanks
> >>> > > >> >>>>> -min
> >>> > > >> >>>>>
> >>> > > >> >>>>> From: John Burwell
> >>><jburw...@basho.com<mailto:jburw...@basho.com>>
> >>> > > >> >>>>> Date: Tuesday, June 18, 2013 8:07 AM
> >>> > > >> >>>>> To: Min Chen
> >>><min.c...@citrix.com<mailto:min.c...@citrix.com>>
> >>> > > >> >>>>> Cc: Thomas O'Dowd
> >>> > > >> >>>>><tpod...@cloudian.com<mailto:tpod...@cloudian.com>>,
> >>> > > >> >>>>>
> >>>"dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>"
> >>> > > >> >>>>>
> >>><dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
> >>> > > >> >>>>> Subject: Re: Query String Request Authentication(QSRA)
> >>>support by
> >>> > > >>S3
> >>> > > >> >>>>> providers
> >>> > > >> >>>>>
> >>> > > >> >>>>> Min,
> >>> > > >> >>>>>
> >>> > > >> >>>>> Is the code checked into the object_store branch? If so,
> >>>which
> >>> > > >>lines
> >>> > > >> >>>>> in
> >>> > > >> >>>>> S3TemplateDownloader?
> >>> > > >> >>>>>
> >>> > > >> >>>>> Thanks,
> >>> > > >> >>>>> -John
> >>> > > >> >>>>>
> >>> > > >> >>>>> On Jun 18, 2013, at 12:39 AM, Min Chen
> >>> > > >> >>>>> <min.c...@citrix.com<mailto:min.c...@citrix.com>> wrote:
> >>> > > >> >>>>>
> >>> > > >> >>>>> Hi John,
> >>> > > >> >>>>>
> >>> > > >> >>>>> This is regarding extractTemplate api, where for
> >>>extractable
> >>> > > >> >>>>>template,
> >>> > > >> >>>>> users can click "Download Template" button from UI to get
> >>>a http
> >>> > > >>url
> >>> > > >> >>>>>to
> >>> > > >> >>>>> download the template already stored at S3 without
> >>>providing S3
> >>> > > >> >>>>> credentials. In 4.1, we don't have this issue, since the
> >>>URL
> >>> > > >>returned
> >>> > > >> >>>>> is
> >>> > > >> >>>>> the public web server location hosted in ssvm, and in 4.2,
> >>>we are
> >>> > > >> >>>>> returning URL pointing to s3 object. Without setting ACL
> >>>to the S3
> >>> > > >> >>>>> object, user cannot directly click the URL returned from
> >>> > > >> >>>>> extractTemplate
> >>> > > >> >>>>> api to download the template without providing
> >>>credentials. By
> >>> > > >> >>>>>reading
> >>> > > >> >>>>> the AWS SDK doc today, I ran across the following API that
> >>>I may
> >>> > > >>be
> >>> > > >> >>>>> able
> >>> > > >> >>>>> to use for this purpose:
> >>> > > >> >>>>>
> >>> > > >> >>>>>
> >>> > > >> >>>>>
> >>> > > >> >>>>>
> >>> > > >>
> >>> > >
> >>>>>>>>>>URL<http://java.sun.com/j2se/1.5.0/docs/api/java/net/URL.html?is-
> >>>>>>>>>>e
> >>>>>>>>>>xt
> >>> > > >>>>>>>er
> >>> > > >> >>>>>na
> >>> > > >> >>>>> l=
> >>> > > >> >>>>> true>
> >>> > > >> >>>>>
> >>> > > >> >>>>>
> >>> > > >>
> >>> > >
> >>>>>>>>>>generatePresignedUrl<http://docs.aws.amazon.com/AWSJavaSDK/latest
> >>>>>>>>>>/
> >>>>>>>>>>ja
> >>> > > >>>>>>>va
> >>> > > >> >>>>>do
> >>> > > >> >>>>> c/
> >>> > > >> >>>>>
> >>> > > >> >>>>>
> >>> > > >>
> >>> > >
> >>>>>>>>>>com/amazonaws/services/s3/AmazonS3Client.html#generatePresignedUr
> >>>>>>>>>>l
> >>>>>>>>>>%2
> >>> > > >>>>>>>8j
> >>> > > >> >>>>>av
> >>> > > >> >>>>> a.
> >>> > > >> >>>>>
> >>> > > >> >>>>>
> >>> > > >>
> >>> > >
> >>>>>>>>>>lang.String,%20java.lang.String,%20java.util.Date,%20com.amazonaw
> >>>>>>>>>>s
> >>>>>>>>>>.H
> >>> > > >>>>>>>tt
> >>> > > >> >>>>>pM
> >>> > > >> >>>>> et
> >>> > > >> >>>>>
> >>> > > >> >>>>>
> >>> > > >>
> >>> > >
> >>>>>>>>>>hod%29>(String<http://java.sun.com/j2se/1.5.0/docs/api/java/lang/
> >>>>>>>>>>S
> >>>>>>>>>>tr
> >>> > > >>>>>>>in
> >>> > > >> >>>>>g.
> >>> > > >> >>>>> ht
> >>> > > >> >>>>> ml?is-external=true> bucketName,
> >>> > > >> >>>>>
> >>> > > >> >>>>>
> >>> > > >>
> >>> > >
> >>>>>>>>>>String<http://java.sun.com/j2se/1.5.0/docs/api/java/lang/String.h
> >>>>>>>>>>t
> >>>>>>>>>>ml
> >>> > > >>>>>>>?i
> >>> > > >> >>>>>s-
> >>> > > >> >>>>> ex
> >>> > > >> >>>>> ternal=true> key,
> >>> > > >> >>>>>
> >>> > > >> >>>>>
> >>> > > >>
> >>> > >
> >>>>>>>>>>Date<http://java.sun.com/j2se/1.5.0/docs/api/java/util/Date.html?
> >>>>>>>>>>i
> >>>>>>>>>>s-
> >>> > > >>>>>>>ex
> >>> > > >> >>>>>te
> >>> > > >> >>>>> rn
> >>> > > >> >>>>> al=true> expiration,
> >>> > > >> >>>>>
> >>> > > >> >>>>>
> >>> > > >>
> >>> > >
> >>>>>>>>>>HttpMethod<http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/c
> >>>>>>>>>>o
> >>>>>>>>>>m/
> >>> > > >>>>>>>am
> >>> > > >> >>>>>az
> >>> > > >> >>>>> on
> >>> > > >> >>>>> aws/HttpMethod.html> method)
> >>> > > >> >>>>> Returns a pre-signed URL for accessing an Amazon
> >>>S3
> >>> > > >>resource.
> >>> > > >> >>>>>
> >>> > > >> >>>>> This is along the same line as QSRA mentioned by Tom, by
> >>>wrapped
> >>> > > >>in
> >>> > > >> >>>>> AmazonS3Client for easy consumption. By using this method,
> >>>I think
> >>> > > >> >>>>> that I
> >>> > > >> >>>>> don't need to change ACL of S3 object to open a security
> >>>hole.
> >>> > > >> >>>>>
> >>> > > >> >>>>> Thanks
> >>> > > >> >>>>> -min
> >>> > > >> >>>>>
> >>> > > >> >>>>> From: John Burwell
> >>><jburw...@basho.com<mailto:jburw...@basho.com>>
> >>> > > >> >>>>> Date: Monday, June 17, 2013 7:38 PM
> >>> > > >> >>>>> To: Min Chen
> >>><min.c...@citrix.com<mailto:min.c...@citrix.com>>
> >>> > > >> >>>>> Cc: Thomas O'Dowd
> >>> > > >> >>>>><tpod...@cloudian.com<mailto:tpod...@cloudian.com>>,
> >>> > > >> >>>>>
> >>>"dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>"
> >>> > > >> >>>>>
> >>><dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
> >>> > > >> >>>>> Subject: Re: Query String Request Authentication(QSRA)
> >>>support by
> >>> > > >>S3
> >>> > > >> >>>>> providers
> >>> > > >> >>>>>
> >>> > > >> >>>>> Min,
> >>> > > >> >>>>>
> >>> > > >> >>>>> Why are we mucking with ACLs at all? The best security
> >>>practice
> >>> > > >> >>>>>would
> >>> > > >> >>>>> be
> >>> > > >> >>>>> to create a bucket for CloudStack's use and assign it a
> >>>dedicated
> >>> > > >> >>>>> access
> >>> > > >> >>>>> key and secret key pair with read/write access only to
> >>>that
> >>> > > >>bucket.
> >>> > > >> >>>>> Requiring an administrative account to an object store
> >>>opens an
> >>> > > >> >>>>> unnecessarily large attack surface. Therefore, as
> >>>implemented in
> >>> > > >> >>>>>4.1,
> >>> > > >> >>>>> we
> >>> > > >> >>>>> should defer bucket creation, ACL assignment, and
> >>>credential
> >>> > > >>creation
> >>> > > >> >>>>> to
> >>> > > >> >>>>> the administrator/operator.
> >>> > > >> >>>>>
> >>> > > >> >>>>> Thanks,
> >>> > > >> >>>>> -John
> >>> > > >> >>>>>
> >>> > > >> >>>>> On Jun 17, 2013, at 1:15 PM, Min Chen
> >>> > > >> >>>>> <min.c...@citrix.com<mailto:min.c...@citrix.com>> wrote:
> >>> > > >> >>>>>
> >>> > > >> >>>>> Tom filed a very good bug for ACL setting change on S3
> >>>object when
> >>> > > >> >>>>> users
> >>> > > >> >>>>> issue extractTemplate API
> >>> > > >> >>>>> (https://issues.apache.org/jira/browse/CLOUDSTACK-3030),
> >>>and his
> >>> > > >> >>>>> recommendation of using Query String Request
> >>>Authentication (QSRA)
> >>> > > >> >>>>> alternative sounds like a right approach to fix this bug.
> >>>Before
> >>> > > >> >>>>> implementing it, I would like to confirm if QSRA should be
> >>> > > >>supported
> >>> > > >> >>>>>by
> >>> > > >> >>>>> all S3 providers if they claim that they are AWS s3
> >>>compatible. If
> >>> > > >> >>>>>so,
> >>> > > >> >>>>> we
> >>> > > >> >>>>> will make this assumption in our code. Based on Tom,
> >>>Cloudian is
> >>> > > >> >>>>> supporting it. How about RiakCS, John?
> >>> > > >> >>>>>
> >>> > > >> >>>>> Thanks
> >>> > > >> >>>>> -min
> >>> > > >> >>>>>
> >>> > > >> >>>>>
> >>> > > >> >>>>
> >>> > > >> >>>
> >>> > > >> >>
> >>> > > >> >
> >>> > > >>
> >>> > > >
> >>> > > >--
> >>> > > >Cloudian KK - http://www.cloudian.com/get-started.html
> >>> > > >Fancy 100TB of full featured S3 Storage?
> >>> > > >Checkout the Cloudian(R) Community Edition!
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>
> >>--
> >>Cloudian KK - http://www.cloudian.com/get-started.html
> >>Fancy 100TB of full featured S3 Storage?
> >>Checkout the Cloudian(R) Community Edition!
> >>
> >
>
--
Cloudian KK - http://www.cloudian.com/get-started.html
Fancy 100TB of full featured S3 Storage?
Checkout the Cloudian® Community Edition!
