Chip,
The source of the problem appears to be clock drift between the SSVM and S3 per
following stack trace:
2013-05-14 06:51:55,400 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:)
Putting directory
/mnt/SecStorage/93fd0cb0-033b-3248-bcd0-ef6d460635ef/template/tmpl/1/5 in S3
bucket jsb-cloudstack-templates.
2013-05-14 06:51:55,401 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:)
Creating S3 client with configuration: [protocol: https, connectionTimeOut:
50000, maxErrorRetry: 3, socketTimeout: 50000]
2013-05-14 06:51:55,403 DEBUG [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-3:) Determining key using account id 1 and template id 5
2013-05-14 06:51:55,403 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:)
Putting file
/mnt/SecStorage/93fd0cb0-033b-3248-bcd0-ef6d460635ef/template/tmpl/1/5/template.properties
into bucket jsb-cloudstack-templates with key
template/tmpl/1/5/template.properties.
2013-05-14 06:51:55,578 ERROR [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-3:) Failed to upload template id 5
Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 970A274E132A9ACB, AWS
Error Code: RequestTimeTooSkewed, AWS Error Message: The difference between the
request time and the current time is too large., S3 Extended Request ID:
9w8a6YBxTn+WlBg96s9stxWuuP8oQ7ksZtg6++wVRHJfE2qmucrilhoEJVetJui4
at
com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:609)
at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:309)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:164)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2863)
at
com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1100)
at
com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:963)
at com.cloud.utils.S3Utils.putDirectory(S3Utils.java:282)
at
com.cloud.storage.resource.NfsSecondaryStorageResource.execute(NfsSecondaryStorageResource.java:414)
at
com.cloud.storage.resource.NfsSecondaryStorageResource.executeRequest(NfsSecondaryStorageResource.java:212)
at com.cloud.agent.Agent.processRequest(Agent.java:525)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
at com.cloud.utils.nio.Task.run(Task.java:83)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
It does not appear that the System VM image has NTP nor does it seem like a
valid assumption that an SSVM would have the connectivity to reach an NTP
server. Therefore, do we have some other means to sync the clock on the SSVM
(e.g. with the host through hypervisor extensions)? In my previous tests, the
SSVM seemed to be syncing its time to the host which addressed the problem.
Thanks,
-John
On May 14, 2013, at 4:58 PM, John Burwell <jburw...@basho.com> wrote:
> Chip,
>
> I am looking into the issue now. There is a failure when the S3 upload
> template command is issued. I working to determine whether or not the cause
> is environmental or code.
>
> Thanks,
> -John
>
> On May 14, 2013, at 4:56 PM, Chip Childers <chip.child...@sungard.com> wrote:
>
>> Hi all,
>>
>> We have a clear bug list (blockers and critical) for 4.1.0. I'm going
>> to cut a new release candidate tonight.
>>
>> If there are *any* outstanding issues known, now's the time to raise
>> them.
>>
>> (I'm specifically looking for an ACK from jburwell here, since he
>> mentioned a possible S3 feature issue. John, if you can please give me
>> a heads up on where you stand in the next few hours, I'd appreciate it.)
>>
>> -chip
>
