Chip, After some further discussion about this issue on IRC, Alex and I determined that system VM clock drift issue not only breaks S3, but has other significant impacts that merit it being a blocker for 4.1 (e.g. timestamps of files written by the SSVM being incorrect, log file correlation difficulties, sensitivity of other storage protocols to time sync, etc). Based on this conversation, I have opened CLOUDSTACK-2492 to capture the issue and track resolution.
Thanks, -John On May 14, 2013, at 6:09 PM, John Burwell <jburw...@basho.com> wrote: > Chip, > > The source of the problem appears to be clock drift between the SSVM and S3 > per following stack trace: > > 2013-05-14 06:51:55,400 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:) > Putting directory > /mnt/SecStorage/93fd0cb0-033b-3248-bcd0-ef6d460635ef/template/tmpl/1/5 in S3 > bucket jsb-cloudstack-templates. > 2013-05-14 06:51:55,401 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:) > Creating S3 client with configuration: [protocol: https, connectionTimeOut: > 50000, maxErrorRetry: 3, socketTimeout: 50000] > 2013-05-14 06:51:55,403 DEBUG [storage.resource.NfsSecondaryStorageResource] > (agentRequest-Handler-3:) Determining key using account id 1 and template id 5 > 2013-05-14 06:51:55,403 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:) > Putting file > /mnt/SecStorage/93fd0cb0-033b-3248-bcd0-ef6d460635ef/template/tmpl/1/5/template.properties > into bucket jsb-cloudstack-templates with key > template/tmpl/1/5/template.properties. > 2013-05-14 06:51:55,578 ERROR [storage.resource.NfsSecondaryStorageResource] > (agentRequest-Handler-3:) Failed to upload template id 5 > Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 970A274E132A9ACB, > AWS Error Code: RequestTimeTooSkewed, AWS Error Message: The difference > between the request time and the current time is too large., S3 Extended > Request ID: 9w8a6YBxTn+WlBg96s9stxWuuP8oQ7ksZtg6++wVRHJfE2qmucrilhoEJVetJui4 > at > com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:609) > at > com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:309) > at > com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:164) > at > com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2863) > at > com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1100) > at > com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:963) > at com.cloud.utils.S3Utils.putDirectory(S3Utils.java:282) > at > com.cloud.storage.resource.NfsSecondaryStorageResource.execute(NfsSecondaryStorageResource.java:414) > at > com.cloud.storage.resource.NfsSecondaryStorageResource.executeRequest(NfsSecondaryStorageResource.java:212) > at com.cloud.agent.Agent.processRequest(Agent.java:525) > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852) > at com.cloud.utils.nio.Task.run(Task.java:83) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > It does not appear that the System VM image has NTP nor does it seem like a > valid assumption that an SSVM would have the connectivity to reach an NTP > server. Therefore, do we have some other means to sync the clock on the SSVM > (e.g. with the host through hypervisor extensions)? In my previous tests, > the SSVM seemed to be syncing its time to the host which addressed the > problem. > > Thanks, > -John > > On May 14, 2013, at 4:58 PM, John Burwell <jburw...@basho.com> wrote: > >> Chip, >> >> I am looking into the issue now. There is a failure when the S3 upload >> template command is issued. I working to determine whether or not the cause >> is environmental or code. >> >> Thanks, >> -John >> >> On May 14, 2013, at 4:56 PM, Chip Childers <chip.child...@sungard.com> wrote: >> >>> Hi all, >>> >>> We have a clear bug list (blockers and critical) for 4.1.0. I'm going >>> to cut a new release candidate tonight. >>> >>> If there are *any* outstanding issues known, now's the time to raise >>> them. >>> >>> (I'm specifically looking for an ACK from jburwell here, since he >>> mentioned a possible S3 feature issue. John, if you can please give me >>> a heads up on where you stand in the next few hours, I'd appreciate it.) >>> >>> -chip >> >
