shweta agarwal created CLOUDSTACK-8790: ------------------------------------------
Summary: SSh to few vms are failing Key: CLOUDSTACK-8790 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-8790 Project: CloudStack Issue Type: Bug Security Level: Public (Anyone can view this level - this is the default.) Affects Versions: 4.6.0 Reporter: shweta agarwal Priority: Critical Repro steps: create a ssh keypair Deploy several VMs with keypair field destroy few vms deploy more vms with keypair Bug: Notice keypair for certain VMs are not set hence ssh for those vms are failing MS log for those vms shows : 2015-08-31 07:30:12,149 ERROR [c.c.u.s.SshHelper] (DirectAgent-210:ctx-8227b3be) SSH execution of command /opt/cloud/bin/router_proxy.sh update_config.py 169.254.2.76 vm_dhcp_entry.json has an error status code in return. result output: [INFO] update_config.py :: Processing incoming file => vm_dhcp_entry.json [INFO] Processing JSON file vm_dhcp_entry.json VM-6a153536-58c4-4951-b99f-478e2c3ca097 VM-2619da1e-43b2-4cff-8cb5-58d7cc3709d3 VM-b9b0adf4-6e7f-488a-b3ae-53eb8eebf779 VM-2619da1e-43b2-4cff-8cb5-58d7cc3709d3 VM-14c664dd-7a0b-453a-bfa1-57ca5227c79f VM-2619da1e-43b2-4cff-8cb5-58d7cc3709d3 VM-ed569557-34e5-4bc1-a395-4d868cfeb54a VM-2619da1e-43b2-4cff-8cb5-58d7cc3709d3 VM-ae103b7d-33f2-4011-9afa-b5e711722f9f VM-2619da1e-43b2-4cff-8cb5-58d7cc3709d3 VM-aa122877-abde-4674-962d-0cc5a8caf4f7 VM-2619da1e-43b2-4cff-8cb5-58d7cc3709d3 VM-3cf6e5d5-5abc-491f-a9dd-994ff51cfa5e VM-2619da1e-43b2-4cff-8cb5-58d7cc3709d3 VM-56dfbd5a-3783-48c8-9366-cf348bf4e1d4 VM-2619da1e-43b2-4cff-8cb5-58d7cc3709d3 VM-e3cee5b1-4631-45d9-9ee3-dc29c91e6828 VM-2619da1e-43b2-4cff-8cb5-58d7cc3709d3 Traceback (most recent call last): File "/opt/cloud/bin/update_config.py", line 140, in <module> process_file() File "/opt/cloud/bin/update_config.py", line 54, in process_file finish_config() File "/opt/cloud/bin/update_config.py", line 44, in finish_config returncode = configure.main([]) File "/opt/cloud/bin/configure.py", line 664, in main metadata.process() File "/opt/cloud/bin/configure.py", line 257, in process self.__createfile(ip, folder, file, data) File "/opt/cloud/bin/configure.py", line 273, in __createfile data = base64.b64decode(data) File "/usr/lib/python2.7/base64.py", line 76, in b64decode raise TypeError(msg) TypeError: Incorrect padding 2015-08-31 07:30:12,149 DEBUG [c.c.a.r.v.VirtualRoutingResource] (DirectAgent-210:ctx-8227b3be) Processing ScriptConfigItem, executing update_config.py vm_dhcp_entry.json took 574ms 2015-08-31 07:30:12,149 WARN [c.c.a.r.v.VirtualRoutingResource] (DirectAgent-210:ctx-8227b3be) Expected 1 answers while executing DhcpEntryCommand but received 2 2015-08-31 07:30:12,149 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-210:ctx-8227b3be) Seq 1-4238168724332367731: Response Received: 2015-08-31 07:30:12,151 DEBUG [c.c.a.t.Request] (DirectAgent-210:ctx-8227b3be) Seq 1-4238168724332367731: Processing: { Ans: , MgmtId: 233845177509810, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: ","null - failed: [INFO] update_config.py :: Processing incoming file => vm_dhcp_entry.json\n[INFO] Processing JSON file vm_dhcp_entry.json\nVM-6a153536-58c4-4951-b99f-478e2c3ca097 VM-2619da1e-43b2-4cff-8cb5-58d7cc3709d3\nVM-b9b0adf4-6e7f-488a-b3ae-53eb8eebf779 VM-2619da1e-43b2-4cff-8cb5-58d7cc3709d3\nVM-14c664dd-7a0b-453a-bfa1-57ca5227c79f VM-2619da1e-43b2-4cff-8cb5-58d7cc3709d3\nVM-ed569557-34e5-4bc1-a395-4d868cfeb54a VM-2619da1e-43b2-4cff-8cb5-58d7cc3709d3\nVM-ae103b7d-33f2-4011-9afa-b5e711722f9f VM-2619da1e-43b2-4cff-8cb5-58d7cc3709d3\nVM-aa122877-abde-4674-962d-0cc5a8caf4f7 VM-2619da1e-43b2-4cff-8cb5-58d7cc3709d3\nVM-3cf6e5d5-5abc-491f-a9dd-994ff51cfa5e VM-2619da1e-43b2-4cff-8cb5-58d7cc3709d3\nVM-56dfbd5a-3783-48c8-9366-cf348bf4e1d4 VM-2619da1e-43b2-4cff-8cb5-58d7cc3709d3\nVM-e3cee5b1-4631-45d9-9ee3-dc29c91e6828 VM-2619da1e-43b2-4cff-8cb5-58d7cc3709d3\nTraceback (most recent call last):\n File \"/opt/cloud/bin/update_config.py\", line 140, in <module>\n process_file()\n File \"/opt/cloud/bin/update_config.py\", line 54, in process_file\n finish_config()\n File \"/opt/cloud/bin/update_config.py\", line 44, in finish_config\n returncode = configure.main([])\n File \"/opt/cloud/bin/configure.py\", line 664, in main\n metadata.process()\n File \"/opt/cloud/bin/configure.py\", line 257, in process\n self.__createfile(ip, folder, file, data)\n File \"/opt/cloud/bin/configure.py\", line 273, in __createfile\n data = base64.b64decode(data)\n File \"/usr/lib/python2.7/base64.py\", line 76, in b64decode\n raise TypeError(msg)\nTypeError: Incorrect padding\n"],"result":false,"wait":0}}] } 2015-08-31 07:30:12,151 DEBUG [c.c.a.t.Request] (Work-Job-Executor-123:ctx-37118a0d job-282/job-283 ctx-7ca63592) Seq 1-4238168724332367731: Received: { Ans: , MgmtId: 233845177509810, via: 1, Ver: v1, Flags: 10, { GroupAnswer } } 2015-08-31 07:30:12,152 DEBUG [c.c.a.ApiServlet] (catalina-exec-6:ctx-90336697) ===START=== 0:0:0:0:0:0:0:1 -- GET signatureversion=3&expires=2015-08-31T11%3A40%3A12%2B0000&jobid=5cf90ad7-d759-4d5c-8258-9eb1e7c8e35c&sessionkey=uK5fXQwO_qeAogIVulVqqM6l0-c&command=queryAsyncJobResult&response=json 2015-08-31 07:30:12,155 DEBUG [c.c.n.NetworkMo -- This message was sent by Atlassian JIRA (v6.3.4#6332)