Site 2 Site VPN on VPC not working after upgrading ACS from 4.2 to 4.8.1.1
Hi, Another issue we noted after upgrading ACS from 4.2 to 4.8.1.1 is that site-to-site VPN is not working. From GUI, I cannot even go to Home > Network > Select view: VPN customer gateway with below error messages shown on the GUI: Caught: com.mysql.jdbc.JDBC4PreparedStatement@45ae1e69: SELECT s2s_customer_gateway.id, s2s_customer_gateway.uuid, s2s_customer_gateway.name, s2s_customer_gateway.gateway_ip, s2s_customer_gateway.guest_cidr_list, s2s_customer_gateway.ipsec_psk, s2s_customer_gateway.ike_policy, s2s_customer_gateway.esp_policy, s2s_customer_gateway.ike_lifetime, s2s_customer_gateway.esp_lifetime, s2s_customer_gateway.dpd, s2s_customer_gateway.force_encap, s2s_customer_gateway.domain_id, s2s_customer_gateway.account_id, s2s_customer_gateway.removed FROM s2s_customer_gateway INNER JOIN account ON s2s_customer_gateway.account_id=account.id WHERE s2s_customer_gateway.removed IS NULL AND (account.type != 5 ) ORDER BY s2s_customer_gateway.id DESC LIMIT 0, 20 This is the error messages on CloudStack management server: 2016-11-06 18:00:30,218 DEBUG [c.c.u.c.DBEncryptionUtil] (http-8080-2:ctx-da483727 ctx-695addac) (logid:3d3a6225) Error while decrypting: 2016-11-06 18:00:30,218 ERROR [c.c.a.ApiServer] (http-8080-2:ctx-da483727 ctx-695addac) (logid:3d3a6225) unhandled exception executing api command: [Ljava.lang.String;@ 6ad8a1d4 com.cloud.utils.exception.CloudRuntimeException: Caught: com.mysql.jdbc.JDBC4PreparedStatement@55edbe9c: SELECT s2s_customer_gateway.id, s2s_customer_gateway.uuid, s2s_ customer_gateway.name, s2s_customer_gateway.gateway_ip, s2s_customer_gateway.guest_cidr_list, s2s_customer_gateway.ipsec_psk, s2s_customer_gateway.ike_policy, s2s_custo mer_gateway.esp_policy, s2s_customer_gateway.ike_lifetime, s2s_customer_gateway.esp_lifetime, s2s_customer_gateway.dpd, s2s_customer_gateway.force_encap, s2s_customer_g ateway.domain_id, s2s_customer_gateway.account_id, s2s_customer_gateway.removed FROM s2s_customer_gateway INNER JOIN account ON s2s_customer_gateway.account_id=account .id WHERE s2s_customer_gateway.removed IS NULL AND (account.type != 5 ) ORDER BY s2s_customer_gateway.id DESC LIMIT 0, 20 Note that on "Error while decrypting: ", I checked that is the cleartext IPsec preshared key (ipsec_psk field on s2s_customer_gateway table on the database). It seems that CloudStack tries to decrypt a clear-text / non-encrypted password? Does the ipsec_psk field on the database supposed to store the encrypted IPsec pre-shared key for the VPN connection under CloudStack version 4.8.1.1? I am able to execute the SELECT command on the database directly without any issues. Any help is greatly appreciated. Thank you.
Re: Site 2 Site VPN on VPC not working after upgrading ACS from 4.2 to 4.8.1.1
Hi, [RESOLVED] Would like to share that I managed to find the workaround to the problem by emptying the password ipsec_psk field on the entry inside s2s_customer_gateway table, delete the VPN connection, modify the VPN customer gateway with the new password (we will notice that the encrypted format of the password will be populated into the ipsec_psk field), and then recreate the VPN connection. The VPN connection will then be established. Hope the above can help others. Thank you. On Sun, Nov 6, 2016 at 6:16 PM, Cloud List wrote: > Hi, > > Another issue we noted after upgrading ACS from 4.2 to 4.8.1.1 is that > site-to-site VPN is not working. From GUI, I cannot even go to Home > > Network > Select view: VPN customer gateway with below error messages shown > on the GUI: > > > Caught: com.mysql.jdbc.JDBC4PreparedStatement@45ae1e69: SELECT > s2s_customer_gateway.id, s2s_customer_gateway.uuid, > s2s_customer_gateway.name, s2s_customer_gateway.gateway_ip, > s2s_customer_gateway.guest_cidr_list, s2s_customer_gateway.ipsec_psk, > s2s_customer_gateway.ike_policy, s2s_customer_gateway.esp_policy, > s2s_customer_gateway.ike_lifetime, s2s_customer_gateway.esp_lifetime, > s2s_customer_gateway.dpd, s2s_customer_gateway.force_encap, > s2s_customer_gateway.domain_id, s2s_customer_gateway.account_id, > s2s_customer_gateway.removed FROM s2s_customer_gateway INNER JOIN account > ON s2s_customer_gateway.account_id=account.id WHERE > s2s_customer_gateway.removed IS NULL AND (account.type != 5 ) ORDER BY > s2s_customer_gateway.id DESC LIMIT 0, 20 > > > This is the error messages on CloudStack management server: > > > 2016-11-06 18:00:30,218 DEBUG [c.c.u.c.DBEncryptionUtil] > (http-8080-2:ctx-da483727 ctx-695addac) (logid:3d3a6225) Error while > decrypting: > 2016-11-06 18:00:30,218 ERROR [c.c.a.ApiServer] (http-8080-2:ctx-da483727 > ctx-695addac) (logid:3d3a6225) unhandled exception executing api command: > [Ljava.lang.String;@ > 6ad8a1d4 > com.cloud.utils.exception.CloudRuntimeException: Caught: com.mysql.jdbc. > JDBC4PreparedStatement@55edbe9c: SELECT s2s_customer_gateway.id, > s2s_customer_gateway.uuid, s2s_ > customer_gateway.name, s2s_customer_gateway.gateway_ip, > s2s_customer_gateway.guest_cidr_list, s2s_customer_gateway.ipsec_psk, > s2s_customer_gateway.ike_policy, s2s_custo > mer_gateway.esp_policy, s2s_customer_gateway.ike_lifetime, > s2s_customer_gateway.esp_lifetime, s2s_customer_gateway.dpd, > s2s_customer_gateway.force_encap, s2s_customer_g > ateway.domain_id, s2s_customer_gateway.account_id, > s2s_customer_gateway.removed FROM s2s_customer_gateway INNER JOIN account > ON s2s_customer_gateway.account_id=account > .id WHERE s2s_customer_gateway.removed IS NULL AND (account.type != 5 ) > ORDER BY s2s_customer_gateway.id DESC LIMIT 0, 20 > > > Note that on "Error while decrypting: ", I checked that > is the cleartext IPsec preshared key (ipsec_psk field on > s2s_customer_gateway table on the database). It seems that CloudStack tries > to decrypt a clear-text / non-encrypted password? Does the ipsec_psk field > on the database supposed to store the encrypted IPsec pre-shared key for > the VPN connection under CloudStack version 4.8.1.1? > > I am able to execute the SELECT command on the database directly without > any issues. > > Any help is greatly appreciated. > > Thank you. >
Re: Long delay in listDomains API response
Can you grep for this line ‘ctx-3f3ff343’ in your log and post it here? On 11/5/16, 11:01 PM, "Cloud List" wrote: Dear all, After researching further, I found out that our problem is exactly the one reported on this JIRA bug: https://issues.apache.org/jira/browse/CLOUDSTACK-9405 I am cc-ing dsclose and Wei Zhou on this email. Anyone know if there's any resolution to the above problem/bug? On the bug report, Wei Zhou mentioned that he added domain level resource limitation, may I know if this limitation is configurable? I can't find it on the Global Settings. Looking forward to your reply, thank you. Cheers. On Sun, Nov 6, 2016 at 10:30 AM, Cloud List wrote: > Thanks Sergey for your assistance. > > I have enabled the TRACE logs as requested. After the listDomains API > call, I see a lot of DB connections being created to the database, > preparing SELECT statements and then immediately being closed thereafter. > This happens many many times in the loop in the course of 10-15 seconds. > For example: > > Creating a DB connection with no txn > Preparing: SELECT > Closing: com.mysql.jdbc.PreparedStatement > Closing DB connection > > The above continues in a loop for 10-15 seconds, sometimes more. Is it > normal? Could it be something wrong during the database schema upgrade when > I upgraded from 4.2 to 4.8? > > Excerpts of the TRACE logs are below. Any advice on how I can resolve the > problem is highly appreciated. > > === > 2016-11-06 09:36:04,893 DEBUG [c.c.a.ApiServlet] > (http-8080-10:ctx-3f3ff343) (logid:0429eaf0) ===START=== X.X.X.X -- GET > apikey=&command=listDomains&response=json&signature= > 2016-11-06 09:36:04,893 TRACE [c.c.u.d.T.Transaction] > (http-8080-10:ctx-3f3ff343 ctx-e486fd2b) (logid:0429eaf0) Using current > transaction: http-8080-10 : http-8080-10, > 2016-11-06 09:36:04,893 TRACE [c.c.u.d.T.Transaction] > (http-8080-10:ctx-3f3ff343 ctx-e486fd2b) (logid:0429eaf0) Using current > transaction: http-8080-10 : http-8080-10, > 2016-11-06 09:36:04,894 TRACE [c.c.u.d.T.Statement] (StatsCollector-2:ctx-293a2a9e) > (logid:725b6dd7) Closing: com.mysql.jdbc.JDBC4PreparedStatement@1bda5f56: > SELECT vm_instance.id, vm_instance.name, vm_instance.vnc_password, > vm_instance.proxy_id, vm_instance.proxy_assign_time, vm_instance.state, > vm_instance.private_ip_address, vm_instance.instance_name, > vm_instance.vm_template_id, vm_instance.guest_os_id, vm_instance.host_id, > vm_instance.last_host_id, vm_instance.pod_id, vm_instance.private_mac_address, > vm_instance.data_center_id, vm_instance.vm_type, vm_instance.ha_enabled, > vm_instance.display_vm, vm_instance.limit_cpu_use, > vm_instance.update_count, vm_instance.created, vm_instance.removed, > vm_instance.update_time, vm_instance.domain_id, vm_instance.account_id, > vm_instance.user_id, vm_instance.service_offering_id, > vm_instance.reservation_id, vm_instance.hypervisor_type, > vm_instance.dynamically_scalable, vm_instance.uuid, > vm_instance.disk_offering_id, vm_instance.power_state, > vm_instance.power_state_update_time, vm_instance.power_state_update_count, > vm_instance.power_host, user_vm.iso_id, user_vm.user_data, > user_vm.display_name, user_vm.update_parameters FROM user_vm INNER JOIN > vm_instance ON user_vm.id=vm_instance.id WHERE vm_instance.type='User' > AND user_vm.id = 4088 AND vm_instance.removed IS NULL > 2016-11-06 09:36:04,894 TRACE [c.c.u.d.T.Connection] (StatsCollector-2:ctx-293a2a9e) > (logid:725b6dd7) Closing DB connection: dbconn1208177431 > 2016-11-06 09:36:04,894 TRACE [c.c.u.d.T.Transaction] > (StatsCollector-2:ctx-293a2a9e) (logid:725b6dd7) Using current > transaction: StatsCollector-2 : StatsCollector-2, > 2016-11-06 09:36:04,895 TRACE [c.c.u.d.T.Connection] > (http-8080-10:ctx-3f3ff343 ctx-e486fd2b) (logid:0429eaf0) Creating a DB > connection with no txn: for 0: dbconn1379204259. Stack: > -TransactionLegacy.prepareStatement:457-TransactionLegacy. > prepareAutoCloseStatement:450-AccountDaoImpl.findUserAccountByApiKey:129- > GeneratedMethodAccessor234.invoke:-1-DelegatingMethodAccessorImpl. > invoke:43-Method.invoke:606-AopUtils.invokeJoinpointUsingReflection:317- > ReflectiveMethodInvocation.invokeJoinpoint:183-ReflectiveMethodInvocation. > proceed:150-TransactionContextInterceptor.invoke:34- > ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor. > invoke:91 > 2016-11-06 09:36:04,895 TRACE [c.c.u.d.T.Statement] > (http-8080-10:ctx-3f3ff343 ctx-e486fd2b) (logid:0429eaf0) Preparing: SELECT > u.id, u.username, u.account_id, u.secret_key, u.state, a.id, > a.account_name, a.type, a.domain_id, a.state FROM `cloud`.`user` u,
Re: Long delay in listDomains API response
Hi Sergey, Thanks for your reply. The grep result is a bit long, total 64K lines, so I have uploaded it here: https://dl.dropboxusercontent.com/u/24788636/ctx-3f3ff343.txt It seems there are a lot of unnecessary creating and closing of connections to the DB, causing the delay? Looking forward to your reply, thank you. Cheers. On Sun, Nov 6, 2016 at 11:30 PM, Sergey Levitskiy < sergey.levits...@autodesk.com> wrote: > Can you grep for this line ‘ctx-3f3ff343’ in your log and post it here? > > > On 11/5/16, 11:01 PM, "Cloud List" wrote: > > Dear all, > > After researching further, I found out that our problem is exactly the > one > reported on this JIRA bug: > > https://issues.apache.org/jira/browse/CLOUDSTACK-9405 > > I am cc-ing dsclose and Wei Zhou on this email. Anyone know if there's > any > resolution to the above problem/bug? > > On the bug report, Wei Zhou mentioned that he added domain level > resource > limitation, may I know if this limitation is configurable? I can't > find it > on the Global Settings. > > Looking forward to your reply, thank you. > > Cheers. > > > On Sun, Nov 6, 2016 at 10:30 AM, Cloud List > wrote: > > > Thanks Sergey for your assistance. > > > > I have enabled the TRACE logs as requested. After the listDomains API > > call, I see a lot of DB connections being created to the database, > > preparing SELECT statements and then immediately being closed > thereafter. > > This happens many many times in the loop in the course of 10-15 > seconds. > > For example: > > > > Creating a DB connection with no txn > > Preparing: SELECT > > Closing: com.mysql.jdbc.PreparedStatement > > Closing DB connection > > > > The above continues in a loop for 10-15 seconds, sometimes more. Is > it > > normal? Could it be something wrong during the database schema > upgrade when > > I upgraded from 4.2 to 4.8? > > > > Excerpts of the TRACE logs are below. Any advice on how I can > resolve the > > problem is highly appreciated. > > > > === > > 2016-11-06 09:36:04,893 DEBUG [c.c.a.ApiServlet] > > (http-8080-10:ctx-3f3ff343) (logid:0429eaf0) ===START=== X.X.X.X -- > GET > > apikey=&command=listDomains&response=json&signature= > > 2016-11-06 09:36:04,893 TRACE [c.c.u.d.T.Transaction] > > (http-8080-10:ctx-3f3ff343 ctx-e486fd2b) (logid:0429eaf0) Using > current > > transaction: http-8080-10 : http-8080-10, > > 2016-11-06 09:36:04,893 TRACE [c.c.u.d.T.Transaction] > > (http-8080-10:ctx-3f3ff343 ctx-e486fd2b) (logid:0429eaf0) Using > current > > transaction: http-8080-10 : http-8080-10, > > 2016-11-06 09:36:04,894 TRACE [c.c.u.d.T.Statement] > (StatsCollector-2:ctx-293a2a9e) > > (logid:725b6dd7) Closing: com.mysql.jdbc.JDBC4PreparedStatement@ > 1bda5f56: > > SELECT vm_instance.id, vm_instance.name, vm_instance.vnc_password, > > vm_instance.proxy_id, vm_instance.proxy_assign_time, > vm_instance.state, > > vm_instance.private_ip_address, vm_instance.instance_name, > > vm_instance.vm_template_id, vm_instance.guest_os_id, > vm_instance.host_id, > > vm_instance.last_host_id, vm_instance.pod_id, > vm_instance.private_mac_address, > > vm_instance.data_center_id, vm_instance.vm_type, > vm_instance.ha_enabled, > > vm_instance.display_vm, vm_instance.limit_cpu_use, > > vm_instance.update_count, vm_instance.created, vm_instance.removed, > > vm_instance.update_time, vm_instance.domain_id, > vm_instance.account_id, > > vm_instance.user_id, vm_instance.service_offering_id, > > vm_instance.reservation_id, vm_instance.hypervisor_type, > > vm_instance.dynamically_scalable, vm_instance.uuid, > > vm_instance.disk_offering_id, vm_instance.power_state, > > vm_instance.power_state_update_time, vm_instance.power_state_ > update_count, > > vm_instance.power_host, user_vm.iso_id, user_vm.user_data, > > user_vm.display_name, user_vm.update_parameters FROM user_vm INNER > JOIN > > vm_instance ON user_vm.id=vm_instance.id WHERE > vm_instance.type='User' > > AND user_vm.id = 4088 AND vm_instance.removed IS NULL > > 2016-11-06 09:36:04,894 TRACE [c.c.u.d.T.Connection] > (StatsCollector-2:ctx-293a2a9e) > > (logid:725b6dd7) Closing DB connection: dbconn1208177431 > > 2016-11-06 09:36:04,894 TRACE [c.c.u.d.T.Transaction] > > (StatsCollector-2:ctx-293a2a9e) (logid:725b6dd7) Using current > > transaction: StatsCollector-2 : StatsCollector-2, > > 2016-11-06 09:36:04,895 TRACE [c.c.u.d.T.Connection] > > (http-8080-10:ctx-3f3ff343 ctx-e486fd2b) (logid:0429eaf0) Creating a > DB > > connection with no txn: for 0: dbconn1379204259. Stack: > > -TransactionLegacy.prepareStatement:457-TransactionLegacy. > > prepareAutoCloseStatement:450-AccountDaoImpl. > findUserAccountByApiKey:129- > > GeneratedMethodAccessor234.invoke:-1-DelegatingMethodAcces
Re: Long delay in listDomains API response
I cannot say it is unnecessary or not for now. There are indeed many db connections when get resource limtiation/count of domains. A workaound is (1) listDomainChildren gets id of domain and sub-domains, then (2) listDomains with specified id. -Wei 2016-11-06 17:23 GMT+01:00 Cloud List : > Hi Sergey, > > Thanks for your reply. The grep result is a bit long, total 64K lines, so > I have uploaded it here: > > https://dl.dropboxusercontent.com/u/24788636/ctx-3f3ff343.txt > > It seems there are a lot of unnecessary creating and closing of > connections to the DB, causing the delay? > > Looking forward to your reply, thank you. > > Cheers. > > > On Sun, Nov 6, 2016 at 11:30 PM, Sergey Levitskiy < > sergey.levits...@autodesk.com> wrote: > >> Can you grep for this line ‘ctx-3f3ff343’ in your log and post it here? >> >> >> On 11/5/16, 11:01 PM, "Cloud List" wrote: >> >> Dear all, >> >> After researching further, I found out that our problem is exactly >> the one >> reported on this JIRA bug: >> >> https://issues.apache.org/jira/browse/CLOUDSTACK-9405 >> >> I am cc-ing dsclose and Wei Zhou on this email. Anyone know if >> there's any >> resolution to the above problem/bug? >> >> On the bug report, Wei Zhou mentioned that he added domain level >> resource >> limitation, may I know if this limitation is configurable? I can't >> find it >> on the Global Settings. >> >> Looking forward to your reply, thank you. >> >> Cheers. >> >> >> On Sun, Nov 6, 2016 at 10:30 AM, Cloud List >> wrote: >> >> > Thanks Sergey for your assistance. >> > >> > I have enabled the TRACE logs as requested. After the listDomains >> API >> > call, I see a lot of DB connections being created to the database, >> > preparing SELECT statements and then immediately being closed >> thereafter. >> > This happens many many times in the loop in the course of 10-15 >> seconds. >> > For example: >> > >> > Creating a DB connection with no txn >> > Preparing: SELECT >> > Closing: com.mysql.jdbc.PreparedStatement >> > Closing DB connection >> > >> > The above continues in a loop for 10-15 seconds, sometimes more. Is >> it >> > normal? Could it be something wrong during the database schema >> upgrade when >> > I upgraded from 4.2 to 4.8? >> > >> > Excerpts of the TRACE logs are below. Any advice on how I can >> resolve the >> > problem is highly appreciated. >> > >> > === >> > 2016-11-06 09:36:04,893 DEBUG [c.c.a.ApiServlet] >> > (http-8080-10:ctx-3f3ff343) (logid:0429eaf0) ===START=== X.X.X.X >> -- GET >> > apikey=&command=listDomains&response=json&signature= >> > 2016-11-06 09:36:04,893 TRACE [c.c.u.d.T.Transaction] >> > (http-8080-10:ctx-3f3ff343 ctx-e486fd2b) (logid:0429eaf0) Using >> current >> > transaction: http-8080-10 : http-8080-10, >> > 2016-11-06 09:36:04,893 TRACE [c.c.u.d.T.Transaction] >> > (http-8080-10:ctx-3f3ff343 ctx-e486fd2b) (logid:0429eaf0) Using >> current >> > transaction: http-8080-10 : http-8080-10, >> > 2016-11-06 09:36:04,894 TRACE [c.c.u.d.T.Statement] >> (StatsCollector-2:ctx-293a2a9e) >> > (logid:725b6dd7) Closing: com.mysql.jdbc.JDBC4PreparedSt >> atement@1bda5f56: >> > SELECT vm_instance.id, vm_instance.name, vm_instance.vnc_password, >> > vm_instance.proxy_id, vm_instance.proxy_assign_time, >> vm_instance.state, >> > vm_instance.private_ip_address, vm_instance.instance_name, >> > vm_instance.vm_template_id, vm_instance.guest_os_id, >> vm_instance.host_id, >> > vm_instance.last_host_id, vm_instance.pod_id, >> vm_instance.private_mac_address, >> > vm_instance.data_center_id, vm_instance.vm_type, >> vm_instance.ha_enabled, >> > vm_instance.display_vm, vm_instance.limit_cpu_use, >> > vm_instance.update_count, vm_instance.created, vm_instance.removed, >> > vm_instance.update_time, vm_instance.domain_id, >> vm_instance.account_id, >> > vm_instance.user_id, vm_instance.service_offering_id, >> > vm_instance.reservation_id, vm_instance.hypervisor_type, >> > vm_instance.dynamically_scalable, vm_instance.uuid, >> > vm_instance.disk_offering_id, vm_instance.power_state, >> > vm_instance.power_state_update_time, vm_instance.power_state_update >> _count, >> > vm_instance.power_host, user_vm.iso_id, user_vm.user_data, >> > user_vm.display_name, user_vm.update_parameters FROM user_vm INNER >> JOIN >> > vm_instance ON user_vm.id=vm_instance.id WHERE >> vm_instance.type='User' >> > AND user_vm.id = 4088 AND vm_instance.removed IS NULL >> > 2016-11-06 09:36:04,894 TRACE [c.c.u.d.T.Connection] >> (StatsCollector-2:ctx-293a2a9e) >> > (logid:725b6dd7) Closing DB connection: dbconn1208177431 >> > 2016-11-06 09:36:04,894 TRACE [c.c.u.d.T.Transaction] >> > (StatsCollector-2:ctx-293a2a9e) (logid:725b6dd7) Using current >> > transaction: StatsCollector-2 : StatsCollector
Re: Long delay in listDomains API response
Hi Wei, Thanks for your reply. Is it possible for us to remove the resource limitation feature -- any global settings that we can set to turn this off? If not, would you be able to let us know where on the source code this is being set, so that we can try changing it and recompile ACS to turn off the feature? If it's possible, how would be the impact to CloudStack operations in general? May I know what was the original purpose the limitation is being put in the first place? Thanks for the workaround recommendation -- will forward to our application vendor to see if the app can be fine-tuned following your recommendation. Looking forward to your reply, thank you. Cheers. On Mon, Nov 7, 2016 at 12:42 AM, Wei ZHOU wrote: > I cannot say it is unnecessary or not for now. > There are indeed many db connections when get resource limtiation/count of > domains. > > A workaound is (1) listDomainChildren gets id of domain and sub-domains, > then (2) listDomains with specified id. > > -Wei > > > > 2016-11-06 17:23 GMT+01:00 Cloud List : > >> Hi Sergey, >> >> Thanks for your reply. The grep result is a bit long, total 64K lines, so >> I have uploaded it here: >> >> https://dl.dropboxusercontent.com/u/24788636/ctx-3f3ff343.txt >> >> It seems there are a lot of unnecessary creating and closing of >> connections to the DB, causing the delay? >> >> Looking forward to your reply, thank you. >> >> Cheers. >> >> >> On Sun, Nov 6, 2016 at 11:30 PM, Sergey Levitskiy < >> sergey.levits...@autodesk.com> wrote: >> >>> Can you grep for this line ‘ctx-3f3ff343’ in your log and post it here? >>> >>> >>> On 11/5/16, 11:01 PM, "Cloud List" wrote: >>> >>> Dear all, >>> >>> After researching further, I found out that our problem is exactly >>> the one >>> reported on this JIRA bug: >>> >>> https://issues.apache.org/jira/browse/CLOUDSTACK-9405 >>> >>> I am cc-ing dsclose and Wei Zhou on this email. Anyone know if >>> there's any >>> resolution to the above problem/bug? >>> >>> On the bug report, Wei Zhou mentioned that he added domain level >>> resource >>> limitation, may I know if this limitation is configurable? I can't >>> find it >>> on the Global Settings. >>> >>> Looking forward to your reply, thank you. >>> >>> Cheers. >>> >>> >>> On Sun, Nov 6, 2016 at 10:30 AM, Cloud List >>> wrote: >>> >>> > Thanks Sergey for your assistance. >>> > >>> > I have enabled the TRACE logs as requested. After the listDomains >>> API >>> > call, I see a lot of DB connections being created to the database, >>> > preparing SELECT statements and then immediately being closed >>> thereafter. >>> > This happens many many times in the loop in the course of 10-15 >>> seconds. >>> > For example: >>> > >>> > Creating a DB connection with no txn >>> > Preparing: SELECT >>> > Closing: com.mysql.jdbc.PreparedStatement >>> > Closing DB connection >>> > >>> > The above continues in a loop for 10-15 seconds, sometimes more. >>> Is it >>> > normal? Could it be something wrong during the database schema >>> upgrade when >>> > I upgraded from 4.2 to 4.8? >>> > >>> > Excerpts of the TRACE logs are below. Any advice on how I can >>> resolve the >>> > problem is highly appreciated. >>> > >>> > === >>> > 2016-11-06 09:36:04,893 DEBUG [c.c.a.ApiServlet] >>> > (http-8080-10:ctx-3f3ff343) (logid:0429eaf0) ===START=== X.X.X.X >>> -- GET >>> > apikey=&command=listDomains&response=json&signature= >>> > 2016-11-06 09:36:04,893 TRACE [c.c.u.d.T.Transaction] >>> > (http-8080-10:ctx-3f3ff343 ctx-e486fd2b) (logid:0429eaf0) Using >>> current >>> > transaction: http-8080-10 : http-8080-10, >>> > 2016-11-06 09:36:04,893 TRACE [c.c.u.d.T.Transaction] >>> > (http-8080-10:ctx-3f3ff343 ctx-e486fd2b) (logid:0429eaf0) Using >>> current >>> > transaction: http-8080-10 : http-8080-10, >>> > 2016-11-06 09:36:04,894 TRACE [c.c.u.d.T.Statement] >>> (StatsCollector-2:ctx-293a2a9e) >>> > (logid:725b6dd7) Closing: com.mysql.jdbc.JDBC4PreparedSt >>> atement@1bda5f56: >>> > SELECT vm_instance.id, vm_instance.name, vm_instance.vnc_password, >>> > vm_instance.proxy_id, vm_instance.proxy_assign_time, >>> vm_instance.state, >>> > vm_instance.private_ip_address, vm_instance.instance_name, >>> > vm_instance.vm_template_id, vm_instance.guest_os_id, >>> vm_instance.host_id, >>> > vm_instance.last_host_id, vm_instance.pod_id, >>> vm_instance.private_mac_address, >>> > vm_instance.data_center_id, vm_instance.vm_type, >>> vm_instance.ha_enabled, >>> > vm_instance.display_vm, vm_instance.limit_cpu_use, >>> > vm_instance.update_count, vm_instance.created, vm_instance.removed, >>> > vm_instance.update_time, vm_instance.domain_id, >>> vm_instance.account_id, >>> > vm_instance.user_id, vm_instance.service_offering_id, >>> > vm_instance.reservation_id, vm_instance.hypervisor_type, >>
[GitHub] cloudstack issue #1743: CLOUDSTACK-8326: Always fill UDP checksums in DHCP r...
Github user wido commented on the issue: https://github.com/apache/cloudstack/pull/1743 Is this one ready to merge? It fixes a rather nasty bug in the VR which causes big problems when no DHCP offer is received by the Instance(s). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #1713: CLOUDSTACK-9552: Allow egress TCP/53 implicitly in B...
Github user wido commented on the issue: https://github.com/apache/cloudstack/pull/1713 LGTMs on this one? Small fix, but would be good for a LTS. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Long delay in listDomains API response
Hi Wei, In addition, can you also advice any specific tweak or tuning of the CloudStack management server and MySQL database's configurations to improve the time it takes to perform the listDomains API query and result? We have tried CloudStack and MySQL tunings as per below documentations and it doesn't seem to improve the listDomains API query time. http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.8/tuning.html http://events.linuxfoundation.org/sites/events/files/slides/CloudSTack-Tuning-Collab-2015.pdf https://www.percona.com/blog/2014/01/28/10-mysql-performance-tuning-settings-after-installation/ https://www.percona.com/blog/2012/01/06/mysql-high-number-connections-per-secon/ Any advice on this is greatly appreciated. Looking forward to your reply, thank you. Cheers. On Mon, Nov 7, 2016 at 2:35 AM, Cloud List wrote: > Hi Wei, > > Thanks for your reply. Is it possible for us to remove the resource > limitation feature -- any global settings that we can set to turn this off? > If not, would you be able to let us know where on the source code this is > being set, so that we can try changing it and recompile ACS to turn off the > feature? > > If it's possible, how would be the impact to CloudStack operations in > general? May I know what was the original purpose the limitation is being > put in the first place? > > Thanks for the workaround recommendation -- will forward to our > application vendor to see if the app can be fine-tuned following your > recommendation. > > Looking forward to your reply, thank you. > > Cheers. > > On Mon, Nov 7, 2016 at 12:42 AM, Wei ZHOU wrote: > >> I cannot say it is unnecessary or not for now. >> There are indeed many db connections when get resource limtiation/count >> of domains. >> >> A workaound is (1) listDomainChildren gets id of domain and sub-domains, >> then (2) listDomains with specified id. >> >> -Wei >> >> >> >> 2016-11-06 17:23 GMT+01:00 Cloud List : >> >>> Hi Sergey, >>> >>> Thanks for your reply. The grep result is a bit long, total 64K lines, >>> so I have uploaded it here: >>> >>> https://dl.dropboxusercontent.com/u/24788636/ctx-3f3ff343.txt >>> >>> It seems there are a lot of unnecessary creating and closing of >>> connections to the DB, causing the delay? >>> >>> Looking forward to your reply, thank you. >>> >>> Cheers. >>> >>> >>> On Sun, Nov 6, 2016 at 11:30 PM, Sergey Levitskiy < >>> sergey.levits...@autodesk.com> wrote: >>> Can you grep for this line ‘ctx-3f3ff343’ in your log and post it here? On 11/5/16, 11:01 PM, "Cloud List" wrote: Dear all, After researching further, I found out that our problem is exactly the one reported on this JIRA bug: https://issues.apache.org/jira/browse/CLOUDSTACK-9405 I am cc-ing dsclose and Wei Zhou on this email. Anyone know if there's any resolution to the above problem/bug? On the bug report, Wei Zhou mentioned that he added domain level resource limitation, may I know if this limitation is configurable? I can't find it on the Global Settings. Looking forward to your reply, thank you. Cheers. On Sun, Nov 6, 2016 at 10:30 AM, Cloud List wrote: > Thanks Sergey for your assistance. > > I have enabled the TRACE logs as requested. After the listDomains API > call, I see a lot of DB connections being created to the database, > preparing SELECT statements and then immediately being closed thereafter. > This happens many many times in the loop in the course of 10-15 seconds. > For example: > > Creating a DB connection with no txn > Preparing: SELECT > Closing: com.mysql.jdbc.PreparedStatement > Closing DB connection > > The above continues in a loop for 10-15 seconds, sometimes more. Is it > normal? Could it be something wrong during the database schema upgrade when > I upgraded from 4.2 to 4.8? > > Excerpts of the TRACE logs are below. Any advice on how I can resolve the > problem is highly appreciated. > > === > 2016-11-06 09:36:04,893 DEBUG [c.c.a.ApiServlet] > (http-8080-10:ctx-3f3ff343) (logid:0429eaf0) ===START=== X.X.X.X -- GET > apikey=&command=listDomains&response=json&signature= > 2016-11-06 09:36:04,893 TRACE [c.c.u.d.T.Transaction] > (http-8080-10:ctx-3f3ff343 ctx-e486fd2b) (logid:0429eaf0) Using current > transaction: http-8080-10 : http-8080-10, > 2016-11-06 09:36:04,893 TRACE [c.c.u.d.T.Transaction] > (http-8080-10:ctx-3f3ff343 ctx-e486fd2b) (logid:0429eaf0) Using current > transaction: http-8080-10 : http-8080-10, > 2016-11-06 09:36:04,894 TRACE
Re: Long delay in listDomains API response
What essentially happens here that for each domain, you have access with your key, ACS executes 12 queries to get domain limits. Even if such query for each resource limit takes 0.03 sec it translates into 0.4 sec for each domain. If you have more than few domains, you will see rather lengthy total execution of listDomains in spite of some parallel execution of such queries. This is apparently far from optimal but I don’t see if it can be improved by optimizing MySQL alone. I looked through the ACS code and there is a way to improve this big time since all resource limits are already in domain_view . We will try to fix it soon for 4.9-4.10 versions. It usually takes at least 2-3 weeks before the code will be reviewed and committed. Until then I suggest to increase any client timeouts accordingly. Thanks, Sergey On 11/6/16, 5:52 PM, "Cloud List" wrote: Hi Wei, In addition, can you also advice any specific tweak or tuning of the CloudStack management server and MySQL database's configurations to improve the time it takes to perform the listDomains API query and result? We have tried CloudStack and MySQL tunings as per below documentations and it doesn't seem to improve the listDomains API query time. http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.8/tuning.html http://events.linuxfoundation.org/sites/events/files/slides/CloudSTack-Tuning-Collab-2015.pdf https://www.percona.com/blog/2014/01/28/10-mysql-performance-tuning-settings-after-installation/ https://www.percona.com/blog/2012/01/06/mysql-high-number-connections-per-secon/ Any advice on this is greatly appreciated. Looking forward to your reply, thank you. Cheers. On Mon, Nov 7, 2016 at 2:35 AM, Cloud List wrote: > Hi Wei, > > Thanks for your reply. Is it possible for us to remove the resource > limitation feature -- any global settings that we can set to turn this off? > If not, would you be able to let us know where on the source code this is > being set, so that we can try changing it and recompile ACS to turn off the > feature? > > If it's possible, how would be the impact to CloudStack operations in > general? May I know what was the original purpose the limitation is being > put in the first place? > > Thanks for the workaround recommendation -- will forward to our > application vendor to see if the app can be fine-tuned following your > recommendation. > > Looking forward to your reply, thank you. > > Cheers. > > On Mon, Nov 7, 2016 at 12:42 AM, Wei ZHOU wrote: > >> I cannot say it is unnecessary or not for now. >> There are indeed many db connections when get resource limtiation/count >> of domains. >> >> A workaound is (1) listDomainChildren gets id of domain and sub-domains, >> then (2) listDomains with specified id. >> >> -Wei >> >> >> >> 2016-11-06 17:23 GMT+01:00 Cloud List : >> >>> Hi Sergey, >>> >>> Thanks for your reply. The grep result is a bit long, total 64K lines, >>> so I have uploaded it here: >>> >>> https://dl.dropboxusercontent.com/u/24788636/ctx-3f3ff343.txt >>> >>> It seems there are a lot of unnecessary creating and closing of >>> connections to the DB, causing the delay? >>> >>> Looking forward to your reply, thank you. >>> >>> Cheers. >>> >>> >>> On Sun, Nov 6, 2016 at 11:30 PM, Sergey Levitskiy < >>> sergey.levits...@autodesk.com> wrote: >>> Can you grep for this line ‘ctx-3f3ff343’ in your log and post it here? On 11/5/16, 11:01 PM, "Cloud List" wrote: Dear all, After researching further, I found out that our problem is exactly the one reported on this JIRA bug: https://issues.apache.org/jira/browse/CLOUDSTACK-9405 I am cc-ing dsclose and Wei Zhou on this email. Anyone know if there's any resolution to the above problem/bug? On the bug report, Wei Zhou mentioned that he added domain level resource limitation, may I know if this limitation is configurable? I can't find it on the Global Settings. Looking forward to your reply, thank you. Cheers. On Sun, Nov 6, 2016 at 10:30 AM, Cloud List wrote: > Thanks Sergey for your assistance. > > I have enabled the TRACE logs as requested. After the listDomains API > call, I see a lot of DB connections being created to the database, > preparing SELECT statements and then immediately being closed thereafter. >
