I posted here: http://pastebin.com/uJ6jR73D the logs, maybe someone else have seen this behavior before. Just to remind the issue, CS 4.1, XenServer 6.1, when the only existing host is rebooted without being in maintenance mode (and stays down for a while), all the management IPs get used (in this case there were 171 management IPs, all show as in use now).
Thanks, Daniel On Wed, Sep 25, 2013 at 8:56 PM, Musayev, Ilya <[email protected]> wrote: > Dan, > > Ping me on IRC, nick serverchief > > Regards > ilya > > > > -----Original Message----- > > From: Daniel Hertanu [mailto:[email protected]] > > Sent: Wednesday, September 25, 2013 6:44 AM > > To: [email protected] > > Subject: Re: Management IP address pool exhausted > > > > Hi > > > > So I re-installed CS, reboot Xen host without entering it first in > Maintenace > > Mode, then got the management IP pool fully used. I followed again your > > indications but to no avail. > > In the meantime SSVM has reached number 112 (s-112-VM). > > > > I would really appreciate any other idea. > > > > Thanks, > > > > Daniel > > > > > > On Wed, Sep 25, 2013 at 9:36 AM, Daniel Hertanu <[email protected]> > > wrote: > > > > > Hello Ilya > > > > > > I submit an issue describing the bug: > > > https://issues.apache.org/jira/browse/CLOUDSTACK-4735 (I picked CS > > > version 4.1.0 as 4.1.1 is reported as non existent, but I've seen this > > > behavior before since version 3). > > > > > > I followed your indications but it didn't work. After changing those > > > records in the database, CS would still give the same error. I revert > > > back to the initial db content, but this time nothing came back > > > (initially the SSVM was still running, it was CPVM that didn't start). > > > After this step I tried some tricks on my own and I ended up with a > botched > > environment. > > > So I'll do a fresh install, reboot the host and try your suggestions > > > again. > > > > > > Thanks, > > > > > > Daniel > > > > > > > > > On Tue, Sep 24, 2013 at 10:46 PM, Musayev, Ilya > > <[email protected]>wrote: > > > > > >> I'm not certain if this is hypervisor specific - I've seen the same > > >> issue for vSphere - but I though it was me, I think we may need to > > >> revise the system vm ip space decommission process. > > >> > > >> Shanker and/or Daniel, > > >> > > >> Do you mind creating a ticket in issues.apache.org and write out ways > > >> to reproduce this issue. > > >> > > >> --- > > >> > > >> *NOTE* this is purely from memory and some notes I had - proceed at > > >> your own risk.... > > >> > > >> > > >> > > >> In the meantime, lets clean up what you have. > > >> > > >> The goal is to release the IPs that are still marked as reserved and > > >> allocated. > > >> > > >> I'd recommend installing mysql workbench > > >> > > >> There are few tables we need to review: > > >> 1) Stop cloudstack management server > > >> 2) Backup mysql db first. > > >> > > >> Note the id of management network referenced in cloud.networks > > >> Select id from cloud.networks where traffic_type="Management" > > >> In my case, Management network has ID of 205 > > >> SELECT * FROM cloud.nics where network_id=205; Alter to > > >> Deallocating the ones that are marked as Reserved. > > >> > > >> Do the same for cloud.user_ip_address SELECT * FROM > > >> cloud.user_ip_address where source_network_id=205; Alter from > > >> Allocated to Free > > >> > > >> Stop any systemvms if running, and alter the state from Running to > > >> Stopped or Expunging in cloud.vm_instance table. This will recreate > > >> the systemvm from scratch. > > >> > > >> I think this should have you covered, if not, revert back the mysql > db. > > >> Do mysql dump and grep for IPs that should have been released. > > >> > > >> Let me know, > > >> > > >> Regards > > >> ilya > > >> > > >> > > >> > > >> > -----Original Message----- > > >> > From: Shanker Balan [mailto:[email protected]] > > >> > Sent: Tuesday, September 24, 2013 9:44 AM > > >> > To: <[email protected]> > > >> > Subject: Re: Management IP address pool exhausted > > >> > > > >> > > > >> > On 24-Sep-2013, at 6:37 PM, Daniel Hertanu <[email protected]> > > wrote: > > >> > > > >> > > Hello Ilya > > >> > > > > >> > > I repeated the issue by rebooting the Xen host without having it > > >> > > in maintenance mode in CS. At this moment I have the SSVM up and > > >> > > running and a virtual router (not related to the management IPs, I > > know). > > >> > > Console Proxy VM can't be started because: > > >> > > > > >> > > 2013-09-24 14:56:24,410 INFO > > >> > > [cloud.vm.VirtualMachineManagerImpl] > > >> > > (Job-Executor-22:job-72) Insufficient capacity > > >> > > com.cloud.exception.InsufficientAddressCapacityException: Unable > > >> > > to get a management ip addressScope=interface com.cloud.dc.Pod; > > >> > > id=1 > > >> > > > > >> > > > >> > > > >> > > > >> > Hi Ilya, > > >> > > > >> > I have run into the same issue too. It is easily reproducible by > > >> > having > > >> only one > > >> > host in the Zone and rebooting it without enabling maintenance mode > > >> > on > > >> it. > > >> > > > >> > This issue exists for both XenServer and KVM host when I tried. > > >> > > > >> > > > >> > -- > > >> > @shankerbalan > > >> > > > >> > M: +91 98860 60539 | O: +91 (80) 67935867 > > >> > [email protected] | www.shapeblue.com | > > >> > Twitter:@shapeblue ShapeBlue Services India LLP, 22nd floor, Unit > > >> > 2201A, World Trade Centre, Bangalore - 560 055 > > >> > > > >> > This email and any attachments to it may be confidential and are > > >> intended > > >> > solely for the use of the individual to whom it is addressed. Any > > >> > views > > >> or > > >> > opinions expressed are solely those of the author and do not > > >> > necessarily represent those of Shape Blue Ltd or related companies. > > >> > If you are not > > >> the > > >> > intended recipient of this email, you must neither take any action > > >> > based upon its contents, nor copy or show it to anyone. Please > > >> > contact the > > >> sender if > > >> > you believe you have received this email in error. Shape Blue Ltd > > >> > is a company incorporated in England & Wales. ShapeBlue Services > > >> > India LLP > > >> is a > > >> > company incorporated in India and is operated under license from > > >> > Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company > > >> > incorporated > > >> in > > >> > Brasil and is operated under license from Shape Blue Ltd. ShapeBlue > > >> > is a registered trademark. > > >> > > >> > > >> > > > > >
