Hello Ilya
I repeated the issue by rebooting the Xen host without having it in
maintenance mode in CS. At this moment I have the SSVM up and running and a
virtual router (not related to the management IPs, I know). Console Proxy
VM can't be started because:
2013-09-24 14:56:24,410 INFO [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-22:job-72) Insufficient capacity
com.cloud.exception.InsufficientAddressCapacityException: Unable to get a
management ip addressScope=interface com.cloud.dc.Pod; id=1
The queries you were asking about:
mysql> SELECT * FROM cloud.nics where state="Reserved" and
(vm_type="ConsoleProxy" or vm_type="SecondaryStorageVM");
+----+--------------------------------------+-------------+-------------------+----------------+---------------+---------------+---------+-----------------+------------+--------+----------+----------+---------------------------+--------------------------------------+-----------+---------------------+----------------+-------------+-------------+--------------------+---------------------+---------+-------------+----------+
| id | uuid | instance_id | mac_address
| ip4_address | netmask | gateway | ip_type |
broadcast_uri | network_id | mode | state | strategy | reserver_name
| reservation_id | device_id |
update_time | isolation_uri | ip6_address | default_nic | vm_type
| created | removed | ip6_gateway | ip6_cidr |
+----+--------------------------------------+-------------+-------------------+----------------+---------------+---------------+---------+-----------------+------------+--------+----------+----------+---------------------------+--------------------------------------+-----------+---------------------+----------------+-------------+-------------+--------------------+---------------------+---------+-------------+----------+
| 1 | c3c2d45d-94f9-4172-9eee-7829106380de | 1 |
06:e9:ea:00:00:0d | 195.238.173.10 | 255.255.255.0 | 195.238.173.1 | Ip4
| vlan://untagged | 204 | Dhcp | Reserved | Start |
DirectPodBasedNetworkGuru | 6591a371-15e3-4396-b14e-d23de671bdaa |
2 | 2013-09-24 14:54:33 | ec2://untagged | NULL | 1 |
SecondaryStorageVm | 2013-09-23 12:06:25 | NULL | NULL | NULL
|
| 2 | d2c11105-22fd-4c94-b54b-4039347e3ec2 | 1 |
0e:00:a9:fe:02:ea | 169.254.2.234 | 255.255.0.0 | 169.254.0.1 | Ip4
| NULL | 202 | Static | Reserved | Start |
ControlNetworkGuru | 6591a371-15e3-4396-b14e-d23de671bdaa |
0 | 2013-09-24 14:54:33 | NULL | NULL | 0 |
SecondaryStorageVm | 2013-09-23 12:06:25 | NULL | NULL | NULL
|
| 3 | 8c1c909c-82fb-4786-b1e6-efc9e3098d51 | 1 |
06:17:8c:00:00:04 | 192.168.0.11 | 255.255.255.0 | 192.168.0.1 | Ip4
| NULL | 201 | Static | Reserved | Start |
PodBasedNetworkGuru | 6591a371-15e3-4396-b14e-d23de671bdaa |
1 | 2013-09-24 14:54:34 | NULL | NULL | 0 |
SecondaryStorageVm | 2013-09-23 12:06:25 | NULL | NULL | NULL
|
| 4 | 0f440319-18b5-4129-9ff9-37088f794107 | 1 |
06:61:fc:00:00:08 | 192.168.0.15 | 255.255.255.0 | 192.168.0.1 | Ip4
| NULL | 203 | Static | Reserved | Start |
StorageNetworkGuru | 6591a371-15e3-4396-b14e-d23de671bdaa |
3 | 2013-09-24 14:54:34 | NULL | NULL | 0 |
SecondaryStorageVm | 2013-09-23 12:06:25 | NULL | NULL | NULL
|
+----+--------------------------------------+-------------+-------------------+----------------+---------------+---------------+---------+-----------------+------------+--------+----------+----------+---------------------------+--------------------------------------+-----------+---------------------+----------------+-------------+-------------+--------------------+---------------------+---------+-------------+----------+
mysql> SELECT * FROM cloud.vm_instance where (name like "s-%" or name like
"v-%");
+----+--------+--------------------------------------+---------------+---------+----------------+-------------+---------------------+--------------------+--------+----------------+---------+--------------+----------+-------------------+----------------------------------------------+------------+---------------+--------------+---------------------+---------------------+---------+--------------------+--------------------+------------+-----------+---------------------+--------------------------------------+-----------------+------------------+------+------+-------+-------+-----------+--------------+---------------+
| id | name | uuid | instance_name |
state | vm_template_id | guest_os_id | private_mac_address |
private_ip_address | pod_id | data_center_id | host_id | last_host_id |
proxy_id | proxy_assign_time | vnc_password
| ha_enabled | limit_cpu_use | update_count | update_time | created
| removed | type | vm_type |
account_id | domain_id | service_offering_id | reservation_id
| hypervisor_type | disk_offering_id | cpu | ram | owner | speed
| host_name | display_name | desired_state |
+----+--------+--------------------------------------+---------------+---------+----------------+-------------+---------------------+--------------------+--------+----------------+---------+--------------+----------+-------------------+----------------------------------------------+------------+---------------+--------------+---------------------+---------------------+---------+--------------------+--------------------+------------+-----------+---------------------+--------------------------------------+-----------------+------------------+------+------+-------+-------+-----------+--------------+---------------+
| 1 | s-1-VM | 2cb5f8db-91eb-4f16-a6fa-4bfed79c51b6 | s-1-VM |
Running | 1 | 133 | 06:17:8c:00:00:04 | 192.168.0.11
| 1 | 1 | 1 | 1 | NULL | NULL
| gGI1kYniavKLLwWWHO1/8oL62JDMUdEbsSpj48XfVCo= | 0 |
0 | 13 | 2013-09-24 13:55:14 | 2013-09-23 12:06:25 |
NULL | SecondaryStorageVm | SecondaryStorageVm | 1 | 1
| 9 | 6591a371-15e3-4396-b14e-d23de671bdaa | XenServer
| NULL | NULL | NULL | NULL | NULL | NULL | NULL
| NULL |
| 2 | v-2-VM | c1975f1c-300f-4a6d-879f-2b70b4842a8f | v-2-VM |
Stopped | 1 | 133 | 06:c4:62:00:00:03 | 192.168.0.10
| 1 | 1 | NULL | 1 | NULL | NULL
| wRV0PYKhMMKR7ENEvJ212Zi3FP//FV1eGmTfIkrkLX4= | 0 |
0 | 371 | 2013-09-24 14:03:35 | 2013-09-23 12:06:26 |
NULL | ConsoleProxy | ConsoleProxy | 1 | 1
| 10 | 1dea74ec-ee0c-4b6c-bc62-43093572f0f1 | XenServer
| NULL | NULL | NULL | NULL | NULL | NULL | NULL
| NULL |
+----+--------+--------------------------------------+---------------+---------+----------------+-------------+---------------------+--------------------+--------+----------------+---------+--------------+----------+-------------------+----------------------------------------------+------------+---------------+--------------+---------------------+---------------------+---------+--------------------+--------------------+------------+-----------+---------------------+--------------------------------------+-----------------+------------------+------+------+-------+-------+-----------+--------------+---------------+
The expunge interval and delay is setup to 60 seconds.
Thanks,
Daniel
On Tue, Sep 24, 2013 at 12:34 AM, Musayev, Ilya <[email protected]> wrote:
> I suspect the issue with IPs not being released right away.
>
> What do you get for running this query
>
> SELECT * FROM cloud.nics where state="Reserved" and
> (vm_type="ConsoleProxy" or vm_type="SecondaryStorageVM");
>
> SELECT * FROM cloud.vm_instance where (name like "s-%" or name like "v-%");
>
> If its set to expunging, what is the expunge interval set to in global
> settings?
>
>
> > -----Original Message-----
> > From: Daniel Hertanu [mailto:[email protected]]
> > Sent: Monday, September 23, 2013 10:08 AM
> > To: [email protected]
> > Subject: Management IP address pool exhausted
> >
> > Hi
> >
> > I'm running CS 4.1.1 on CentOS 6.4 64 bit with one host Citrix Xenserver
> 6.1.
> >
> > Few days ago I ran into a problem while I was configuring CS. The zone
> was
> > active, Xen host added, SSVM and CPVM up & running when someone from
> > our team rebooted the Xen host for whatever reason. Even though
> > CloudStack was complaining about the missing host in the logs, it
> deleted the
> > old SSVM and tried to deploy another one, it failed, then another one,
> and so
> > on, until the management IP addresses pool was exhausted, moment when
> > CloudStack reported the new error - that there are not enough management
> > IPs. The host came back online later but the situation didn't changed. I
> > lowered the expunge.delay and expunge.interval to 5 minutes, still no
> > change.
> > I wiped out CS installation and I start all over again. I was curious
> about the
> > previous issue, so I manually rebooted the Xen host (without putting it
> in
> > Maintenance mode in CS) and the issue showed up again.
> > While I admit that having one host is not ideal, I have two questions:
> > - why would CS try to deploy any VM when it actually knows there are no
> > valid hosts
> > - how do I make those management IPs available again after CS tried to
> > deploy 10-20 SSVMs and it exhausted the management IPs pool?
> >
> > Thanks,
> >
> > Daniel
>
>
