Re: [Openstack-operators] Lets talk capacity monitoring

One good topic to try to pin down at the Ops meet up would be how we could do the flavour/aggregate/project/hypervisor mappings. We’ve got a local patch for some function but it was not possible to get the right way to do it agreed (https://blueprints.launchpad.net/nova/+spec/multi-tenancy-isola

Re: [Openstack-operators] Way to check compute <-> rabbitmq connectivity

We’ve had a lot of issues with Icehouse related to rabbitMQ. Basically the change from openstack.rpc to oslo.messaging broke things. These things are now fixed in oslo.messaging version 1.5.1, there is still an issue with heartbeats and that patch is making it’s way through review process now.

Re: [Openstack-operators] Way to check compute <-> rabbitmq connectivity

If you are using ha queues, use a version of rabbitmq > 3.3.0. There was a change in that version where consumption on queues was automatically enabled when a master election for a queue happened. Previous versions only informed clients that they had to reconsume on a queue. It was the clients resp

Re: [Openstack-operators] Lets talk capacity monitoring

I've found histograms to be pretty useful in figuring out patterns during sizable time deltas... and anomaly detection there can highlight stuff you might want to check out ( ie raise the alert condition on that device ). example of a histogram i did many many moons ago to track disk sizes from ou

Re: [Openstack-operators] Way to check compute <-> rabbitmq connectivity

Here is the bug I’ve been tracking related to this for a while. I haven’t really kept up to speed with it, so I don’t know the current status. https://bugs.launchpad.net/nova/+bug/856764 From: Kris Lindgren mailto:klindg...@godaddy.com>> Date: Thursday, January 15, 2015 at 12:10 PM To: Gustavo

Re: [Openstack-operators] Small openstack (part 2), distributed glance

We do not using centralized storages (all instances running with local drives). And I just can't express my happiness about this. Every time monitoring send me '** PROBLEM ALERT bla-bla-bla', I know it not a big deal. Just one server. I do not want to turn gray prematurely. Just light glance o

Re: [Openstack-operators] Lets talk capacity monitoring

On 01/15/2015 06:43 PM, Jesse Keating wrote: We have a need to better manage the various openstack capacities across our numerous clouds. We want to be able to detect when capacity of one system or another is approaching the point where it would be a good idea to arrange to increase that capaci

Re: [Openstack-operators] Small openstack (part 2), distributed glance

That specific bottleneck can be solved by running glance on ceph, and running ephemeral instances also on ceph. Snapshots are a quick backend operation then. But you've made your installation on a house of cards. On Thursday, January 15, 2015, George Shuklin wrote: > Hello everyone. > > One more

[Openstack-operators] Small openstack (part 2), distributed glance

Hello everyone. One more thing in the light of small openstack. I really dislike tripple network load caused by current glance snapshot operations. When compute do snapshot, it playing with files locally, than it sends them to glance-api, and (if glance API is linked to swift), glance sends t

Re: [Openstack-operators] The state of nova-network to neutron migration

On Thu, Jan 15, 2015 at 2:19 PM, Kris G. Lindgren wrote: > Is the fact that neutron security groups don’t provide the same level > of isolation as nova security groups on your guys radar? > > Specifically talking about: > https://bugs.launchpad.net/neutron/+bug/1274034 > > That bug is actually

Re: [Openstack-operators] The state of nova-network to neutron migration

On 01/16/2015 09:19 AM, Kris G. Lindgren wrote: > Is the fact that neutron security groups don’t provide the same level of > isolation as nova security groups on your guys radar? > > Specifically talking about: https://bugs.launchpad.net/neutron/+bug/1274034 > > I am sure their are a few other

Re: [Openstack-operators] [ha-guide] HA Guide update next steps

Matt, can u please send the link for the wiki page? On Thu, Jan 15, 2015 at 7:17 AM, Matt Griffin wrote: > Just a reminder that we're going to meet today (and every Thursday) from > 3:00-3:30pm US Central. > Like last time, let's chat in #openstack-haguide on freenode. > > A bit later today (be

Re: [Openstack-operators] The state of nova-network to neutron migration

Is the fact that neutron security groups don’t provide the same level of isolation as nova security groups on your guys radar? Specifically talking about: https://bugs.launchpad.net/neutron/+bug/1274034 I am sure their are a few other thigns that nova is doing that neutron is currently not. __

Re: [Openstack-operators] The state of nova-network to neutron migration

Will do. On Thu, Jan 15, 2015 at 12:08 PM, Anita Kuno wrote: > On 01/14/2015 01:06 PM, matt wrote: > > Hey Mike! > > > > Thanks for this info. Super helpful to me at least. I am very > interested > > in hearing more about nova-network to neutron migrations. > > > > -Matt > > > Hello Matt: > >

Re: [Openstack-operators] The state of nova-network to neutron migration

On 01/14/2015 01:06 PM, matt wrote: > Hey Mike! > > Thanks for this info. Super helpful to me at least. I am very interested > in hearing more about nova-network to neutron migrations. > > -Matt > Hello Matt: Please start attending the weekly meetings: https://wiki.openstack.org/wiki/Meetings

Re: [Openstack-operators] Fwd: HAPROXY 504 errors in HA conf

False alarm, after more tests the issue persisted, so I switched to backup mode in the other haproxy nodes and now everything works as expected. Thanks Em 15/01/2015 12:13, "Pedro Sousa" escreveu: > Hi all, > > the culprit was haproxy, I had "option httpchk" when I disabled this > stopped having

Re: [Openstack-operators] Way to check compute <-> rabbitmq connectivity

During the Atlanta ops meeting this topic came up and I specifically mentioned about adding a "no-op" or healthcheck ping to the rabbitmq stuff to both nova & neutron. The dev's in the room looked at me like I was crazy, but it was so that we could exactly catch issues as you described. I am a

Re: [Openstack-operators] Way to check compute <-> rabbitmq connectivity

Just to add one more background scenario, we also had similar problems trying to load balance rabbitmq via F5 Big IP LTM. For that reason we don't use it now. Our installation is a single rabbitmq instance and no intermediaries (albeit network switches). We use Folsom and Icehouse, the problem bein

Re: [Openstack-operators] Cinder api enpoint not found error while attach volume to instance

On 01/15/2015 06:22 AM, Geo Varghese wrote: Hi Jay/Abel, Thanks for your help. Just fixed issue by changing following line in nova.conf cinder_catalog_info=volumev2:cinderv2:publicURL to cinder_catalog_info=volume:cinder:publicURL Now attachment successfuly done. Do guys know how this fixe

Re: [Openstack-operators] Lets talk capacity monitoring

I know we've been working on that on our commercial product side at big switch with an analyzer... the issue i think you are going to run into is getting insight into network upstream info from your top of racks and spine switches. Setting up a uniform access to ovs stats in the API or in an exter

Re: [Openstack-operators] Lets talk capacity monitoring

On 2015-01-15 11:43 AM, Jesse Keating wrote: We have a need to better manage the various openstack capacities across our numerous clouds. We want to be able to detect when capacity of one system or another is approaching the point where it would be a good idea to arrange to increase that capacity

Re: [Openstack-operators] metadata-api 500 errors

Hmmm, I did do some playing with security groups, but even the last network (network5) that I created was unchanged from "default" - maybe it has something to do with that, I'm testing a brand new tenant/project now with a similar setup, except I'm setting up all the networking / networks first thi

Re: [Openstack-operators] metadata-api 500 errors

Did you change the security groups? I will try to compare the iptables configuration for each name space Edgar From: Alex Leonhardt mailto:aleonhardt...@gmail.com>> Date: Thursday, January 15, 2015 at 9:03 AM To: Edgar Magana mailto:edgar.mag...@workday.com>>, openstack-operators mailto:opensta

Re: [Openstack-operators] metadata-api 500 errors

Hi Edgar, that's the crazy thing - so all the gre tunnels are up, I can see them in openvswitch and also can see that there are some openflow rules applied. I've craeted VMs on every hypervisor (including the controller, as it's a test install) on network1 (192.168.1.0), every VM (and that is stil

Re: [Openstack-operators] metadata-api 500 errors

Alex, Did you follow the networking recommendations: http://docs.openstack.org/openstack-ops/content/network_troubleshooting.html It will ell you if you write your own topology and complete a packet trace to find out the issue. Make sure all tunnels are established between your three nodes. Tha

Re: [Openstack-operators] I WANT TO SETUP AND CONFIGURE HAVANA ON CENTOS 6.5

It seems that you got some answers already. Basically, whatever "SECURE" that you want to generate. I will provide a future post about how to secure your Cloud, which mean a best practices way to generate these secure code and keep them away from everybody, even your own folks! :-) Let's empo

Re: [Openstack-operators] Way to check compute <-> rabbitmq connectivity

We did have an issue using celery on an internal application that we wrote - but I believe it was fixed after much failover testing and code changes. We also use logstash via rabbitmq and haven't noticed any issues there either. So this seems to be just openstack/oslo related. We have tried a

[Openstack-operators] Lets talk capacity monitoring

We have a need to better manage the various openstack capacities across our numerous clouds. We want to be able to detect when capacity of one system or another is approaching the point where it would be a good idea to arrange to increase that capacity. Be it volume space, VCPU capability, obje

Re: [Openstack-operators] Openstack-Keystone error

On 1/15/15 2:17 AM, Anwar Durrani wrote: ​I did following steps earlier : These steps don't mention doing the keystone-manage db_sync action. When you install keystone itself and configure it to connect to a sql service, and you have created a keystone database within the sql service, the ne

Re: [Openstack-operators] I WANT TO SETUP AND CONFIGURE HAVANA ON CENTOS 6.5

On 1/15/15 1:48 AM, Anwar Durrani wrote: Thanks Edgar for help, i have question in following section : * Edit /etc/keystone/keystone.conf: vim /etc/keystone/keystone.conf [DEFAULT] admin_token=ADMIN log_dir=/var/log/keystone [database] connection = mysql://keyst

Re: [Openstack-operators] New services disable reason

:) did that too just now. ta! On Thu Jan 15 2015 at 16:28:36 Michael Dorman wrote: > +1 as well, for the same reasons. I also added by +1 to the review. > Thanks! > > > From: Alex Leonhardt > Date: Thursday, January 15, 2015 at 1:02 AM > To: Belmiro Moreira , OpenStack > Operators > Subje

Re: [Openstack-operators] Way to check compute <-> rabbitmq connectivity

Hi Kris, Our experience is pretty much the same on anything that is using rabbitmq > - not just nova-compute. > Just to clarify: have you experienced this outside of OpenStack (or Oslo)? We've seen similar issues with rabbitmq and OpenStack. We used to run rabbit through haproxy and tried a myr

Re: [Openstack-operators] metadata-api 500 errors

In case it helps - attached a screenshot of the topology from openstack dashboard. alex On Thu Jan 15 2015 at 14:36:24 Alex Leonhardt wrote: > hi, > > i've got a test openstack install with 3 nodes, using gre tunneling -- > > initially it all worked fine, but, after creating > 2 networks, VMs

Re: [Openstack-operators] Way to check compute <-> rabbitmq connectivity

On 1/15/15 7:34 AM, Gustavo Randich wrote: Hi, I'm experiencing some issues with nova-compute services not responding to rabbitmq messages, despite the service reporting OK state via periodic tasks. Apparently the TCP connection is open but in a stale or unresponsive state. This happens sporadic

Re: [Openstack-operators] New services disable reason

+1 as well, for the same reasons. I also added by +1 to the review. Thanks! From: Alex Leonhardt mailto:aleonhardt...@gmail.com>> Date: Thursday, January 15, 2015 at 1:02 AM To: Belmiro Moreira mailto:moreira.belmiro.email.li...@gmail.com>>, OpenStack Operators mailto:openstack-operators@lis

Re: [Openstack-operators] Way to check compute <-> rabbitmq connectivity

+1 on this. In general rabbitmq connectivity/failover is pretty terrible. Services look to be connected to rabbitmq but in reality they aren't, monitoring on the server to see if it has an established connection to rabbitmq isn't enough. Our experience is pretty much the same on anything that

[Openstack-operators] metadata-api 500 errors

hi, i've got a test openstack install with 3 nodes, using gre tunneling -- initially it all worked fine, but, after creating > 2 networks, VMs in networks 3,4,5 do not seem to get the metadata due to it erroring with 500 errors. whilst this is happening, VMs in networks 1 and 2 are still working

[Openstack-operators] Way to check compute <-> rabbitmq connectivity

Hi, I'm experiencing some issues with nova-compute services not responding to rabbitmq messages, despite the service reporting OK state via periodic tasks. Apparently the TCP connection is open but in a stale or unresponsive state. This happens sporadically when there is some not yet understood ne

Re: [Openstack-operators] [ha-guide] HA Guide update next steps

Just a reminder that we're going to meet today (and every Thursday) from 3:00-3:30pm US Central. Like last time, let's chat in #openstack-haguide on freenode. A bit later today (before our meeting), I'll review the wiki so we can charge ahead as soon as possible on updating areas. Best, Matt --

Re: [Openstack-operators] Fwd: HAPROXY 504 errors in HA conf

Hi all, the culprit was haproxy, I had "option httpchk" when I disabled this stopped having timeouts rebooting the servers. Thank you all. On Wed, Jan 14, 2015 at 5:29 PM, John Dewey wrote: > I would verify that the VIP failover is occurring. > > Your master should have the IP address. If y

Re: [Openstack-operators] Cinder api enpoint not found error while attach volume to instance

Hi Jay/Abel, Thanks for your help. Just fixed issue by changing following line in nova.conf cinder_catalog_info=volumev2:cinderv2:publicURL to cinder_catalog_info=volume:cinder:publicURL Now attachment successfuly done. Do guys know how this fixed the issue? On Thu, Jan 15, 2015 at 12:01 P

Re: [Openstack-operators] I WANT TO SETUP AND CONFIGURE HAVANA ON CENTOS 6.5

If you are doing setup for self do it on ubuntu with juno release http://docs.openstack.org/havana/install-guide/install/yum/content/ On 15-Jan-2015 2:45 PM, "Anwar Durrani" wrote: > Hello everyone, > > I want to setup Havana, Do anyone has installation guide for the same ? > > Thanks > > --

Re: [Openstack-operators] Openstack-Keystone error

this is probably the issue then: 2015-01-15 01:08:34.128 50243 TRACE keystone.common.wsgi ProgrammingError: (ProgrammingError) (1146, "Table 'keystone.token' doesn't exist") 'SELECT token.idAS token_id, token.expires AS token_expires, token.extra AS token_extra, token.valid AS token_valid, token.u

Re: [Openstack-operators] I WANT TO SETUP AND CONFIGURE HAVANA ON CENTOS 6.5

means generate something "secure" :) On Thu Jan 15 2015 at 09:50:34 Anwar Durrani wrote: > Thanks Edgar for help, i have question in following section : > > >- > >Edit /etc/keystone/keystone.conf: > >vim /etc/keystone/keystone.conf > >[DEFAULT] >admin_token=ADMIN >log_dir

Re: [Openstack-operators] I WANT TO SETUP AND CONFIGURE HAVANA ON CENTOS 6.5

Thanks Edgar for help, i have question in following section : - Edit /etc/keystone/keystone.conf: vim /etc/keystone/keystone.conf [DEFAULT] admin_token=ADMIN log_dir=/var/log/keystone [database] connection = mysql://keystone:password@controller/keystone admin_token=A

Re: [Openstack-operators] I WANT TO SETUP AND CONFIGURE HAVANA ON CENTOS 6.5

Go for Icehouse: https://github.com/emagana/OpenStack-Icehouse-Install-Guide/blob/master/OpenStack-Icehouse-Installation.rst Edgar From: Anwar Durrani mailto:durrani.an...@gmail.com>> Date: Thursday, January 15, 2015 at 1:12 AM To: openstack-operators mailto:openstack-operators@lists.openstack.o

[Openstack-operators] I WANT TO SETUP AND CONFIGURE HAVANA ON CENTOS 6.5

Hello everyone, I want to setup Havana, Do anyone has installation guide for the same ? Thanks -- Thanks & regards, Anwar M. Durrani +91-8605010721 ___ OpenStack-operators mailing list OpenStack-op

Re: [Openstack-operators] Openstack-Keystone error

Hi Alex, below is error in log file 2015-01-15 01:08:34.128 50243 ERROR keystone.common.wsgi [-] (ProgrammingError) (1146, "Table 'keystone.token' doesn't exist") 'SELECT token.id AS token_id, token.expires AS token_expires, token.extra AS token_extra, token.valid AS token_valid, token.user_id AS

Re: [Openstack-operators] Openstack-Keystone error

I don't think anyone should try to install OS manually :) .. But check the keystone logs for what caused the 500? Maybe the admin tenant/project already exists? On Thu, 15 Jan 2015 08:29 Anwar Durrani wrote: > Hi everyone, > > i am getting below error while running below command > > [root@localh

[Openstack-operators] Openstack-Keystone error

Hi everyone, i am getting below error while running below command [root@localhost ~]# keystone tenant-create --name admin --description "Admin Tenant" An unexpected error prevented the server from fulfilling your request. (HTTP 500) [root@localhost ~]# Prior to run this command i have done follo

Re: [Openstack-operators] New services disable reason

Our install is still quite small and we take the risk of hitting that compute node whilst the service is starting, but we haven't actually encountered that yet (probably a user base size issue) ... IMHO, having a reason why stuff is disabled avoids hours of confusion and trying to find the person