Re: [Openstack-operators] Properties missing in Nova Scheduler Filter

2016-11-10 Thread Matt Fischer
Mario, If I remember right I had a similar issue with getting image_props when I was doing this to pull in custom properties. Through some trial and error and poking around with pdb I ended up with this: image_props = spec_obj.get('request_spec', {}).\ get('image', {}).get('pr

Re: [Openstack-operators] [puppet] openstack provider errors with openrc and keystone v3

2016-11-11 Thread Matt Fischer
There is a known issue where some providers fail when you have an openrc sourced. I remember it being glance that failed. Bug #1524599 On Nov 11, 2016 4:15 AM, "Justin Cattle" wrote: > There was two problems here! > > The puppet libs in use were coming from the wrong environment - so a > pretty

Re: [Openstack-operators] Properties missing in Nova Scheduler Filter

2016-11-12 Thread Matt Fischer
bility=), > '_obj_force_hosts': None, 'VERSION': u'1.5', '_obj_force_nodes': None, > '_obj_pci_requests': InstancePCIRequests(instance_ > uuid=22313c7f-0338-4bed-9131-900b458347d9,requests=[]), '_obj_retry': > SchedulerRetrie

[Openstack-operators] feedback on pymysql

2016-11-17 Thread Matt Fischer
As a part of our upgrades to Newton we are transitioning our services to use pymysql rather than the deprecated MySQL-Python [1]. I believe pymsql has been the default in devstack and the gate for sometime now and that MySQL-Python is essentially untested and not updated, hence our desire to switch

Re: [Openstack-operators] [nova] Live migration performance tests on 100 compute nodes

2016-12-29 Thread Matt Fischer
On Wed, Dec 28, 2016 at 6:11 AM, Koniszewski, Pawel < pawel.koniszew...@intel.com> wrote: > Hello everyone, > > We made a research to see how live migration performance varies between > different configurations, especially we aimed to test tunneled vs > non-tunneled live migrations. To test live m

Re: [Openstack-operators] RabbitMQ 3.6.x experience?

2017-01-05 Thread Matt Fischer
MIke, I did a bunch of research and experiments on this last fall. We are running Rabbit 3.5.6 on our main cluster and 3.6.5 on our Trove cluster which has significantly less load (and criticality). We were going to upgrade to 3.6.5 everywhere but in the end decided not to, mainly because there wa

Re: [Openstack-operators] RabbitMQ 3.6.x experience?

2017-01-10 Thread Matt Fischer
On Tue, Jan 10, 2017 at 4:08 PM, Sam Morrison wrote: > > > On 10 Jan 2017, at 11:04 pm, Tomáš Vondra wrote: > > > > The version is 3.6.2, but the issue that I believe is relevant is still > not fixed: > > https://github.com/rabbitmq/rabbitmq-management/issues/41 > > Tomas > > > > Yeah we found t

Re: [Openstack-operators] What would you like in Pike?

2017-01-17 Thread Matt Fischer
Another +1 for mult-attach please. On Mon, Jan 16, 2017 at 6:09 AM, Amrith Kumar wrote: > I echo this sentiment; attaching a single Cinder volume or a group of > volumes in a consistency group to multiple instances would be something I’d > like to see in Pike. > > > > -amrith > > > > *From:* Yag

Re: [Openstack-operators] OsOps Reboot

2017-01-23 Thread Matt Fischer
Will there be enough of us at the PTG for an impromptu session there as well? On Mon, Jan 23, 2017 at 9:18 AM, Mike Dorman wrote: > +1! Thanks for driving this. > > > > > > *From: *Edgar Magana > *Date: *Friday, January 20, 2017 at 1:23 PM > *To: *"m...@mattjarvis.org.uk" , Melvin Hillsman < >

Re: [Openstack-operators] [openstack-operators] Dynamically Determine Rabbit Transport URL?

2017-02-06 Thread Matt Fischer
You would know all this at install time as that's when this would be determined. If that information is not available to you currently, you can look at some other service's config files I suppose. You'll need enough rabbit creds to create a Congress rabbit user unless you are just going to re-use a

Re: [Openstack-operators] Sharing fernet tokens

2017-02-07 Thread Matt Fischer
Do you mean sharing tokens or keys? On Feb 7, 2017 11:34 AM, "Ignazio Cassano" wrote: > Hi everybody, > Can anyone talk me about Sebring fernet tokens in an openstack with more > than one controller? > Regards > Ignazio > > > > ___ > OpenStack-operator

Re: [Openstack-operators] Sharing fernet tokens

2017-02-08 Thread Matt Fischer
which simplifies the problem for you. On Tue, Feb 7, 2017 at 9:25 PM, Matt Fischer wrote: > Do you mean sharing tokens or keys? > > On Feb 7, 2017 11:34 AM, "Ignazio Cassano" > wrote: > >> Hi everybody, >> Can anyone talk me about Sebring fernet tokens

Re: [Openstack-operators] Sharing fernet tokens

2017-02-09 Thread Matt Fischer
http://www.mattfischer.com/blog/?p=648 https://www.youtube.com/watch?v=702SRZHdNW8 On Wed, Feb 8, 2017 at 8:14 AM, Matt Fischer wrote: > I think that you just replied to me directly. But you are asking about > sharing keys. > > Since keys do not need to be in-sync on all nodes at the

Re: [Openstack-operators] Scaling the Ops Meetup

2015-07-02 Thread Matt Fischer
Are you proposing an Operators committee or do you mean the OpenStack BoD? On Thu, Jul 2, 2015 at 12:15 PM, Jesse Keating wrote: > Honestly I'm fine with the elected board helping to make this decision. > Folks that want to underwrite the event can submit a proposal to host, > board picks from t

Re: [Openstack-operators] Dynamic Policy

2015-08-05 Thread Matt Fischer
Jumping in with another "us too" here. We have some custom Horizon extensions that allow project owners to manage some of this stuff. On Wed, Aug 5, 2015 at 4:14 PM, Marc Heckmann wrote: > Echoing what others have said, we too have an abstraction layer in the > form of a custom UI to allow proje

Re: [Openstack-operators] Gaining access to consoles.

2015-08-10 Thread Matt Fischer
On Sun, Aug 9, 2015 at 11:59 PM, Tony Breeds wrote: > Hi All, > Nova has bug: https://bugs.launchpad.net/nova/+bug/1447679 (service > No-VNC > (port 6080) doesn't require authentication). > > Which explains that if you know the 'token'[1] associated with an instances > console you can get acc

Re: [Openstack-operators] [openstack-dev] Gaining access to consoles.

2015-08-11 Thread Matt Fischer
On Tue, Aug 11, 2015 at 8:16 PM, Tony Breeds wrote: > On Mon, Aug 10, 2015 at 07:16:43PM -0600, Matt Fischer wrote: > > > I'm not excited about making this the default until token revocations > don't > > impact performance the way that they do now. I don't kno

Re: [Openstack-operators] [openstack-dev] Gaining access to consoles.

2015-08-11 Thread Matt Fischer
Oh.. oops. Yeah if that's the case then sorry, you can just ignore me! On Tue, Aug 11, 2015 at 8:39 PM, Tony Breeds wrote: > On Tue, Aug 11, 2015 at 08:24:10PM -0600, Matt Fischer wrote: > > It was covered some here: > > http://lists.openstack.org/pipermail/openstack-dev/2

Re: [Openstack-operators] Rate limit an max_count

2015-09-10 Thread Matt Fischer
While I think there is probably some value in rate limiting API calls, I think your "user wants to launch x000 instances" is extremely limited. There's maybe 1 or 2 (or 0) operators that have that amount of spare capacity just sitting around that they can allow a user to have a quota of 2000 instan

Re: [Openstack-operators] Tokyo Summit Ops Design Summit Tracks - Draft Agenda

2015-09-22 Thread Matt Fischer
Tom, Can you make the columns a bit wider? I don't seem to have permissions to do so and I cant read everything. I've resorted to copying and pasting stuff into another window so I can read it. On Mon, Sep 21, 2015 at 11:04 PM, Tom Fifield wrote: > Hi all, > > I've started wrangling things tow

Re: [Openstack-operators] [puppet] feedback request about puppet-keystone

2015-09-27 Thread Matt Fischer
On Fri, Sep 25, 2015 at 11:01 AM, Emilien Macchi wrote: > > > So after 5 days, here is a bit of feedback (13 people did the poll [1]): > > 1/ Providers > Except for 1, most of people are managing a few number of Keystone > users/tenants. > I would like to know if it's because the current implement

Re: [Openstack-operators] Milti-site Keystone & Galera

2015-09-28 Thread Matt Fischer
Yes. We have a separate DB cluster for global stuff like Keystone & Designate, and a regional cluster for things like nova/neutron etc. On Mon, Sep 28, 2015 at 10:43 AM, Curtis wrote: > Hi, > > For organizations with the keystone database shared across regions via > galera, do you just have keys

Re: [Openstack-operators] "Master" keystone and "sub" keystone

2015-09-28 Thread Matt Fischer
On Mon, Sep 28, 2015 at 1:46 PM, Jonathan Proulx wrote: > On Mon, Sep 28, 2015 at 03:31:54PM -0400, Adam Young wrote: > :On 09/26/2015 11:19 PM, RunnerCheng wrote: > :>Hi All, > :>I'm a newbie of keystone, and I'm doing some research about it > :>recently. I have a question about how to deploy it

Re: [Openstack-operators] [openstack-dev] [cinder] [all] The future of Cinder API v1

2015-09-28 Thread Matt Fischer
Yes, people are probably still using it. Last time I tried to use V2 it didn't work because the clients were broken, and then it went back on the bottom of my to do list. Is this mess fixed? http://lists.openstack.org/pipermail/openstack-operators/2015-February/006366.html On Mon, Sep 28, 2015 at

Re: [Openstack-operators] [openstack-dev] [cinder] [all] The future of Cinder API v1

2015-09-29 Thread Matt Fischer
> > > > I agree with John Griffith. I don't have any empirical evidences to back > my "feelings" on that one but it's true that we weren't enable to enable > Cinder v2 until now. > > Which makes me wonder: When can we actually deprecate an API version? I > *feel* we are fast to jump on the deprecat

Re: [Openstack-operators] how to deal with always-growing notifications.info queue?

2015-09-30 Thread Matt Fischer
I'd recommend a few things. The first is you need to disable notifications in your services including nova. You just set the notifications driver to noop. Second, you should have some monitoring in place that looks for queues that go over a certain threshold. There's not a lot of queues that shoul

Re: [Openstack-operators] [openstack-dev] [cinder] [all] The future of Cinder API v1

2015-09-30 Thread Matt Fischer
M, Mark Voelker wrote: > > Mark T. Voelker > > > > > On Sep 29, 2015, at 12:36 PM, Matt Fischer wrote: > > > > > > > > I agree with John Griffith. I don't have any empirical evidences to back > > my "feelings" on that one but it

Re: [Openstack-operators] External API access

2015-10-20 Thread Matt Fischer
One simple workaround for this if you ssh directly to your Keystone node and run the admin commands from there. Once you bootstrap your project with the proper tenants and users it's not an operation that most people do all that often. We expose an admin endpoint on an internal load balancer URL bu

Re: [Openstack-operators] External API access

2015-10-20 Thread Matt Fischer
n endpoint to a public url? > > > On Oct 20, 2015, at 5:28 PM, Matt Fischer wrote: > > > > One simple workaround for this if you ssh directly to your Keystone node > and run the admin commands from there. Once you bootstrap your project with > the proper tenants and user

Re: [Openstack-operators] Problems with https endpoints with IceHouse-->Juno-->Kilo migration

2015-10-27 Thread Matt Fischer
What's your output from keystone endpoint-list or keystone catalog (or the DB table)? Is it possible the admin URL is simply listed as http? On Tue, Oct 27, 2015 at 9:32 PM, Alvise Dorigo wrote: > I have an IceHouse OpenStack installation, where the endpoints are using > https as protocol (i.e.

Re: [Openstack-operators] OPs Midcycle location discussion.

2015-11-16 Thread Matt Fischer
I think that sticking with a singular official one is the plan. It's difficult enough for the foundation to line up sponsors/hosts etc for a single meet-up. I also think that there are some US/Asia folks that will attend a midcycle in Europe and by also hosting a competing one locally you may reduc

Re: [Openstack-operators] OPs Midcycle location discussion.

2015-11-17 Thread Matt Fischer
On Mon, Nov 16, 2015 at 1:00 PM, Donald Talton wrote: > I’ll +1 option 1 too, if we can get remote participation that would > suffice. > > > Having been to several of these I think that we can call remote participation a stretch goal at best, and if I'm being honest, I just don't think it's going

Re: [Openstack-operators] OPs Midcycle location discussion.

2015-11-17 Thread Matt Fischer
> > > We're deciding not to innovate a solution to allow people to > participate in a group that is attempting to provide innovative ideas. > How ironic. I actually don't think it would require much innovation. > The Ceph guys run their entire design summit remotely, and I'm certain > that it way b

Re: [Openstack-operators] [openstack-operators][osops] tools-contrib is open for business!

2015-11-19 Thread Matt Fischer
Is there a reason why we can't license the entire repo with Apache2 and if you want to contribute you agree to that? Otherwise it might become a bit of a nightmare. Or maybe at least do "Apache2 unless otherwise stated"? On Thu, Nov 19, 2015 at 9:17 PM, Joe Topjian wrote: > Thanks, JJ! > > It l

Re: [Openstack-operators] How do I install specific versions of openstack/puppet-keystone

2015-11-25 Thread Matt Fischer
I'd second the vote for r10k. You need to do this however otherwise you'll get the master branch: mod 'nova', :git => 'https://github.com/openstack/puppet-nova.git', :ref => 'stable/kilo' mod 'glance', :git => 'https://github.com/openstack/puppet-glance.git', :ref => 'stable/kilo' mod 'c

Re: [Openstack-operators] How do I install specific versions of openstack/puppet-keystone

2015-11-26 Thread Matt Fischer
I'm just going to be crystal clear. Use r10k with a Puppetfile that points at specific branches (or tags) and all your problems will go away. World peace, etc. I am never quite sure what librarian is up to, and I've found it's caching annoying. r10k just works. gem install --no-rdoc r10k r10k pupp

Re: [Openstack-operators] instances floating IPs not reachable while restarting nova-network

2015-11-26 Thread Matt Fischer
For reference, neutron has similar issues when restarting some neutron services, for example the ovs-agent plugin, The delay in coming back up scales on the number of routers you are hosting. For this reason we don't let puppet restart the senstitive services and our "rabbit connections are broken

Re: [Openstack-operators] Galera setup testing

2015-12-07 Thread Matt Fischer
On Mon, Dec 7, 2015 at 3:54 AM, Ajaya Agrawal wrote: > Hi everyone, > > We are deploying Openstack and planning to run multi-master Galera setup > in production. My team is responsible for running a highly available > Keystone. I have two questions when it comes to Galera with Keystone. > > 1. Ho

Re: [Openstack-operators] Galera setup testing

2015-12-11 Thread Matt Fischer
On Fri, Dec 11, 2015 at 12:25 AM, Ajaya Agrawal wrote: > Thanks Matt. That surely is helpful. If you could share some numbers or > problems you faced when you were storing UUID tokens in database, it would > be awesome. In my test setup with Keystone Kilo, Fernet token creation and > validation w

Re: [Openstack-operators] kilo keystone with liberty nova

2016-01-06 Thread Matt Fischer
We've done the opposite, newer Keystone with older code. No issues that we've seen. On Wed, Jan 6, 2016 at 8:15 AM, Kevin Bringard (kevinbri) < kevin...@cisco.com> wrote: > We've even done later versions of keystone with older versions of other > stuff (Specifically Kilo Keystone with Juno Glance

Re: [Openstack-operators] [nova] Recovering instances from old system

2016-01-11 Thread Matt Fischer
Personally, I'd just try to load the instance images like you said. If you try to load Icehouse records onto Liberty code its not going to work. Typically you'd do the upgrade one step at a time with database migrations done at every step. On Sun, Jan 10, 2016 at 9:58 PM, Liam Haworth wrote: > H

Re: [Openstack-operators] Help with horizon and v3 auth

2016-02-05 Thread Matt Fischer
Are you seeing the cinder Volume limit error? If that's the issue the work around is here in the bug description. https://bugs.launchpad.net/tripleo/+bug/1521639 On Feb 4, 2016 10:31 PM, "Abel Lopez" wrote: > Hey everyone, > In my liberty testing, I've got keystone v3 setup, and everything seem

Re: [Openstack-operators] [kolla] Question about how Operators deploy

2016-02-12 Thread Matt Fischer
We also use 2 VIPs. public and internal, with admin being a CNAME for internal. On Fri, Feb 12, 2016 at 7:28 AM, Fox, Kevin M wrote: > We usually use two vips. > > Thanks, > Kevin > > -- > *From:* Steven Dake (stdake) > *Sent:* Friday, February 12, 2016 6:04:45 AM > *

Re: [Openstack-operators] [nova] VM HA support in trunk

2016-02-15 Thread Matt Fischer
I believe that either have your customers design their apps to handle failures or have tools that are reactive to failures. Unfortunately like many other private cloud operators we deal a lot with legacy applications that aren't scaled horizontally or fault tolerant and so we've built tooling to h

Re: [Openstack-operators] [openstack-dev] [all] A proposal to separate the design summit

2016-02-22 Thread Matt Fischer
Cross-post to openstack-operators... As an operator, there's value in me attending some of the design summit sessions to provide feedback and guidance. But I don't really need to be in the room for a week discussing minutiae of implementations. So I probably can't justify 2 extra trips just to giv

Re: [Openstack-operators] [openstack-dev] [puppet] Austin Design Summit space needs

2016-02-24 Thread Matt Fischer
On Wed, Feb 24, 2016 at 8:30 AM, Emilien Macchi wrote: > Puppet OpenStack folks, > > As usual, Thierry Carrez sent an e-mail to PTLs about space needs for > the next OpenStack Summit in Austin. > > > We can have 3 kinds of slots: > > * Fishbowl slots (Wed-Thu) - we had 2 in Tokyo. > Our tradition

Re: [Openstack-operators] Horizon bug fixed in Liberty, how should we ask a backport to Kilo ?

2016-03-03 Thread Matt Fischer
The backport is pretty easy. You click on Cherry pick and if there's no conflict it just works. Like so: https://review.openstack.org/#/c/287928/ It still needs to go through the review process so you will need to ping some horizon developers in IRC. Getting that packaged may take longer. On Th

Re: [Openstack-operators] Liberty Identity install: keystone.service not being created in DB

2016-03-05 Thread Matt Fischer
I think you can ignore that no handlers message, it's not the issue. You should check /var/log/keystone/keystone-manage.Log to find the original issue. You can also run the dbsync with the verbose flag IIRC. On Mar 5, 2016 3:38 PM, "Christopher Hull" wrote: > > Hi all; > > I'm attempting an insta

Re: [Openstack-operators] [openstack-operators] Fernet key rotation

2016-03-19 Thread Matt Fischer
Fernet key rotation is easy. 1) You don't need a maintenance window 2) You can do one node at a time even with a long delay between 3) You don't need to restart anything We rotate approximately weekly. On Wed, Mar 16, 2016 at 3:44 PM, Ajay Kalambur (akalambu) < akala...@cisco.com> wrote: > Hi >

Re: [Openstack-operators] [nova] RFEs: communication channel and process

2016-03-21 Thread Matt Fischer
On Mar 21, 2016 3:28 PM, "Tim Bell" wrote: > > On 21/03/16 17:24, "Markus Zoeller" wrote: > > >Hello dear ops, > > > >I'd like to make you aware of discussion [1] on the openstack-dev ML. > >I'm in the role of maintaining the bug list in Nova and was looking > >for a way to gain an overview agai

Re: [Openstack-operators] [nova] Removing seeded flavors

2016-04-04 Thread Matt Fischer
Another remove vote. The only people this may affect are people standing up test clouds or new to OpenStack. For those folks that use puppet, the puppet community will be adding a provider to setup flavors since it's a feature that's been missing. I'll add a vote for removal, given how varied priv

Re: [Openstack-operators] [glance] glance-registry deprecation: Request for feedback

2016-05-12 Thread Matt Fischer
On May 11, 2016 10:03 PM, "Flavio Percoco" wrote: > > Greetings, > > The Glance team is evaluating the needs and usefulness of the Glance Registry > service and this email is a request for feedback from the overall community > before the team moves forward with anything. > > Historically, there ha

Re: [Openstack-operators] Meeting summary, minutes, definition of 'users'

2016-05-17 Thread Matt Fischer
It's a google group. The only clue I had was this in the headers: X-Auto-Response-Suppress: All X-MS-Exchange-Inbox-Rules-Loop: tgree...@outlook.com X-MS-TNEF-Correlator: I reached out to that person and no response. On Tue, May 17, 2016 at 10:42 AM, Jeremy Stanley wrote: > On 2016-05-17 17:3

Re: [Openstack-operators] Meeting summary, minutes, definition of 'users'

2016-05-17 Thread Matt Fischer
t;openstack-private@some.random.domain" and > > nothing we're in control of, but I guess I'll find out when it > > bounces back to my reply. > > Aah, as Matt Fischer pointed out in IRC just now, it seems to be > forwarded through an outlook.com subscriber acco

Re: [Openstack-operators] Uptime and SLA's

2016-06-02 Thread Matt Fischer
We do this a few different ways, some of which may meet your needs. For API calls we measure a simple, quick, and impactless call for each service (like heat stack-list) and we monitor East from West and vice versa. The goal here is nothing added to the DBs, so nothing like neutron net-create. The

Re: [Openstack-operators] How are people dealing with API rate limiting?

2016-06-14 Thread Matt Fischer
I will posit that anyone who is interested in rate limiting is probably already load balancing their API servers. We've been looking into rate limiting at the load balancers, but have not needed to implement it yet. That will likely be our solution when its finally implemented. Question: If there

Re: [Openstack-operators] How are people dealing with API rate limiting?

2016-06-14 Thread Matt Fischer
On Tue, Jun 14, 2016 at 9:37 AM, Sean Dague wrote: > On 06/14/2016 11:02 AM, Matt Riedemann wrote: > > A question came up in the nova IRC channel this morning about the > > api_rate_limit config option in nova which was only for the v2 API. > > > > Sean Dague explained that it never really worked

Re: [Openstack-operators] [Openstack-Operators] Keystone cache strategies

2016-06-21 Thread Matt Fischer
I don't have a solution for you, but I will concur that adding revocations kills performance especially as that tree grows. I'm curious what you guys are doing revocations on, anything other than logging out of Horizon? On Tue, Jun 21, 2016 at 5:45 AM, Jose Castro Leon wrote: > Hi all, > > While

Re: [Openstack-operators] [Openstack-Operators] Keystone cache strategies

2016-06-21 Thread Matt Fischer
On Tue, Jun 21, 2016 at 4:21 PM, Sam Morrison wrote: > > On 22 Jun 2016, at 1:45 AM, Matt Fischer wrote: > > I don't have a solution for you, but I will concur that adding revocations > kills performance especially as that tree grows. I'm curious what you guys &

Re: [Openstack-operators] [Openstack-Operators] Keystone cache strategies

2016-06-21 Thread Matt Fischer
Have you setup token caching at the service level? Meaning a Memcache cluster that glance, Nova etc would talk to directly? That will really cut down the traffic. On Jun 21, 2016 5:55 PM, "Sam Morrison" wrote: > > On 22 Jun 2016, at 9:42 AM, Matt Fischer wrote: > > On Tue

Re: [Openstack-operators] [Openstack-Operators] Keystone cache strategies

2016-06-21 Thread Matt Fischer
On Tue, Jun 21, 2016 at 7:04 PM, Sam Morrison wrote: > > On 22 Jun 2016, at 10:58 AM, Matt Fischer wrote: > > Have you setup token caching at the service level? Meaning a Memcache > cluster that glance, Nova etc would talk to directly? That will really cut > down the traffic

Re: [Openstack-operators] Keystone's DB_SYNC from Kilo to Liberty

2016-06-23 Thread Matt Fischer
IIRC there are some debug/verbose flags you can pass in. Get anything from them? On Jun 23, 2016 5:37 AM, "Alvise Dorigo" wrote: > Hi, > I've a Kilo installation which I want to migrate to Liberty. > I've installed the Liberty Keystone's RPMs and configured the minimun to > upgrade the DB schema

[Openstack-operators] OpenStack Trove Ocata Virtual Midcycle

2016-06-25 Thread Matt Fischer
cross-posting per Amrith Kumar to operators: (note I'd recommend a reply to the openstack-dev thread or directly to amr...@tesora.com) After we discussed and announced this mid-cycle, there has been some feedback that (a) it would be better to hold the mid-cycle earlier, and (b) NYC was not the

Re: [Openstack-operators] Bandwidth limitations

2016-06-29 Thread Matt Fischer
We've been using this for some time now (since at least Kilo). We set them per flavor not per instance. https://wiki.openstack.org/wiki/InstanceResourceQuota Bandwidth limits Nova Extra Specs keys: - vif_inbound_average - vif_outbound_average - vif_inbound_peak - vif_outbound_peak

Re: [Openstack-operators] Bandwidth limitations

2016-06-29 Thread Matt Fischer
in Openstack, > however I'd like them to be applied automatically. Using predefined flavors > as described by Matt Fischer above seems like a good approach, are there > any solutions for non-predefined flavors? > > > - Original message - > From: Assaf Muller >

Re: [Openstack-operators] [nova] Rabbit-mq 3.4 crashing (anyone else seen this?)

2016-07-05 Thread Matt Fischer
Yes! This happens often but I'd not call it a crash, just the mgmt db gets behind then eats all the memory. We've started monitoring it and have runbooks on how to bounce just the mgmt db. Here are my notes on that: restart rabbitmq mgmt server - this seems to clear the memory usage. rabbitmqctl

Re: [Openstack-operators] [nova] Rabbit-mq 3.4 crashing (anyone else seen this?)

2016-07-05 Thread Matt Fischer
For the record we're on 3.5.6-1. On Jul 5, 2016 11:27 AM, "Mike Lowe" wrote: > I was having just this problem last week. We updated to 3.6.2 from 3.5.4 > on ubuntu and stated seeing crashes due to excessive memory usage. I did > this on each node of my rabbit cluster and haven’t had any problems

Re: [Openstack-operators] [puppet] [desginate] An update on the state of puppet-designate (and designate in RDO)

2016-07-05 Thread Matt Fischer
We're using Designate but still on Juno. We're running puppet from around then, summer of 2015. We'll likely try to upgrade to Mitaka at some point but Juno Designate "just works" so it's been low priority. Look forward to your efforts here. On Tue, Jul 5, 2016 at 7:47 PM, David Moreau Simard wro

Re: [Openstack-operators] Getting help with Python API

2016-07-06 Thread Matt Fischer
When you make the API calls you're going to get back a list python objects which you need to iterate. I believe some APIs will let you ask for specific fields only, but this is simple enough: keystone = client.Client(username=Username, password=Password, tenant_name=Tenant, auth_url='h

Re: [Openstack-operators] Next Ops Midcycle NYC August 25-26

2016-07-07 Thread Matt Fischer
That's my comment I spoke to Mark V about it this morning and he's working on it already, so you may want to coordinate with him. On Thu, Jul 7, 2016 at 11:20 AM, Amrith Kumar wrote: > I see a comment in https://etherpad.openstack.org/p/NYC-ops-meetup about > “OpenStack East Discount?”. > > > >

Re: [Openstack-operators] Ops MidCycle Registration

2016-07-14 Thread Matt Fischer
Thanks Erin. I did this just now and it charged me $22.09. Not a big deal, but what's the extra? Taxes? On Jul 14, 2016 3:43 PM, "Erin Disney" wrote: > All- > > Thank you for your patience as we finalized details for the Ops MidCycle > in New York this August. If you plan to attend, please RSVP

Re: [Openstack-operators] Ops MidCycle Registration

2016-07-15 Thread Matt Fischer
t; > > > -amrith > > > > *From:* Matt Fischer [mailto:m...@mattfischer.com] > *Sent:* Thursday, July 14, 2016 6:26 PM > *To:* Erin Disney > *Cc:* openstack-operators@lists.openstack.org > *Subject:* Re: [Openstack-operators] Ops MidCycle Registration > > > &

Re: [Openstack-operators] External access to OpenStack services

2016-07-30 Thread Matt Fischer
I'd say that operators running Glance, which is probably almost everyone, just put a public glance endpoint in the catalog. Maybe there's some special cases beyond that but that's the base design. On Jul 30, 2016 6:22 PM, "Serguei Bezverkhi (sbezverk)" wrote: > Hi Joseph, > > > > I am working on

Re: [Openstack-operators] Mid-Cycle Meetup, NYC, August 25th, call for additional working group sessions

2016-08-09 Thread Matt Fischer
I didn't see any plus ones on my idea for the db cleanup session so if we need to drop it to fit something that works for me. On Aug 9, 2016 12:29 PM, "Chris Morgan" wrote: > WG6, day one? That's 40 minutes. Would run alongside Large Deployment. > Currently that has the main room. Would nova be

Re: [Openstack-operators] Mid-Cycle Meetup, NYC, August 25th, call for additional working group sessions

2016-08-11 Thread Matt Fischer
morning (pre-lunch) on Friday, or Thursday please. >> >> >> >> Thanks, >> >> >> >> -amrith >> >> >> >> *From:* Chris Morgan [mailto:mihali...@gmail.com] >> *Sent:* Tuesday, August 09, 2016 4:38 PM >> *To:* Matt

Re: [Openstack-operators] [oslo] RabbitMQ queue TTL issues moving to Liberty

2016-08-15 Thread Matt Fischer
Has anyone had any luck improving the statsdb issue by upgrading rabbit to 3.6.3 or newer? We're at 3.5.6 now and 3.6.2 has parallelized stats processing, then 3.6.3 has additional memory leak fixes for it. What we've been seeing is that we occasionally get slow & steady climbs of rabbit memory usa

Re: [Openstack-operators] Keystone upgrade issues

2016-08-25 Thread Matt Fischer
Jonathan, Are you using caching for tokens (not the middleware cache but keystone cache)? There's a bug in the caching so that when it tries to read the cache and unpack the token its missing some fields. It's been fixed and backported but may not be in your packages: https://bugs.launchpad.net/ke

Re: [Openstack-operators] Database cleanup scripts?

2016-09-02 Thread Matt Fischer
Hi Ed, Good to meet you in NYC last week. And fortunate timing for the question, I just published a summary of my experiences here: http://www.mattfischer.com/blog/?p=744 I know that the Nova DB cleanup stuff was broken in the past, and IIRC you are on Kilo, so it may not work for you until you g

Re: [Openstack-operators] Database cleanup scripts?

2016-09-02 Thread Matt Fischer
On Fri, Sep 2, 2016 at 8:57 AM, Abel Lopez wrote: > For cinder, since kilo, we've had 'cinder-manage db purge-deleted' > > This is the issue we see with this tool in Liberty, I think this might be fixed in M. # cinder-manage db purge 365 (some stuff works here) ... 2016-09-02 15:07:02.196

Re: [Openstack-operators] Murano in Production

2016-09-18 Thread Matt Fischer
+1 This was our concern also with Trove. If a tenant DoSes Trove we probably don't all get fired. The rest of rabbit is just too important to risk sharing. On Sun, Sep 18, 2016 at 6:53 PM, Sam Morrison wrote: > We run completely separate clusters. I’m sure vhosts give you acceptable > security b

Re: [Openstack-operators] Auto start running Nova Instances after reboot

2016-09-19 Thread Matt Fischer
On Mon, Sep 19, 2016 at 7:29 AM, Tobias Urdin wrote: > Hello, > > On your compute nodes in nova.conf > > [DEFAULT] > > resume_guests_state_on_host_boot = True > > > All instances that had a running state when the reboot occured will be > started again. > > Best regards > And this works regardles

Re: [Openstack-operators] Murano in Production

2016-09-23 Thread Matt Fischer
Other that #1 that's exactly the same design we used for Trove. Glad to see someone else using it too for validation. Thanks. On Sep 22, 2016 11:39 PM, "Serg Melikyan" wrote: > Hi Joe, > > I can share some details on how murano is configured as part of the > default Mirantis OpenStack configurat

Re: [Openstack-operators] Custom VM FQDNs and DNS integration

2016-10-10 Thread Matt Fischer
The last time I tried this, which was probably 18 months ago to be fair, there is no way for the VM to get it's own tenant name. You could pass it in with cloud-init if you want but its not in the metadata that I recall. For Designate however I don't know why you'd want this. You want the format a

Re: [Openstack-operators] How do you even test for that?

2016-10-17 Thread Matt Fischer
This does not cover all your issues but after seeing mysql bugs between I and J and also J to K we now export and restore production control plane data into a dev environment to test the upgrades. If we have issues we destroy this environment and run it again. For longer running instances that's t

Re: [Openstack-operators] [Nova][icehouse]Any way to rotating log by size

2016-10-19 Thread Matt Fischer
On Wed, Oct 19, 2016 at 10:22 AM, Sean M. Collins wrote: > Zhang, Peng wrote: > > [logger_root] > > level = DEBUG > > > So, you're setting the logging to level to DEBUG - if I understand > correctly. In a production environment that is going to fill up your > disks very quickly. Which is why even

Re: [Openstack-operators] Ceilometer/oslo.messaging connect to multiple RMQ endpoints

2016-11-03 Thread Matt Fischer
Unless this has drastically changed I thought the multiple entries was sort of like a "pick one" scenario rather than a "connect to all of them". You specify all the nodes in case one or more is down. I don't think it can be used to talk to multiple rabbit clusters. On Thu, Nov 3, 2016 at 5:28 PM,

Re: [Openstack-operators] [openstack-dev] [keystone][tripleo][ansible][puppet][all] changing default token format

2016-11-07 Thread Matt Fischer
How to add yourself to Planet OpenStack: https://wiki.openstack.org/wiki/AddingYourBlog As for SuperUser you could reach out to them if you think it's interesting for users/operators. Generally they'll want to publish it there first then you follow-up with your blog post a few days later. On Mon,

Re: [Openstack-operators] FYI: Rabbit Heartbeat Patch Landed

2015-03-19 Thread Matt Fischer
I think everyone is highly interested in running this change or a newer OSLO messaging in general + this change in Juno rather than waiting for Kilo. Hopefully everyone could provide updates as they do experiments. On Thu, Mar 19, 2015 at 1:22 PM, Kevin Bringard (kevinbri) < kevin...@cisco.com> wr

[Openstack-operators] heat delete woes (juno)

2015-03-24 Thread Matt Fischer
We've been having some issues with heat delete-stack in Juno. The issues generally fall into three categories: 1) it takes multiple calls to heat to delete a stack. Presumably due to heat being unable to figure out the ordering on deletion and resources being in use. 2) undeleteable stacks. Stack

[Openstack-operators] [heat] heat delete woes

2015-03-26 Thread Matt Fischer
Nobody on the operators list had any ideas on this, so re-posting here. We've been having some issues with heat delete-stack in Juno. The issues generally fall into three categories: 1) it takes multiple calls to heat to delete a stack. Presumably due to heat being unable to figure out the orderi

Re: [Openstack-operators] [heat] heat delete woes

2015-03-26 Thread Matt Fischer
Sorry operators. I fail at email today. This was for -dev. On Thu, Mar 26, 2015 at 12:05 PM, Matt Fischer wrote: > Nobody on the operators list had any ideas on this, so re-posting here. > > We've been having some issues with heat delete-stack in Juno. The issues > generall

Re: [Openstack-operators] Security around enterprise credentials and OpenStack API

2015-03-31 Thread Matt Fischer
Mathieu, We LDAP (AD) with a fallback to MySQL. This allows us to store service accounts (like nova) and "team accounts" for use in Jenkins/scripts etc in MySQL. We only do Identity via LDAP and we have a forked copy of this driver (https://github.com/SUSE-Cloud/keystone-hybrid-backend) to do this

[Openstack-operators] logging for Keystone on user/project delete/create operations

2015-04-15 Thread Matt Fischer
I'd like to have some better logging when certain CRUD operations happen in Keystone, for example, when a project is deleted. I specifically mean "any" when I say better since right now I'm not seeing anything even when Verbose is enabled. This is pretty frustrating for me because these are rather

Re: [Openstack-operators] logging for Keystone on user/project delete/create operations

2015-04-16 Thread Matt Fischer
> I’m not involved in the keystone project, but I’d recommend you to >> > start by filling a blueprint >> > asking for it, and explaining what you just said here: >> > >> > https://blueprints.launchpad.net/keystone >> > >> > Adding a blueprint

Re: [Openstack-operators] expanding to 2nd location

2015-05-05 Thread Matt Fischer
We do it with some of our databases (horizon, designate, and keystone) and we run a arbitrator process (garbd) in a 3rd DC. We have lots of low latency bandwidth which you have to be careful with. My recommendation would be that you need to know your network well and have good monitoring in place.

Re: [Openstack-operators] Packaging, Deployment CI/CD - Moderators Needed (Vancouver)

2015-05-10 Thread Matt Fischer
Tom, This doesn't solve your problem, but I will gladly swap Database for Deployment/CI/CD. I have more experience on that topic and am even presenting on it. On Sun, May 10, 2015 at 9:31 PM, Tom Fifield wrote: > Hi all, > > We're in need of moderators for these ops sessions in Vancouver: > > 1

[Openstack-operators] Deployment & CI/CD at the Summit

2015-05-11 Thread Matt Fischer
Greetings operators, I am moderating the Deployments/CI/CD design session at the summit next week on Tuesday at 3:40 PM in Room 220 [1]. This is a large, wide-ranging, and important topic for operators, so I'd like to get some help filling out the Etherpad [2] with things you'd like to discuss. I'

[Openstack-operators] Ops CI/CD, Deployments Session

2015-05-23 Thread Matt Fischer
Thanks to everyone who attended the CI/CD, Deployments sessions. We had a great discussion, but unfortunately etherpad was broken for the duration of the time. If anyone would like to add any notes on some of the new tools discussed during that talk, please add them. I don't recall any specific act

Re: [Openstack-operators] [openstack-dev][openstack-operators][chef] OpenStack+Chef is part of the big tent

2015-05-26 Thread Matt Fischer
Congrats and welcome! On May 26, 2015 5:35 PM, "JJ Asghar" wrote: > Hey everyone! > > I’d like to just drop a note to the list saying thank you and > congratulations to our general community. > > As of 2015-05-26 we’ve been merged into the “big tent”[1] sanctioning us > as an official OpenStack p

Re: [Openstack-operators] [Puppet] Puppet classes for Kilo?

2015-06-04 Thread Matt Fischer
Cynthia, There are a few things we're waiting to land, keystone v3 support being one major one. The kilo branches should get cut soon though, until then please use master. There will be an announcement on OpenStack dev when they're ready. ___ OpenStack-o

  1   2   >