On 08/13/2013 10:39 AM, Angus Salkeld wrote: > On 12/08/13 16:52 -0400, Doug Hellmann wrote: >> On Mon, Aug 12, 2013 at 4:11 PM, Clint Byrum <[email protected]> wrote: >> >>> Excerpts from Doug Hellmann's message of 2013-08-12 12:08:58 -0700: >>> > On Fri, Aug 9, 2013 at 11:56 AM, Clint Byrum <[email protected]> >>> wrote: >>> > >>> > > Excerpts from Sandy Walsh's message of 2013-08-09 06:16:55 -0700: >>> > > > >>> > > > On 08/08/2013 11:36 PM, Angus Salkeld wrote: >>> > > > > On 08/08/13 13:16 -0700, Clint Byrum wrote: >>> > > > >> Last night while reviewing a feature which would add more >>> events >>> to >>> > > the >>> > > > >> event table, it dawned on me that the event table really >>> must be >>> > > removed. >>> > > > > >>> > > > > >>> > > > >> >>> > > > >> https://bugs.launchpad.net/heat/+bug/1209492 >>> > > > >> >>> > > > >> tl;dr: users can write an infinite number of rows to the event >>> table >>> > > at >>> > > > >> a fairly alarming rate just by creating and updating a very >>> large >>> > > stack >>> > > > >> that has no resources that cost any time or are even billable >>> (like an >>> > > > >> autoscaling launch configuration). >>> > > > >> >>> > > > >> The table has no purge function, so the only way to clear >>> out old >>> > > events >>> > > > >> is to delete the stack, or manually remove them directly in >>> the >>> > > database. >>> > > > >> >>> > > > >> We've all been through this before, logging to a database >>> seems >>> great >>> > > > >> until you actually do it. >>> > > > >> >>> > > > >> I have some ideas for how to solve it, but I wanted to get >>> a wider >>> > > > >> audience: >>> > > > >> >>> > > > >> 1) Make the event list a ring buffer. Have rows 0 - >>> $MAX_BUFFER_SIZE >>> > > in >>> > > > >> each stack, and simply write each new event to the next open >>> position, >>> > > > >> wrapping at $MAX_BUFFER_SIZE. Pros: little change to >>> current code, >>> > > > >> just need an offset column added and code that will >>> properly wrap >>> to 0 >>> > > > >> at $MAX_BUFFER_SIZE. Cons: still can incur heavy transactional >>> load on >>> > > > >> the database server.A >>> > > > >> >>> > > > >> 1.b) Same, but instead of rows, just maintain a blob and >>> append >>> the >>> > > rows >>> > > > >> as json list. Lowers transactional load but would push some >>> load >>> onto >>> > > > >> the API servers and such to parse these out, and would make >>> pagination >>> > > > >> challenging. Blobs also can be a drain on DB server >>> performance. >>> > > > >> >>> > > > >> 2) Write a purge script. Delete old ones. Pros: No code >>> change, >>> just >>> > > > >> new code to do purging. Cons: same as 1, plus more >>> vulnerability >>> to an >>> > > > >> aggressive attacker who can fit a lot of data in between >>> purges. >>> Also >>> > > > >> large scale deletes can be really painful (see: keystone >>> sql token >>> > > > >> backend). >>> > > > >> >>> > > > >> 3) Log events to Swift. I can't seem to find information on >>> how/if >>> > > > >> appending works there. Tons of tiny single-row files is an >>> option, >>> > > but I >>> > > > >> want to hear from people with more swift knowledge if that >>> is a >>> > > viable, >>> > > > >> performant option. Pros: Scale to the moon. Can charge >>> tenant for >>> > > usage >>> > > > >> and let them purge events as needed. Cons: Adds swift as a >>> requirement >>> > > > >> of Heat. >>> > > > >> >>> > > > >> 4) Provide a way for users to receive logs via HTTP POST. >>> Pros: >>> Simple >>> > > > >> and punts the problem to the users. Cons: users will be SoL if >>> they >>> > > > >> don't have a place to have logs posted to. >>> > > > >> >>> > > > >> 5) Provide a way for users to receive logs via messaging >>> service >>> like >>> > > > >> Marconi. Pros/Cons: same as HTTP, but perhaps a little more >>> confusing >>> > > > >> and ambitious given Marconi's short existence. >>> > > > >> >>> > > > >> 6) Provide a pluggable backend for logging. This seems like >>> the >>> way >>> > > most >>> > > > >> OpenStack projects solve these issues, which is to let the >>> deployers >>> > > > >> choose and/or provide their own way to handle a sticky >>> problem. >>> Pros: >>> > > > >> Simple and flexible for the future. Cons: Would require >>> writing at >>> > > least >>> > > > >> one backend provider that does what the previous 5 options >>> suggest. >>> > > > >> >>> > > > >> To be clear: Heat cannot really exist without this, as it >>> is the >>> only >>> > > way >>> > > > >> to find out what your stack is doing or has done. >>> > > > > >>> > > > > btw Clint I have ditched that "Recorder" patch as Ceilometer is >>> > > > > getting a Alarm History api soon, so we can defer to that >>> for that >>> > > > > functionality (alarm transitions). >>> > > > > >>> > > > > But we still need a better way to record events/logs for the >>> user. >>> > > > > So I make this blueprint a while ago: >>> > > > > https://blueprints.launchpad.net/heat/+spec/user-visible-logs >>> > > > > >>> > > > > I am becomming more in favor of user options rather than >>> deployer >>> > > > > options if possible. So provide resources for Marconi, >>> Meniscus and >>> > > > > what ever... >>> > > > > Although what is nice about Marconi is you could then hook >>> up what >>> > > > > ever you want to it. >>> > > > >>> > > > Logs are one thing (and Meniscus is a great choice for that), but >>> events >>> > > > are the very thing CM is designed to handle. Wouldn't it make >>> sense >>> to >>> > > > push them back into there? >>> > > > >>> > > >>> > > I'm not sure these events make sense in the current Ceilometer (I >>> assume >>> > > that is "CM" above) context. These events are: >>> > > >>> > > ... Creating stack A >>> > > ... Creating stack A resource A >>> > > ... Created stack A resource A >>> > > ... Created stack A >>> > > >>> > > Users will want to be able to see all of the events for a stack, >>> and >>> > > likely we need to be able to paginate through them as well. >>> > > >>> > > They are fundamental and low level enough for Heat that I'm not >>> sure >>> > > putting them in Ceilometer makes much sense, but maybe I don't >>> understand >>> > > Ceilometer.. or "CM" is something else entirely. :) >>> > > >>> > >>> > CM is indeed ceilometer. >>> > >>> > The plan for the event API there is to make it admin-only (at >>> least for >>> > now). If this is data the user wants to see, that may change the >>> plan for >>> > the API or may mean storing it in ceilometer isn't a good fit. >>> > >>> >>> Visibility into these events is critical to tracking the progress of >>> any action done to a Heat stack: >>> >>> >>> +---------------------+----+------------------------+--------------------+----------------------+ >>> >>> | logical_resource_id | id | resource_status_reason | >>> resource_status | >>> event_time | >>> >>> +---------------------+----+------------------------+--------------------+----------------------+ >>> >>> | AccessPolicy | 24 | state changed | >>> CREATE_IN_PROGRESS | >>> 2013-08-12T19:45:36Z | >>> | AccessPolicy | 25 | state changed | >>> CREATE_COMPLETE | >>> 2013-08-12T19:45:36Z | >>> | User | 26 | state changed | >>> CREATE_IN_PROGRESS | >>> 2013-08-12T19:45:36Z | >>> | Key | 28 | state changed | >>> CREATE_IN_PROGRESS | >>> 2013-08-12T19:45:38Z | >>> | User | 27 | state changed | >>> CREATE_COMPLETE | >>> 2013-08-12T19:45:38Z | >>> | Key | 29 | state changed | >>> CREATE_COMPLETE | >>> 2013-08-12T19:45:39Z | >>> | notcompute | 30 | state changed | >>> CREATE_IN_PROGRESS | >>> 2013-08-12T19:45:40Z | >>> >>> +---------------------+----+------------------------+--------------------+----------------------+ >>> >>> >>> So unless there is a plan to make this a user centric service, it does >>> not seem like a good fit. >>> >>> > Are these "events" transmitted in the same way as notifications? >>> If so, >>> we >>> > may already have the data. >>> > >>> >>> The Heat engine records them while working on the stack. They have a >>> fairly narrow, well defined interface, so it should be fairly easy to >>> address the storage issue with a backend abstraction. >>> >> >> OK. The term "event" frequently means "notification" for ceilometer, >> but it >> sounds like it's completely different in this case. > > Yeah, not really related. But we need to add RPC Notifications to Heat > soon so people can bill on a stack basis (create/update/delete/exist). > > What we are talking about here is logging really but we want to get it > to the end user (that does not have access to the infrastructure's > syslog). > Do we really want to enable having a billing policy on stacks? A stack invokes resources which have their own billing usage policies.
_______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
