Hi Flavio, In your discussions around a developing Kafka plugin for Marconi would that be potentially be done by adding a Kafka transport to oslo.messaging? That is something that I'm very interested in for the monitoring as a service project I'm working on.
Thanks --Roland On 6/4/14, 3:06 AM, "Flavio Percoco" <fla...@redhat.com> wrote: >On 02/06/14 07:52 -0700, Keith Newstadt wrote: >>Thanks for the responses Flavio, Roland. >> >>Some background on why I'm asking: we're using Kafka as the message >>queue for a stream processing service we're building, which we're >>delivering to our internal customers as a service along with OpenStack. >>We're considering building a high throughput ingest API to get the >>clients' data streams into the stream processing service. It occurs to >>me that this API is simply a messaging API, and so I'm wondering if we >>should consider building this high throughput API as part of the Marconi >>project. >> >>Has this topic come up in the Marconi team's discussions, and would it >>fit into the vision of the Marconi roadmap? > >Yes it has and I'm happy to see this coming up in the ML, thanks. > >Some things that we're considering in order to have a more flexible >architecture that will support a higher throughput are: > >- Queue Flavors (Terrible name). This is for marconi what flavors are > for Nova. It basically defines a set of properties that will belong > to a queue. Some of those properties may be related to the messages > lifetime or the storage capabilities (in-memory, freaking fast, > durable, etc). This is yet to be done. > >- 2 new drivers (AMQP, redis). The former adds support to brokers and > the later to well, redis, which brings in support for in-memory > queues. Work In Progress. > >- A new transport. This is something we've discussed but we haven't > reached an agreement yet on when this should be done nor what it > should be based on. The gist of this feature is adding support for > another protocol that can serve Marconi's API alongside the HTTP > one. We've considered TCP and websocket so far. The former is > perfect for lower level communications without the HTTP overhead > whereas the later is useful for web apps. > >That said. A Kafka plugin is something we heard a lot about at the >summit and we've discussed it a bit. I'd love to see that happening as >an external plugin for now. There's no need to wait for the rest to >happen. > >I'm more than happy to help with guidance and support on the repo >creation, driver structure etc. > >Cheers, >Flavio > >> >>Thanks, >>Keith Newstadt >>keith_newst...@symantec.com >>@knewstadt >> >> >>Date: Sun, 1 Jun 2014 15:01:40 +0000 >>From: "Hochmuth, Roland M" <roland.hochm...@hp.com> >>To: OpenStack List <openstack-dev@lists.openstack.org> >>Subject: Re: [openstack-dev] [Marconi] Kafka support and high >> throughput >>Message-ID: <cfae6524.762da%roland.hochm...@hp.com> >>Content-Type: text/plain; charset="us-ascii" >> >>There are some folks in HP evaluating different messaging technologies >>for >>Marconi, such as RabbitMQ and Kafka. I'll ping them and maybe they can >>share >>some information. >> >>On a related note, the Monitoring as a Service solution we are working >>on uses Kafka. This was just open-sourced at, >>https://github.com/hpcloud-mon, >>and will be moving over to StackForge starting next week. The >>architecture >>is at, >>https://github.com/hpcloud-mon/mon-arch. >> >>I haven't really looked at Marconi. If you are interested in >>throughput, low latency, durability, scale and fault-tolerance Kafka >>seems like a great choice. >> >>It has been also pointed out from various sources that possibly Kafka >>could be another oslo.messaging transport. Are you looking into that as >>that would be very interesting to me and something that is on my task >>list that I haven't gotten to yet. >> >> >>On 5/30/14, 7:03 AM, "Keith Newstadt" <keith_newst...@symantec.com> >>wrote: >> >>>Has anyone given thought to using Kafka to back Marconi? And has there >>>been discussion about adding high throughput APIs to Marconi. >>> >>>We're looking at providing Kafka as a messaging service for our >>>customers, in a scenario where throughput is a priority. We've had good >>>luck using both streaming HTTP interfaces and long poll interfaces to >>>get >>>high throughput for other web services we've built. Would this use case >>>be appropriate in the context of the Marconi roadmap? >>> >>>Thanks, >>>Keith Newstadt >>>keith_newst...@symantec.com >>> >> >> >> >> >>Keith Newstadt >>Cloud Services Architect >>Cloud Platform Engineering >>Symantec Corporation >>www.symantec.com >>________________________________ >> >>Office: (781) 530-2299 Mobile: (617) 513-1321 >>Email: keith_newst...@symantec.com >>Twitter: @knewstadt >>________________________________ >> >> >> >>This message (including any attachments) is intended only for the use of >>the individual or entity to which it is addressed and may contain >>information that is non-public, proprietary, privileged, confidential, >>and exempt from disclosure under applicable law or may constitute as >>attorney work product. If you are not the intended recipient, you are >>hereby notified that any use, dissemination, distribution, or copying of >>this communication is strictly prohibited. If you have received this >>communication in error, notify us immediately by telephone and (i) >>destroy this message if a facsimile or (ii) delete this message >>immediately if this is an electronic communication. >> >> >> >>_______________________________________________ >>OpenStack-dev mailing list >>OpenStack-dev@lists.openstack.org >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >-- >@flaper87 >Flavio Percoco _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
