Hi Flavio,
On 09/04/2014 08:08 AM, Flavio Percoco wrote:
- Concern on should we really reinvent a queue system rather than
piggyback on one
As mentioned in the meeting on Tuesday, Zaqar is not reinventing message
brokers. Zaqar provides a service akin to SQS from AWS with an OpenStack
flavor on
On 09/04/2014 09:44 PM, Kurt Griffiths wrote:
Thanks for your comments Gordon. I appreciate where you are coming from
and I think we are actually in agreement on a lot of things.
I just want to make it clear that from the very beginning of the project
the team has tried to communicate (but perha
On 09/04/2014 09:44 PM, Kurt Griffiths wrote:
Does a Qpid/Rabbit/Kafka provisioning service make sense? Probably.
I think something like that would be valuable, especially in conjunction
with some application layer proxying and mapping between 'virtual'
addresses/endpoints and specific queues
On 09/04/2014 01:14 PM, Sean Dague wrote:
https://dague.net/2014/08/26/openstack-as-layers/
Just wanted to say that I found this article very useful indeed and
agree with the points you make in it.
___
OpenStack-dev mailing list
OpenStack-dev@lists
On 09/10/2014 09:58 AM, Flavio Percoco wrote:
To clarify the doubts of what Zaqar is or it's not, let me quote what's
written in the project's overview section[0]:
"Zaqar is a multi-tenant cloud messaging service for web developers.
How are different tenants isolated from each other? C
On 09/10/2014 01:51 PM, Thierry Carrez wrote:
I think we do need, as Samuel puts it, "some sort of durable
message-broker/queue-server thing". It's a basic application building
block. Some claim it's THE basic application building block, more useful
than database provisioning. It's definitely a l
On 09/10/2014 12:47 AM, Devananda van der Veen wrote:
As was pointed out in the TC meeting today, Zaqar is (was?) actually
aiming to provide Messaging-as-a-Service -- not queueing as a service!
This is another way of saying "it's more like email and less like
AMQP"
The glossary[1] describes a m
On 09/10/2014 07:29 PM, Clint Byrum wrote:
Excerpts from Gordon Sim's message of 2014-09-10 06:18:52 -0700:
On 09/10/2014 09:58 AM, Flavio Percoco wrote:
Other OpenStack components can integrate with Zaqar to surface events
to end users and to communicate with guest agents that run in the
On 09/12/2014 09:50 AM, Flavio Percoco wrote:
Zaqar supports once and only once delivery.
For the transfer from Zaqar to consumers it does (providing the claim id
can be recovered). For transfer from producers to Zaqar I believe it is
more limited.
If the connection to Zaqar fails during a
On 09/11/2014 07:46 AM, Flavio Percoco wrote:
On 09/10/2014 03:18 PM, Gordon Sim wrote:
On 09/10/2014 09:58 AM, Flavio Percoco wrote:
Other OpenStack components can integrate with Zaqar to surface events
to end users and to communicate with guest agents that run in the
"over-cloud&q
On 09/16/2014 08:55 AM, Flavio Percoco wrote:
pub/sub doesn't necessarily guarantees messages delivery, it really
depends on the implementation.
As I understand it, the model for pub-sub in Zaqar is to have multiple
subscribers polling the queue with gets, and have the messages removed
from t
On 09/18/2014 12:31 PM, Flavio Percoco wrote:
On 09/17/2014 10:36 PM, Joe Gordon wrote:
My understanding of Zaqar is that it's like SQS. SQS uses distributed
queues, which have a few unusual properties [0]:
Message Order
Amazon SQS makes a best effort to preserve order in messages, but d
On 09/18/2014 03:45 PM, Flavio Percoco wrote:
On 09/18/2014 04:09 PM, Gordon Sim wrote:
Is the replication synchronous or asynchronous with respect to client
calls? E.g. will the response to a post of messages be returned only
once the replication of those messages is confirmed? Likewise when
What are the expected delivery guarantees for messages for rpc
transports in oslo.messaging?
Are failures to be handled by the applications using the rpc interface
(e.g. by retrying on timeout)? Or is the transport expected to provide
the necessary reliability of messages? Or is this something
The BaseDriver - which I take to be the API that must be fulfilled by
any transport implementation - has a listen() method taking a target,
but no way to stop listening on that specific target.
Is this because there is no need to ever cancel a subscription?
There is a cleanup() method on the B
On 11/18/2013 04:44 PM, Mark McLoughlin wrote:
On Mon, 2013-11-18 at 11:29 -0500, Doug Hellmann wrote:
IIRC, one of the concerns when oslo.messaging was split out was
maintaining support for existing deployments with configuration files that
worked with oslo.rpc. We had said that we would use UR
On 12/09/2013 04:10 PM, Russell Bryant wrote:
From looking it appears that RabbitMQ's support is via an experimental
plugin. I don't know any more about it. Has anyone looked at it in
detail?
I believe initial support was added in 3.1.0:
http://www.rabbitmq.com/release-notes/README-3.1.0.txt
On 12/09/2013 07:15 PM, Russell Bryant wrote:
On 12/09/2013 12:56 PM, Gordon Sim wrote:
In the case of Nova (and others that followed Nova's messaging
patterns), I firmly believe that for scaling reasons, we need to move
toward it becoming the norm to use peer-to-peer messaging for most
t
On 12/09/2013 10:37 PM, Russell Bryant wrote:
On 12/09/2013 05:16 PM, Gordon Sim wrote:
On 12/09/2013 07:15 PM, Russell Bryant wrote:
Understood. The Dispatch Router was indeed created from an understanding
of the limitations and drawbacks of the 'federation' feature of qpidd
(whi
On 12/09/2013 11:29 PM, Mark McLoughlin wrote:
On Mon, 2013-12-09 at 16:05 +0100, Flavio Percoco wrote:
Greetings,
As $subject mentions, I'd like to start discussing the support for
AMQP 1.0[0] in oslo.messaging. We already have rabbit and qpid drivers
for earlier (and different!) versions of A
On 12/10/2013 12:36 AM, Mike Wilson wrote:
This is the first time I've heard of the dispatch router, I'm really
excited now that I've looked at it a bit. Thx Gordon and Russell for
bringing this up. I'm very familiar with the scaling issues associated
with any kind of brokered messaging solution.
Hi Mark,
Thanks for the response! Some further followup inline...
On 12/09/2013 09:53 PM, Mark McLoughlin wrote:
It's not a completely unreasonable approach to take, but my thinking was
that a transport URL connects you to a conduit which multiple
applications could be sharing so you need the a
On 12/12/2013 02:14 PM, Flavio Percoco wrote:
I've a draft in my head of how the amqp 1.0 driver could be
implemented and how to map the current expectations of the messaging
layer to the new protocol.
I think a separate thread to discuss this mapping is worth it. There
are some critical areas t
On 12/16/2013 10:37 AM, Gordon Sim wrote:
On 12/12/2013 02:14 PM, Flavio Percoco wrote:
I've a draft in my head of how the amqp 1.0 driver could be
implemented and how to map the current expectations of the messaging
layer to the new protocol.
I think a separate thread to discuss this ma
On 12/17/2013 01:53 PM, Flavio Percoco wrote:
On 16/12/13 10:37 +, Gordon Sim wrote:
On 12/12/2013 02:14 PM, Flavio Percoco wrote:
I've a draft in my head of how the amqp 1.0 driver could be
implemented and how to map the current expectations of the messaging
layer to the new protoco
On 12/20/2013 05:27 PM, Herndon, John Luke wrote:
On Dec 20, 2013, at 8:48 AM, Julien Danjou
wrote:
Anyway, my main concern here is that I am not very enthusiast
about using the executor to do that. I wonder if there is not a way
to ask the broker to get as many as message as it has up to a
li
On 12/20/2013 07:13 PM, Gordon Sim wrote:
AMQP (in all it's versions) allows for a subscription with a
configurable amount of 'prefetch', which means the broker can send lots
of messages without waiting for the client to request them one at a time.
Just as an aside, the impl
On 12/20/2013 09:26 PM, Herndon, John Luke wrote:
On Dec 20, 2013, at 12:13 PM, Gordon Sim wrote:
On 12/20/2013 05:27 PM, Herndon, John Luke wrote:
Other protocols may support bulk consumption. My one concern with
this approach is error handling. Currently the executors treat
each
On 01/02/2014 10:46 PM, Herndon, John Luke wrote:
On 1/2/14, 11:36 AM, "Gordon Sim" wrote:
On 12/20/2013 09:26 PM, Herndon, John Luke wrote:
On Dec 20, 2013, at 12:13 PM, Gordon Sim wrote:
On 12/20/2013 05:27 PM, Herndon, John Luke wrote:
Other protocols may support bulk c
I have been working with Ken Giusti on an AMQP 1.0 messaging driver for
olso.messaging. The code for this is now available for review at
https://review.openstack.org/#/c/75815/
As previously mentioned on this list, this uses the Apache Qpid Proton
'protocol engine' library. This is not a standa
On 06/10/2014 09:48 AM, Mark McLoughlin wrote:
Perhaps the first point to get super clear on is why drivers for
traditional message brokers are needed. What problems would such drivers
address? Who would the drivers help? Would the Marconi team recommend
using any of those drivers for a productio
On 06/09/2014 08:31 PM, Kurt Griffiths wrote:
Lately we have been talking about writing drivers for traditional
message brokers that will not be able to support the message feeds part
of the API.
Could you elaborate a little on this point? In some sense of the term at
least, handling message f
On 06/10/2014 12:03 PM, Dina Belova wrote:
Hello, stackers!
Oslo.messaging is future of how different OpenStack components
communicate with each other, and really I’d love to start discussion
about how we can make this library even better then it’s now and how can
we refactor it make more produ
On 06/10/2014 05:27 PM, Kurt Griffiths wrote:
I think the crux of the issue is that Marconi follows the REST
architectural style. As such, the client must track the state of where it
is in the queue it is consuming (to keep the server stateless). So, it
must be given some kind of marker, allowing
On 06/10/2014 06:33 PM, Janczuk, Tomasz wrote:
From my perspective the key promise of Marconi is to provide a
*multi-tenant*,*HTTP* based queuing system. Think an OpenStack equivalent
of SQS or Azure Storage Queues.
As far as I know there are no off-the-shelve message brokers out these
that fi
On 06/10/2014 09:59 PM, Mark McLoughlin wrote:
Now why is there a desire to implement these requirements using
traditional message brokers?
I would speculate that any desire to utilise a message broker as opposed
to a database would be to achieve different performance characteristics
for the
On 06/10/2014 09:57 PM, Janczuk, Tomasz wrote:
> Using processes to isolate tenants is certainly possible. There is a range
> of isolation mechanisms that can be used, from VM level isolation
> (basically a separate deployment of the broker per-tenant), to process
> level isolation, to sub-process
On 06/11/2014 12:31 PM, Flavio Percoco wrote:
On 06/10/2014 09:59 PM, Mark McLoughlin wrote:
Now why is there a desire to implement these requirements using
traditional message brokers?
[...]
There are 2 main reasons that I don't believe are strong enough to
change the way Marconi works right
On 06/11/2014 01:28 PM, Flavio Percoco wrote:
There are three things that I consider really valuable:
1. Sharding support: It allows admins to have separate clusters/nodes
of stores and configure marconi to use them all based on
pre-configured rules. It's similar to what qpid-dispatch does but t
On 06/11/2014 06:05 PM, Janczuk, Tomasz wrote:
Process level isolation is more costly than sub-process level isolation
primarily due to larger memory consumption. For example, CGI has worse
cost characteristics than FastCGI when scaled out. But the example closer
to Marconi¹s use case is database
On 06/13/2014 02:06 PM, Ihar Hrachyshka wrote:
On 10/06/14 15:40, Alexei Kornienko wrote:
On 06/10/2014 03:59 PM, Gordon Sim wrote:
>>>
I think there could be a lot of work required to significantly
improve that driver, and I wonder if that would be better spent
on e.g. the AMQP 1
I have recently been trying to get some API level functional tests for
the olso.messaging library. The idea is that these tests could be run
with any driver and configured 'backend' and would test the basic
functional guarantees that the API makes.
I have an initial set of such tests now. The
I believe that ordering of notifications at different levels is not
guaranteed when receiving those notifications using a notification
listener in olso.messaging.
I.e. with something like:
notifier = notifier.Notifier(get_transport(CONF), 'compute')
notifier.info(ctxt, event_type, payl
On 03/31/2014 04:11 PM, Doug Hellmann wrote:
On Mon, Mar 31, 2014 at 8:54 AM, Gordon Sim mailto:g...@redhat.com>> wrote:
Another slight annoyance from the testing pov is that the API
provides no way of determining when a server is 'ready' i.e. when
its subscriptions
On 03/31/2014 03:36 PM, Sandy Walsh wrote:
If they're on different queues, the order they appear depends on the
consumer(s). It's not really an oslo.messaging issue.
Well, they are on two queues because that's how the oslo.messaging
notification listener is implemented. However...
[...]
I t
On 03/31/2014 04:41 PM, Doug Hellmann wrote:
On Mon, Mar 31, 2014 at 11:36 AM, Gordon Sim mailto:g...@redhat.com>> wrote:
On 03/31/2014 04:11 PM, Doug Hellmann wrote:
On Mon, Mar 31, 2014 at 8:54 AM, Gordon Sim mailto:g...@redhat.com>
<mailto:g...@redhat.c
On 03/31/2014 04:11 PM, Doug Hellmann wrote:
On Mon, Mar 31, 2014 at 8:54 AM, Gordon Sim mailto:g...@redhat.com>> wrote:
I have recently been trying to get some API level functional tests
for the olso.messaging library. The idea is that these tests could
be run with any
This may be a known and accepted fact, but it was a surprise to me so
thought it was worth a mention on this list.
If you use the transport_url config option (or pass a url directly to
get_transport()), then the amqp connection pooling appears to be
disabled[1]. That means a new connection is
On 09/19/2014 08:53 AM, Flavio Percoco wrote:
On 09/18/2014 09:25 PM, Gordon Sim wrote:
I don't see that the claim mechanism brings any stronger guarantee, it
just offers a competing consumer behaviour where browsing is
non-competing (non-destructive). In both cases you require the client
On 09/22/2014 08:00 AM, Flavio Percoco wrote:
oslo messaging is highly tight to AMQP semantics whereas Zaqar is not.
The API for oslo.messaging is actually quite high level and could easily
be mapped to different messaging technologies.
There is some terminology that comes from older version
On 09/22/2014 10:56 AM, Flavio Percoco wrote:
What I meant is that oslo.messaging is an rpc library and it depends on
few very specific message delivery patterns that are somehow tight/based
on AMQP semantics.
RPC at it's core is the request-response pattern which is directly
supported by many
On 09/19/2014 09:13 PM, Zane Bitter wrote:
SQS offers very, very limited guarantees, and it's clear that the reason
for that is to make it massively, massively scalable in the way that
e.g. S3 is scalable while also remaining comparably durable (S3 is
supposedly designed for 11 nines, BTW).
Zaqa
On 09/22/2014 03:40 PM, Geoff O'Callaghan wrote:
That being said I'm not sure why a well constructed zaqar with an rpc
interface couldn't meet the requirements of oslo.messsaging and much more.
What Zaqar is today and what it might become may of course be different
things but as it stands toda
On 09/22/2014 05:58 PM, Zane Bitter wrote:
On 22/09/14 10:11, Gordon Sim wrote:
As I understand it, pools don't help scaling a given queue since all the
messages for that queue must be in the same pool. At present traffic
through different Zaqar queues are essentially entirely ortho
Apologies in advance for possible repetition and pedantry...
On 09/24/2014 02:48 AM, Devananda van der Veen wrote:
2. Single Delivery - each message must be processed *exactly* once
Example: Using a queue to process votes. Every vote must be counted only
once.
It is also important to consi
On 09/24/2014 06:07 PM, Clint Byrum wrote:
I just wanted to commend Flavio Percoco and the Zaqar team for
maintaining poise and being excellent citizens of OpenStack whilst
being questioned intensely by the likes of me, and others.
+1
___
OpenStack-d
On 06/26/2014 09:38 PM, Alexei Kornienko wrote:
Benchmark for oslo.messaging is really simple:
You create a client that sends messages infinitively and a server that
processes them. After you can use rabbitmq management plugin to see
average throughput in the queue.
Simple example can be found he
A question about the new 'retry' option. The doc says:
By default, cast() and call() will block until the
message is successfully sent.
What does 'successfully sent' mean here? Does it mean 'written to the
wire' or 'accepted by the broker'?
For the impl_qpid.py driver, each send is sy
I do apologise for omitting the subject qualifiers in my previous mail!
On 06/27/2014 05:02 PM, Gordon Sim wrote:
A question about the new 'retry' option. The doc says:
By default, cast() and call() will block until the
message is successfully sent.
What does 'su
On 06/28/2014 10:49 PM, Mark McLoughlin wrote:
On Fri, 2014-06-27 at 17:02 +0100, Gordon Sim wrote:
A question about the new 'retry' option. The doc says:
By default, cast() and call() will block until the
message is successfully sent.
What does 'successfully
On 06/30/2014 12:22 PM, Ihar Hrachyshka wrote:
Alexei Kornienko wrote:
Some performance tests may be introduced but they would be more
like functional tests since they require setup of actual
messaging server (rabbit, etc.).
Yes. I think we already have some. F.e.
tests/drivers/test_impl_qpid.
On 07/01/2014 03:52 PM, Ihar Hrachyshka wrote:
On 01/07/14 15:55, Alexei Kornienko wrote:
And in addition I've provided some links to existing
implementation with places that IMHO cause bottlenecks. From my
point of view that code is doing obviously stupid things (like
closing/opening sockets fo
On 07/01/2014 04:14 PM, Alexei Kornienko wrote:
A lot of driver details leak outside the API and it makes it hard to
improve driver without changing the API.
I agree that some aspects of specific driver implementations leak into
the public API for the messaging library as a whole. There are al
On 07/03/2014 04:27 PM, Mark McLoughlin wrote:
Ceilometer's code is run in response to various I/O events like REST API
requests, RPC calls, notifications received, etc. We eventually want the
asyncio event loop to be what schedules Ceilometer's code in response to
these events. Right now, it is
On 07/07/2014 03:12 PM, Victor Stinner wrote:
The first step is to patch endpoints to add @trollius.coroutine to the methods,
and add yield From(...) on asynchronous tasks.
What are the 'endpoints' here? Are these internal to the oslo.messaging
library, or external to it?
Later we may modif
On 07/07/2014 09:53 PM, Victoria Martínez de la Cruz wrote:
During the last couple of weeks I've been working on adding support for
AMQP 1.0 in Marconi transport. For that, I've based my research on
current Marconi WSGI API, Apache Proton API [0] and Azure Service Bus
docs [1].
There are several
On 07/07/2014 07:18 PM, Mark McLoughlin wrote:
I'd expect us to add e.g.
@asyncio.coroutine
def call_async(self, ctxt, method, **kwargs):
...
to RPCClient. Perhaps we'd need to add an AsyncRPCClient in a separate
module and only add the method there - I don't have a good sense of i
On 07/28/2014 09:20 AM, Bogdan Dobrelya wrote:
Hello.
I'd like to bring your attention to major RPC failover issue in
impl_rabbit.py [0]. There are several *related* patches and a number of
concerns should be considered as well:
- Passive exchanges fix [1] (looks like the problem is much deeper t
On 07/07/2015 05:48 PM, Clint Byrum wrote:
all of the call sites I checked _do not appear to resend_, they
simply explode on timeout waiting for reply. This is how calling code
should work and I'm ok with code in nova, cinder, et. al. being
written this way, because I'd expect my messaging layer
On 05/27/2015 05:08 PM, Hayes, Graham wrote:
It was agreed that Zaqar would look at a oslo_messaging driver for the
services that did not agree with the 3 options presented.
Another option there would be to add amqp 1.0 support to zaqar, and then
you could use the amqp 1.0 driver with zaqar as
On 05/28/2015 05:14 PM, Fox, Kevin M wrote:
Thats like saying you should implement a sql engine inside of
Berkeley DB since so many folks like sql... Not a good fit. If you
want AMQP, get an AMQP server. Zaqar's intentionally lighter weight
then that. Hardly any OpenStack services use but a fract
On 05/28/2015 07:20 PM, Fox, Kevin M wrote:
I don't know AMQP very well. At the protocol level, are Exchanges, Queues, etc
things? Are fanout, durability, and other things required or optional to be
implemented?
AMQP 1.0 is very different from the pre 1.0 versions. Where 0.8/0.9
focus on def
On 06/12/2015 09:41 PM, Alec Hothan (ahothan) wrote:
One long standing issue I can see is the fact that the oslo messaging API
documentation is sorely lacking details on critical areas such as API
behavior during fault conditions, load conditions and scale conditions.
I very much agree, particu
On 06/16/2015 08:51 PM, Alec Hothan (ahothan) wrote:
I saw Sean Dague mention in another email that RabbitMQ is used by 95% of
OpenStack users - and therefore does it make sense to invest in ZMQ (legit
question).
I believe it's used by 95% of users because there is as yet no
compelling alterna
On 06/19/2015 07:14 AM, Flavio Percoco wrote:
The qpid family is a set of tools that provide messaging capabilities.
Among those you find qpidd (broker daemon), qpid-proton (amqp1.0
library), qpid-dispatch (message router). It's confusing indeed.
Apache Qpid is an Apache Software Foundation pro
On 01/27/2015 06:31 PM, Doug Hellmann wrote:
On Tue, Jan 27, 2015, at 12:28 PM, Denis Makogon wrote:
I'd like to build tool that would be able to profile messaging over
various deployments. This "tool" would give me an ability to compare
results of performance testing produced by native tools an
76 matches
Mail list logo
