On Thu, Sep 18, 2014 at 7:45 AM, Flavio Percoco <fla...@redhat.com> wrote:
> On 09/18/2014 04:09 PM, Gordon Sim wrote: > > On 09/18/2014 12:31 PM, Flavio Percoco wrote: > >> On 09/17/2014 10:36 PM, Joe Gordon wrote: > >>> My understanding of Zaqar is that it's like SQS. SQS uses distributed > >>> queues, which have a few unusual properties [0]: > >>> > >>> > >>> Message Order > >>> > >>> Amazon SQS makes a best effort to preserve order in messages, but due > to > >>> the distributed nature of the queue, we cannot guarantee you will > >>> receive messages in the exact order you sent them. If your system > >>> requires that order be preserved, we recommend you place sequencing > >>> information in each message so you can reorder the messages upon > >>> receipt. > >>> > >> > >> Zaqar guarantees FIFO. To be more precise, it does that relying on the > >> storage backend ability to do so as well. Depending on the storage used, > >> guaranteeing FIFO may have some performance penalties. > > > > Would it be accurate to say that at present Zaqar does not use > > distributed queues, but holds all queue data in a storage mechanism of > > some form which may internally distribute that data among servers but > > provides Zaqar with a consistent data model of some form? > > I think this is accurate. The queue's distribution depends on the > storage ability to do so and deployers will be able to choose what > storage works best for them based on this as well. I'm not sure how > useful this separation is from a user perspective but I do see the > relevance when it comes to implementation details and deployments. > > > > [...] > >> As of now, Zaqar fully relies on the storage replication/clustering > >> capabilities to provide data consistency, availability and fault > >> tolerance. > > > > Is the replication synchronous or asynchronous with respect to client > > calls? E.g. will the response to a post of messages be returned only > > once the replication of those messages is confirmed? Likewise when > > deleting a message, is the response only returned when replicas of the > > message are deleted? > > It depends on the driver implementation and/or storage configuration. > For example, in the mongodb driver, we use the default write concern > called "acknowledged". This means that as soon as the message gets to > the master node (note it's not written on disk yet nor replicated) zaqar > will receive a confirmation and then send the response back to the > client. This is also configurable by the deployer by changing the > default write concern in the mongodb uri using `?w=SOME_WRITE_CONCERN`[0]. > This means that by default Zaqar cannot guarantee a message will be delivered at all. A message can be acknowledged and then the 'master node' crashes and the message is lost. Zaqar's ability to guarantee delivery is limited by the reliability of a single node, while something like swift can only loose a piece of data if 3 machines crash at the same time. > > [0] http://docs.mongodb.org/manual/reference/connection-string/#uri.w > > > > >> However, as far as consuming messages is concerned, it can > >> guarantee once-and-only-once and/or at-least-once delivery depending on > >> the message pattern used to consume messages. Using pop or claims > >> guarantees the former whereas streaming messages out of Zaqar guarantees > >> the later. > > > > From what I can see, pop provides unreliable delivery (i.e. its similar > > to no-ack). If the delete call using pop fails while sending back the > > response, the messages are removed but didn't get to the client. > > Correct, pop works like no-ack. If you want to have pop+ack, it is > possible to claim just 1 message and then delete it. > > > > > What do you mean by 'streaming messages'? > > I'm sorry, that went out wrong. I had the "browsability" term in my head > and went with something even worse. By streaming messages I meant > polling messages without claiming them. In other words, at-least-once is > guaranteed by default, whereas once-and-only-once is guaranteed just if > claims are used. > > > > > [...] > >> Based on our short conversation on IRC last night, I understand you're > >> concerned that FIFO may result in performance issues. That's a valid > >> concern and I think the right answer is that it depends on the storage. > >> If the storage has a built-in FIFO guarantee then there's nothing Zaqar > >> needs to do there. In the other hand, if the storage does not have a > >> built-in support for FIFO, Zaqar will cover it in the driver > >> implementation. In the mongodb driver, each message has a marker that is > >> used to guarantee FIFO. > > > > That marker is a sequence number of some kind that is used to provide > > ordering to queries? Is it generated by the database itself? > > It's a sequence number to provide ordering to queries, correct. > Depending on the driver, it may be generated by Zaqar or the database. > In mongodb's case it's generated by Zaqar[0]. > > [0] > > https://github.com/openstack/zaqar/blob/master/zaqar/queues/storage/mongodb/queues.py#L103-L185 > > -- > @flaper87 > Flavio Percoco > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
