the problem with this strategy is that queues can come and go.. it’s a scheduler system for a web crawler. so if a URL request comes in for an IP we place it in that queue. And if, a month from now, we have to request another resources from that IP, we place it in a queue for that IP.
MOST of the time, we just have a few IPs because most web properties are large. However, sometimes, we have a request once per hour to an IP. In that situation we want to GC the queue 60 seconds later. Kevin On Mon, Feb 2, 2015 at 11:17 AM, Tim Bain <tb...@alumni.duke.edu> wrote: > It sounds like the advisory message you mentioned should let you work > around the problem. > > But what it sounds like you really need is a way to know, for sure, that > the producer is finished using a queue. If your producers write an EOF > message when they're through, your consumer can then safely go away without > having to worry about stragglers. Then your only remaining problem is how > to know about producers that die unexpectedly without writing the EOF or > that take a really long time to producer their next message. A heartbeat > mechanism would solve those issues: every X amount of time, the producer > produces a small no-op heartbeat message that tells the consumer it's still > there; failure to receive one of those messages after N heartbeat intervals > indicates that the producer is no longer alive. The only thing it doesn't > protect you against is temporary network perturbations (since they would > make it appear that the producer is gone when it's not really), but you > might be able to live with that. > > I'm not sure that heartbeats are a better solution than processing the > advisory messages as you've suggested, but I think at a minimum you should > have an EOF that's always sent (during normal operations) when the producer > closes, and to which your consumer reacts by disconnecting as well. Then > the advisory messages can be used only for handling truly exceptional > conditions. > > On Mon, Feb 2, 2015 at 11:29 AM, Kevin Burton <bur...@spinn3r.com> wrote: > > > > ActiveMQ provides no means to guarantee that a producer will fail to > > > deliver > > > a message when there are no consumers interested in the message. It is > > > actually designed more for cases of consumers being away and returning > > at a > > > later time. > > > > > > > > Perhaps. But I think the advisory queue message > > > > ActiveMQ.Advisory.NoConsumer.Queue > > > > … actually solves my problem. > > > > If a producer creates a queue, and no one is consuming, I’ll just create > a > > new consumer once I get the advisory message about the queue. > > > > This is actually how I do it *now* except I was listening to NEW queues… > > > > I’ve actually gotten a lot of use out of the advisory queue system. > > > > > > > As you noted correctly, ActiveMQ automatically recreates destinations > > every > > > time they are needed, so even if they were GC'ed, they would just be > > > automatically recreated when a producer posted a message. > > > > > > > Yes. And in my case, I think that’s ok. > > > > I might actually write up a blog post about this design. It’s novel but > AMQ > > actually does seem to be working out well for us without the limitations > > you describe due to advisory messages. > > > > Kevin > > > > -- > > > > Founder/CEO Spinn3r.com > > Location: *San Francisco, CA* > > blog: http://burtonator.wordpress.com > > … or check out my Google+ profile > > <https://plus.google.com/102718274791889610666/posts> > > <http://spinn3r.com> > > > -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile <https://plus.google.com/102718274791889610666/posts> <http://spinn3r.com>
