I implemented a JMS based solution about a year ago because I thought it
would solve my atomicity problem and give me a centralized way of
indexing, you'll have to use the pluggable persistence (if you use
ActiveMQ) to be able to recover from a failure and you'll also need some
way of maintaining which records have and haven't been indexed in a
persistent store so you can run sanitization later on, assuming that
every time you add a document to one of you clustered indecies
everything goes well is a sure way of guaranteeing that you end up with
n indecies with no two alike especially if you have a high volume
indecies with high rates of change.
Nader Henein
Stephane Bailliez wrote:
Hi Paul,
Thanks for the reply. Many interesting points.
Paul Smith wrote:
Why not try using JMS messaging to send messages to the indexing
server that Document X needs to be updated via a JMS queue? This
gives you the flexibility to have the indexing system down but not
lose the message that it needs to be indexed, and also allows the
indexing server to be 'busy' without affecting the application that
is performing the updates from slowing down too.
Excellent idea.
If you use ActiveMQ for JMS, you can take advantage of it's
Composite Destination feature and have a virtual Queue/Topic that is
actually several Queues/Topics. This is what we use to keep a
mirror index server completely in sync. The application sends an
update message to a queue named "queue://index1, queue://index2",
which becomes 2 separate queues for the 2 servers, allowing them to
process the same message whenever they can get around to it.
Ah, the composite topic, is indeed a good nice. But out of
curiosity...did you put your 2 nodes (consumers) as embedded brokers
or is the producer as the main broker ?
We then place Apache in front of these 2 mirrored Index/Search nodes
so the application can use web-services to query the search node
without actually being aware that there is 2 of them behind the
scenes, leaving Apache to do the load-balancing and fail-over as the
index/search nodes come up/down without the main application knowing
anything about it.
Ideally, the 2 nodes have the same state when running.
What happens when a node fails and that you put it back online and
that it needs to catch up with all missing messages in its queue ?
Is it considered 'offline' until it catches up ? If yes how do you do
it ? If no, I guess you don't mind that a search request may not give
the same result depending on the node it is load-balanced, correct ?
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---
--
Nader S. Henein
Senior Applications Architect
Bayt.com
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
