Hi all, we currently consider the use of JMS implemented by ActiveMQ in the following scenario:
A producer will produce roughly 300 to 400 rather large (20-30 MB) data objects per minute (in total ~17000 objects). Seven different processor-objects process the data objects in parallel. The setting is purely local, so the embedded message-broker is sufficient. Durable messages are not required. Naively you would assume, that the best setup will be one non-durable topic, where the seven processors register as subscribers and the producer sending the data objects to that topic. Everything within this setting is purely java. Our problem is the memory consumption of the setup. There is a limit of roughly 100 of the data objects, which can be kept in memory at a given time. Typically, the seven processors are roughly equally fast but there may be situations, where one is lagging behind. In this case, the remaining fast processors should proceed without decreasing speed, while for the lagging processor, it is acceptable to complete its task with a delay. Until now we have identified several options how to deal with this problem and neither makes us really happy: 1) Intelligent data objects: The data objects get some intelligence and time to live. Once the time to live is up, the objects starts persisting the larger data chunks contained, thus freeing memory. Of course this solution requires some rematerialization, once the object is needed again by the lagging processor. This of course requires some tedious implementation (plus the question as to how the data object gets notified, that it should persist itself without interfering with gc). 2) (Ab)Use of JMS techniques: Try to achieve the following: Via priority assure, that newest messages (and objects) are consumed first. If an old object gets too old (or better: queue gets to long), the object is subject to persisting via the JMS mechanisms. Question: is this a violation of the design principles? Is it technically even possible? 3) Re-Aquiring data objects: Use the ActiveMQ mechanism to drop old messages, via a second mechanism (simple java communication) re-request the data-objects that were lost afterwards. This of course minimizes our chances to ever make use of the distributed mechanisms of JMS and seems to involve a lot of manual work. 4) Central observer: Use a central observer to notify about consumed messages and filled queues. Controll persisting/dropping/redelivery of messages via the central observer. Again: lots of manual work. All in all our opinion is, that the requirements are somewhat standard and we have the feeling, we might be missing some plain nice and easy way to make use of standard-JMS-techniques (or ActiveMQ enhancements to the standard). Has someone some idea where to start looking or some input to the possibilities we already identified? Kind regards, Jonathan -- View this message in context: http://activemq.2283324.n4.nabble.com/Question-concerning-JMS-ActiveMQ-Design-tp3645904p3645904.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
