When you already have a JMS broker then you can sure test it out. When not
you need to sure set up (best fault tolerant), this complicate things.
>From my experience, one can use Camel to move the data/commands in
parallel. Use the integration patterns to route, transform the data (Camel
is very fast and powerful in this area), but I put the executing and
conflict handling to a separate application/service. When using queues this
application/service can read from it when it needs (Kafka is also good for
this). When the command is executed and it needs to send something back,
this could be a separate integration. In this way parallel and async are
optimized. I've seen all-in-one solutions with a lot more more issues then
when it is divided into separate (loosely-coupled) simple integrations and
services. Less is more.
Raymond
On Thu, Sep 8, 2022 at 4:46 PM Mansour Al Akeel <mansour.alak...@gmail.com>
wrote:
> Raymond,
>
> I am not sure how to simplify this. I have multiple clients, sending
> commands. Commands that act on the same object, may conflict. I need to
> orchestrate the execution of these commands to:
>
> - avoid two commands to execute for the same object-id,
> - ensure ordering are preserved, as commands from the same client depends
> on the execution of the previous command.
> - ensure parallelism.
>
> I don't see how this is a complex requirement and requires more
> simplification, but I would love to hear other opinions.
> As Claus suggested, to use jms-group might be a good idea, and use cases
> described in the page he refers to are very similar.
>
> If you have an idea to simplify this and remove the smell, please share.
>
>
>
> On Thu, Sep 8, 2022 at 4:20 AM ski n <raymondmees...@gmail.com> wrote:
>
> > Hi Mansour,
> >
> > Your use case sounds a bit complicated. On one hand you need parallel
> > processing, on the other hand you need blocks/sequential processing. This
> > may have a bad smell, but I don't know the details of your use case. I
> > would say try to come up with a way for real parallel processing where
> > ordering doesn't matter. At the end this will make it both simpler and
> > faster.
> >
> > Based on your use case, here are some things you could look at:
> >
> > 1) A dynamic endpoint of 1
> >
> > toD("vm:${header.groupId}?queueSize=1&concurrentConsumers=1&
> > defaultBlockWhenFull=true")
> >
> > The question is when this is really dynamic who will listen to this
> > endpoint?
> >
> > https://camel.apache.org/components/3.18.x/vm-component.html
> >
> > 2) Some other constructs:
> >
> > https://camel.apache.org/manual/try-catch-finally.html
> >
> > https://camel.apache.org/components/3.18.x/eips/throttle-eip.html
> >
> > https://camel.apache.org/components/3.18.x/eips/recipientList-eip.html
> (It
> > has parallel processing)
> >
> > https://camel.apache.org/components/3.18.x/eips/split-eip.html (It has
> > parralel processing)
> >
> > But as said combinations of above are not obvious. A throttle can be used
> > for example when an external API has rate limiting, but for internal use
> in
> > Camel I try to avoid it. You may eloborate a bit about what you trying to
> > achieve functionally.
> >
> > Raymond
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Sep 8, 2022 at 4:07 AM Mansour Al Akeel <
> mansour.alak...@gmail.com
> > >
> > wrote:
> >
> > > I have a stream of commands that needs to be processed in parallel for
> > > performance reasons, and to be grouped on a key for sequential
> processing
> > > to avoid conflicts.
> > >
> > > I don't know the number of groups in advance. One way that came to my
> > mind,
> > > is dynamic channels/queues. So the stream of commands is sorted into
> > > queues, where each queue represents a group. Then handle those
> > > sequentially.
> > >
> > > I have been looking at toD, and dynamic router, but no luck.
> > > I am hoping to be able to achieve something like:
> > >
> > > from("direct:commands")
> > > .to("direct:${header.groupId})
> > > .process(new MyProcessor())
> > >
> > >
> > > Therefore processing commands in parallel, across all queues, while
> > > blocking on the same queue.
> > > If an error occurs, during the process of a command, I would like to be
> > > able to flush/return the failed, and all the pending ones. In other
> > words,
> > > empty the queue.
> > >
> > >
> > > How to achieve this ?
> > > I am open for advice about better ideas/setup.
> > >
> >
>
