Hello Camel Community,
I am in the process of developing an application to provide filtered change
data capture for a proprietary database. The initial plan is to use the
following stack:
* Spring Boot
* Debezium to create standardized messages from the WAL (we may use the app
for other dbs in the future)
* RxJava to efficiently process database change events based on the filter
criteria
* Apache Camel to forward events of interest to subscribers. This will
allow the application to forward events to a variety of endpoints.
In my POC app, I'm having trouble meeting performance requirements and Camel is
the bottleneck. For example, in one functional test suite I run two tests. Each
test generates 3000000 events for a relatively realistic db stream and pushes
them through the processing engine as fast as backpressure will allow. The
first test simply counts events as they exit the RxJava event processing
stream. The second test forwards events to Camel mock endpoints using the Camel
Reactive Streams Service. Processing time in the first test is ~1 second. The
second test takes > 1 minute. This is the only Difference:
var subscriber = camelRxService.streamSubscriber("camelMock",
DatabaseChangeMessage.class);
dataFlow.subscribe(subscriber);
And then I wait for all the messages to reach the mock endpoint. I've played
around with batching and other performance tweaks without much success. Is
there just too much overhead associated with Camel messages/exchanges for this
use case? I know it would be faster with Quarkus, but I'm stuck with Spring
Boot for now.
Any suggestions will be appreciated.
Thank you!
