Hi
I’ve been working with Camel for the last few month.
I have a program that processes several CVS files in order and inserts them
into a DB. Once they are all inserted I have to check for duplicate and create
new CVS files without duplicates.
That is all working fine but I have one big problem - a memory leak.
I’ve tracked it down to a enrich(“sql”) component.
I have removed everything after that enrich and just added a to(“mock:result”).
Once the CVS files have been processed and inserted into the DB I send an
exchange to
from("seda:check_duplicates?concurrentConsumers=4")
.log("Checking for duplicates on ${header.NOTIFY_BY} for event
${header.EVENT_ID} #{header.SFTP_DIRECTORY}")
.enrich("sql:select NOTIFICATION_ID, NOTIFY_BY, NOTIFY_BY_VALUE, LANGUAGE,
:#EVENT_ID as
EVENT_ID,PDP_SR_PARTICIPANTS_PARTICIPANT_ID,EVENT_DISPLAY_PREMISE_ADDR,
CSV_LINE, count(*) as c from PDP_SR_NOTIFICATIONS where NOTIFY_BY = :#NOTIFY_BY
and PDP_SR_EVENT_EVENT_ID = :#EVENT_ID group by
PDP_SR_PARTICIPANTS_PARTICIPANT_ID, NOTIFY_BY_VALUE, LANGUAGE;")
.to("mock:result");
Below is a screenshot of the memory consumption for two sets of CVS files. You
can see the jump in memory when the enrich(“sql”).
I would like to iterate over the result set (not load everything in one go as
it can be quite large but the useIterator=true does not seem to be supported
(at least I’ve tried) and the memory is never released again.
What am I doing wrong here? It is mainly chars in a HashMap that seems to be
taking up the memory which makes sense if the data is that of the sql query.
Regards
Immo
Dr. Immo Benjes
Senior Software Architect
Castle Court
41 London Road
Reigate
Surrey RH2 9RJ
United Kingdom
Tel: +44 (0)1737 735012
Web: www.locussoftware.com
Email: immoben...@locussoftware.com
