[
https://issues.apache.org/jira/browse/CAMEL-25607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on CAMEL-25607 started by shashank.
----------------------------------------
> camel-mongodb-gridfs - with the default TimeStamp query strategy (and
> PersistentTimestamp) the consumer routes every file uploaded since it started
> again at each poll, every 500 ms
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25607
> URL: https://issues.apache.org/jira/browse/CAMEL-25607
> Project: Camel
> Issue Type: Bug
> Components: camel-mongodb-gridfs
> Reporter: shashank
> Assignee: shashank
> Priority: Major
>
> {{GridFsConsumer}} with a timestamp query strategy
> ({{queryStrategy=TimeStamp}}, the default, and {{PersistentTimestamp}}) asks
> for the files whose {{uploadDate}} is greater than the date of the newest
> file it has routed. Since CAMEL-17121 (3.13, the loop was replaced by a
> repeated task) that date is lost between the polls ({{GridFsConsumer}} at
> main a7169d7c6ab6):
> {code:java}
> // run(), lines 128-133: the start date is captured once
> Date finalFromDate = fromDate;
> task.run(endpoint.getCamelContext(),
> () -> processCollection(finalFromDate, usesTimestamp,
> persistsTimestamp, usesAttribute, finalPtsCollection,
> finalPersistentTimestamp));
> // processCollection(Date fromDate, ...), lines 180-183: moves the parameter
> of this call only
> if (usesTimestamp && file.getUploadDate().compareTo(fromDate) > 0) {
> fromDate = file.getUploadDate();
> dateModified = true;
> }
> {code}
> Every poll therefore queries {{uploadDate > <date the consumer started>}}
> again. A file uploaded after the consumer started is routed by the first poll
> that sees it and then again by every following poll, every {{delay}} (500 ms
> by default), for as long as the route runs. Before CAMEL-17121 {{fromDate}}
> was a local variable of the loop and the file was routed once.
> Configuration: the defaults. {{from("mongodb-gridfs:myDb?database=test")}}
> and one file created in the bucket are enough; the route receives that file
> about twice per second.
> With {{PersistentTimestamp}} the effect is the same until the route is
> restarted (the stored timestamp is written again at every poll, with the same
> value; a restart then reads it and starts after the file). The strategies
> that use the file attribute ({{FileAttribute}},
> {{TimeStampAndFileAttribute}}, {{PersistentTimestampAndFileAttribute}}) are
> not affected: their query also asks for files without the attribute.
> The integration test of the module ({{GridFsConsumerIT}}) does not see it:
> all files have the same body, and the mock endpoint is asserted as soon as
> the expected number of messages has arrived, so a file that is routed again a
> poll later is either not seen or looks like the next expected message.
> CAMEL-24905 (flaky {{GridFsConsumerIT.testAttribute}}, "Expected: <1> but
> was: <2>") is possibly related, not reproduced: the test that failed there
> uses the {{FileAttribute}} strategy, which is not affected, and the root
> causes found in that ticket were the shared mock endpoint and buckets that
> were not cleaned between the tests.
> h3. Reproduction
> New {{GridFsConsumerTimestampTest}} (3 tests, no MongoDB needed): the real
> {{GridFsConsumer}} polls a fake bucket (Mockito) that answers the
> {{uploadDate > date}} query of each poll from a list of files, the way the
> server does. All three fail with the consumer of main:
> * {{timestampStrategyRoutesAFileOnce}}: {{three polls saw the file, it must
> be routed by the first of them only ==> expected: <[first]> but was: <[first,
> first, first, first]>}} (three or four times, depending on the poll that is
> running when the test asserts)
> * {{timestampStrategyRoutesALaterFileOnce}}: {{expected: <[first, second]>
> but was: <[first, first, first, first, second, first, second, first,
> second]>}}
> * {{persistentTimestampStrategyRoutesAFileOnce}}: {{expected: <[first]> but
> was: <[first, first, first]>}}
> h3. Proposed fix
> Keep the date in a field of the consumer, so that a poll starts from the date
> the previous poll reached. The field is {{volatile}}: {{run()}} sets the
> start value on the thread of the consumer's {{executor}} before it schedules
> the polls, and the polls read and move it on the thread of the
> {{taskExecutor}}. Nothing else changes.
> A stop and start of the consumer is as before: {{doStart}} runs {{run()}}
> again, which sets the date to "now" for {{TimeStamp}} and reads the stored
> timestamp for {{PersistentTimestamp}}.
> Behaviour change, with a 4.24 upgrade-guide entry: a file is routed once.
> Not changed, seen in the same method: a failed exchange still moves the
> timestamp (and sets the file attribute to {{done}}), an exception thrown by
> {{process}} is still swallowed, and an exception of the poll itself (MongoDB
> not reachable) still ends the repeated task. These were the same before
> CAMEL-17121 or are a different defect, and would be separate tickets.
> camel-mongodb-gridfs tests with the fix: 3 run, 0 failures (the module has no
> other unit test) (the integration tests need a MongoDB container and were not
> run).
> How it was found: a small TLA+ model of the polls (upload, query, route, end
> of poll). For the code of main TLC reports a violation of "a file is routed
> at most once" after two polls (7 states) and of "the timestamp is stored only
> when it moved"; with the date kept between the polls both hold, and no file
> is skipped. The model is a sketch of the code; the evidence is the test above.
> Affected: main (a7169d7c6ab6, tested). Read, not run: camel-4.23.x
> (197e70656a41), camel-4.22.x (8dfae2fa437c), camel-4.18.x (c3fd5317a5a0),
> camel-4.14.x, camel-3.14.x and the camel-3.13.0 tag have the same lines
> ({{finalFromDate}} captured by the task, {{processCollection(Date fromDate,
> ...)}}), so every release since 3.13.0. The camel-3.12.0 tag still has the
> loop with a local variable.
> Duplicate check (2026-10-13): JIRA text "GridFsConsumer" (3: CAMEL-24286,
> CAMEL-24272, CAMEL-17121, none about duplicates), "gridfs consumer" (3),
> "gridfs duplicate" (2), "gridfs timestamp" (1), "mongodb-gridfs" (26, none
> about the consumer routing a file more than once); GitHub pull requests
> "gridfs": only #26787 (CAMEL-24905, test only) touches the consumer tests.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
