[
https://issues.apache.org/jira/browse/CAMEL-25548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen resolved CAMEL-25548.
---------------------------------
Resolution: Fixed
Fixed by https://github.com/apache/camel/pull/27695 (merged to main as
7d6bab0ccabe, 4.24.0).
_Claude Code on behalf of Claus Ibsen_
> camel-twitter - twitter-search ignores filterOld, sinceId, lang, count and
> the geolocation options since the Twitter4J 4.1 upgrade, so the consumer
> routes the same tweets again at every poll; a twitter-timeline poll sends its
> request twice
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25548
> URL: https://issues.apache.org/jira/browse/CAMEL-25548
> Project: Camel
> Issue Type: Bug
> Components: camel-twitter
> Reporter: shashank
> Assignee: shashank
> Priority: Major
> Fix For: 4.24.0
>
>
> Since Twitter4J 4.1 a {{twitter4j.v1.Query}} is immutable: {{sinceId}},
> {{maxId}}, {{lang}}, {{count}} and {{geoCode}} return a new {{Query}}. When
> Camel 4.0.0 moved to Twitter4J 4.1.2, the setters in
> {{SearchConsumerHandler}} and {{SearchProducer}} were renamed, but the
> results are not used (lines at main 9432ba16d026: {{SearchConsumerHandler}}
> 66, 88, 92, 103; {{SearchProducer}} 66, 75, 81, 91, 100):
> {code:java}
> Query query = Query.of(keywords);
> if (endpoint.getProperties().isFilterOld()) {
> query.sinceId(getLastId()); // returns a new Query, which is dropped
> }
> ...
> query.lang(endpoint.getProperties().getLang());
> query.count(endpoint.getProperties().getCount());
> query.geoCode(location, radius, unit);
> QueryResult qr = twitter.v1().search().search(query); // only the keywords
> {code}
> So the query that is sent has only the keywords:
> * The consumer ignores {{filterOld}} (default {{true}}): every poll (default
> every 30 seconds) gets the most recent tweets of the search again and routes
> them again. Nothing is logged.
> * The consumer ignores {{lang}}, {{count}} (default 5; Twitter's default of
> 15 applies) and {{latitude}}/{{longitude}}/{{radius}}/{{distanceMetric}}.
> * The producer ignores {{sinceId}}, {{lang}}, {{count}} and the headers
> {{CamelTwitterSinceId}}, {{CamelTwitterMaxId}},
> {{CamelTwitterSearchLanguage}} and {{CamelTwitterCount}}.
> A second defect in the same class, from a later cleanup (c261528d20, 4.1.0):
> {{SearchConsumerHandler.directConsume}} returns an empty list when keywords
> are given, because the blank check was inverted ({{keywords == null ||
> !keywords.isBlank()}}), so a consumer with {{type=direct}} never routes
> anything at startup.
> A third defect in the same module, in the timeline consumers (since
> CAMEL-15651, which introduced the class):
> {{AbstractStatusConsumerHandler.pollConsume}} (lines 38-45), used by the
> home, mentions, user and list timelines, calls {{doPoll()}}, the request to
> Twitter, twice:
> {code:java}
> List<Status> statusList = doPoll();
> for (Status status : doPoll()) {
> setLastIdIfGreater(status.getId());
> }
> return TwitterEventType.STATUS.createExchangeList(endpoint, statusList);
> {code}
> The tweets of the first response are routed, and the second response is only
> used for the id of the last tweet, which the next poll sends as {{since_id}}.
> Every poll sends two requests where one is enough (the home timeline allows
> 15 requests per 15 minutes), and a tweet posted between the two requests is
> only in the second response, so it is skipped.
> h3. Reproduction
> New {{TwitterSearchQueryOptionsTest}} (no network: a mocked {{Twitter}}
> client is set on the endpoint with {{TwitterConfiguration.setTwitter}}; its
> search records the query and returns the tweets with an id above {{since_id}}
> and up to {{max_id}}, as the search API does; the consumer is created with
> {{startScheduler=false}} and polled directly). On main, twice, five of the
> six tests fail:
> {noformat}
> consumerPollsOnlyTheTweetsNewerThanTheLastOne: The second poll must not route
> the same tweets again ==> expected: <[10, 11]> but was: <[10, 11, 10, 11]>
> consumerSendsLanguageCountAndGeolocation: expected: <en> but was: <null>
> directConsumerSearchesTheKeywordsAtStartup: expected: <[10, 11]> but was:
> <[]>
> producerSendsSinceIdMaxIdLanguageAndCountOfTheHeaders: expected: <[11, 12]>
> but was: <[10, 11, 12, 13]>
> producerSendsTheOptionsOfTheEndpoint: expected: <1> but was: <3>
> {noformat}
> The sixth test ({{filterOld=false}} polls the tweets again) passes on main
> too.
> New {{TwitterTimelinePollTest}} (same setup; the home timeline of the mocked
> client returns the tweets above {{since_id}}). On main both tests fail:
> {noformat}
> pollSendsOneRequest: A poll must send one request to the timeline ==>
> expected: <1> but was: <2>
> tweetPostedDuringAPollIsRoutedByTheNextPoll: The tweet posted during the
> first poll must not be skipped ==> expected: <[10, 11]> but was: <[10]>
> {noformat}
> h3. Proposed fix
> Keep the {{Query}} that each call returns ({{query = query.sinceId(...)}} and
> so on), in the consumer handler and in the producer, and correct the blank
> check of {{directConsume}}. The options then work as documented and as in
> Camel 3.x. Visible to users: a poll routes only tweets newer than the last
> one consumed, and a page has {{count}} tweets (default 5), so the 4.23
> upgrade guide has a note (raise {{count}} and {{numberOfPages}} for a search
> that matches more tweets between two polls). In
> {{AbstractStatusConsumerHandler.pollConsume}} the loop uses the list of the
> first, and now only, request (no upgrade note: same tweets, one request per
> poll). camel-twitter with the fix: 14 unit tests, 0 failures; the integration
> tests need Twitter credentials and were not run. Test dependency added:
> mockito-junit-jupiter.
> Not changed here: in the producer, {{filterOld}} is overridden by {{sinceId}}
> (default 1) as it was in Camel 3.x; and
> {{AbstractTwitterConsumerHandler.getLastIdPaging}} passes {{numberOfPages}}
> to {{Paging.ofPage}}, so {{numberOfPages=3}} requests the third page only
> (the default of 1 is not affected).
> Found with a search for calls whose result is dropped on immutable types, and
> a small Lean 4 model of the poll (the matching tweets as a list of ids, the
> API as "ids above since_id and up to max_id"): on main a second poll with no
> new tweet is proved to route all the tweets again, for every list of tweets;
> with the fix it is proved to route nothing, and a poll routes exactly the
> tweets above the last id. Confirmed with the real consumer and producer.
> Affected: main, camel-4.22.x, camel-4.18.x, camel-4.14.x (same code, GitHub
> contents API); the query options since 4.0.0, the {{type=direct}} check since
> 4.1.0, the double timeline request in all of them.
> Duplicate check (2026-10-11): JIRA text "twitter" since 2023-06 (6, none
> about search), "twitter-search" (30, none since the 4.0 upgrade), "twitter4j"
> (22, none after 2022); GitHub pull requests with "twitter" in the title: none
> open. No open pull request changes {{SearchConsumerHandler}},
> {{SearchProducer}} or {{AbstractStatusConsumerHandler}}.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
