[
https://issues.apache.org/jira/browse/CAMEL-25575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen resolved CAMEL-25575.
---------------------------------
Resolution: Fixed
Fixed by https://github.com/apache/camel/pull/27759 (merged to main as
75ddf4853b21, 4.24.0).
_Claude Code on behalf of Claus Ibsen_
> camel-dropbox - search returns only the first page of the matches (100 by
> default) although its javadoc promises all files found; the rest is dropped
> silently (get of a folder: first page of the listing)
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25575
> URL: https://issues.apache.org/jira/browse/CAMEL-25575
> Project: Camel
> Issue Type: Bug
> Components: camel-dropbox
> Reporter: shashank
> Assignee: shashank
> Priority: Major
> Fix For: 4.24.0
>
>
> *{{search}}* is the main case. {{DropboxAPIFacade.search}} (lines 245-259 at
> main c3829069cb72), used by the {{dropbox:search}} producer and consumer:
> {code:java}
> SearchV2Result listing = client.files().searchV2Builder(query)
> .withOptions(SearchOptions.newBuilder().withPath(remotePath).build())
> .start();
> return new DropboxSearchResult(listing.getMatches());
> {code}
> The request does not set {{max_results}}, and {{SearchOptions}} of
> dropbox-core-sdk 7.1.1 uses 100 then (the constant in its no-argument
> constructor, read in the jar). {{SearchV2Result}} has {{getHasMore()}} and
> {{getCursor()}} for the next page ({{searchContinueV2(cursor)}}); the facade
> reads neither. So with the default options a search with more matches than
> one page returns the first page only: the body, and the {{FOUND_FILES}}
> header that is built from it, lack the rest. Nothing is logged and the
> exchange succeeds. The javadoc of the method says "@return a result object
> containing all the files found." (line 242), and the component has no option
> or header to ask for the next page, so a route cannot get the rest.
> *{{get}} of a folder* is the lesser case ({{downloadFilesInFolder}}, lines
> 307-318), used by the {{dropbox:get}} producer and consumer:
> {code:java}
> ListFolderResult folderResult = client.files().listFolder(path.equals("/") ?
> "" : path);
> for (Metadata entry : folderResult.getEntries()) { ... download ... }
> {code}
> {{ListFolderResult}} has {{getHasMore()}} and {{getCursor()}} as well
> ({{listFolderContinue(cursor)}}). The facade downloads the files of the first
> response only, although the javadoc says "Get the content of every file
> inside the remote path" (line 296). It needs a folder larger than one page of
> the listing; that size is decided by the service (the request sets no limit)
> and I did not measure it. {{get}} keeps the content of every file in memory,
> so it is not the operation for a very large folder anyway.
> Not verified (nothing was run against Dropbox): the page size the service
> really uses for a search (100 is what the SDK asks for), whether the service
> limits the number of matches a search can return in total, and the page size
> of a folder listing.
> h3. Reproduction
> New {{DropboxAPIFacadePagingTest}} (3 tests): the Dropbox client is a Mockito
> mock that answers a search and a folder listing in three pages (page,
> {{has_more}}, cursor; the fake fails the test after 10 continue requests). On
> main 2 tests fail:
> * {{searchReturnsTheMatchesOfEveryPage}}: {{the matches of the three pages,
> each once and in order ==> expected: <[/docs/report-1.txt,
> /docs/report-2.txt, /docs/report-3.txt, /docs/report-4.txt,
> /docs/report-5.txt]> but was: <[/docs/report-1.txt, /docs/report-2.txt]>}}.
> * {{getDownloadsTheFilesOfEveryPageOfTheFolder}}: {{the files of the three
> pages, each once and in order ==> expected: <[/docs/a.txt, /docs/b.txt,
> /docs/c.txt, /docs/d.txt]> but was: <[/docs/a.txt, /docs/b.txt]>}}.
> The third, {{resultThatFitsInOnePageNeedsNoFurtherRequest}}, passes on main
> as well; it is a guard that a result of one page sends no continue request (a
> folder listing returns a cursor with its last page too).
> h3. Proposed fix
> In {{DropboxAPIFacade}}: {{search}} calls {{searchContinueV2}} with the
> cursor of the response while {{getHasMore()}} is true (and a cursor is
> present) and returns the matches of all pages; {{downloadFilesInFolder}}
> calls {{listFolderContinue}} the same way and downloads the files of each
> page. The order is the order of the service. {{mockito-core}} is added to the
> module as a test dependency.
> Behaviour change, with a 4.24 upgrade-guide entry: a search returns the
> matches of every page, and {{get}} of a large folder downloads all its files.
> Consequences that a user can notice:
> * a search with many matches sends one request per page instead of one
> request; the search consumer does so at every poll;
> * when a request for a later page fails, the operation fails and the matches
> that were already read are not returned. That is as before for a failed
> search, but there are now more requests that can fail;
> * the number of pages is decided by the service ({{has_more}}); the facade
> has no limit of its own;
> * {{get}} keeps the content of all files in the message body, as before, so a
> large folder needs more memory than when only the first page was read.
> No new option. camel-dropbox unit tests with the fix: 4, 0 failures (the
> integration tests need a Dropbox account and were not run).
> Found with a Lean 4 model of a token-paged service: the collection is a list,
> a response is a {{take}} / {{drop}} slice of it plus a token when items
> remain. Property: the result is the collection, every item once and in order.
> Counterexample for "first page only": with a page of two, {{[1, 2, 3]}} gives
> {{[1, 2]}} ({{cex_third_match_dropped}}); with a page of 100, 101 matches
> give 100 ({{cex_101_matches}}); {{main_complete_iff}}: the first page is the
> whole result exactly when the result fits in one page. In the model,
> following the token returns the whole list ({{fix_meets_spec}}) and equals
> main's result when it fits in one page ({{fix_eq_main_one_page}}). The model
> says nothing about the real service. Then confirmed with the real class as
> described.
> Affected: main (4.23.0-SNAPSHOT at c3829069cb72), and the release branches
> camel-4.18.x (c3fd5317a5a0), camel-4.22.x (2821f284056c) and camel-4.23.x
> (197e70656a41): {{DropboxAPIFacade}} has the same lines on all three (javadoc
> at 242 and 296, {{searchV2Builder}} at 252, {{listFolder}} at 309, no
> {{getHasMore}} / {{getCursor}} / continue call in the file). Read, not run.
> Duplicate check (2026-10-12): JIRA text "dropbox search has_more" (2,
> unrelated: CAMEL-13025, CAMEL-12955), "dropbox search results truncated" (1,
> unrelated: CAMEL-12955), "dropbox listFolder" (none); GitHub pull requests
> "dropbox search": none about the result of a search; no open pull request
> touches camel-dropbox.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
