[
https://issues.apache.org/jira/browse/CAMEL-25536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18127131#comment-18127131
]
Federico Mariani commented on CAMEL-25536:
------------------------------------------
Fixed on main by https://github.com/apache/camel/pull/27814 (fix version
4.24.0: merged after the camel-4.23.x branch was cut). A small follow-up to a
late review comment (guide boost 1.5, order-independent test) is in
https://github.com/apache/camel/pull/27837.
_Claude Code on behalf of Croway_
> camel-jbang-mcp - camel_migration_guide_search ranks whole release sections
> by word count and ignores the source version, so it returns unrelated upgrade
> notes
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25536
> URL: https://issues.apache.org/jira/browse/CAMEL-25536
> Project: Camel
> Issue Type: Bug
> Components: camel-jbang-mcp
> Affects Versions: 4.22.1
> Reporter: Federico Mariani
> Assignee: Federico Mariani
> Priority: Major
> Fix For: 4.24.0
>
>
> {{camel_migration_guide_search}} returns whole release sections of the
> upgrade guides, ranked by how many query words they contain anywhere, and
> shows the first 30 lines of the section rather than the part that matched. A
> Camel 2 to 4 question is answered from the 4.21 or 4.14 upgrade notes, with a
> snippet about an unrelated component.
> In an evaluation of coding agents (Claude Code and IBM Bob) migrating a Camel
> 2 Spring XML application (camel-cxf SOAP endpoint on the servlet transport,
> camel-mail) to Camel 4 YAML, 9 sessions made 13 successful calls to this tool
> (camel-jbang-mcp 4.22.1). None returned a relevant section; the agents then
> fell back to the 20 K component pages. One agent passed {{camelVersion:
> 4.22.1}} to the tool hoping to scope it; the tool has no such argument.
> h3. Reproduction
> {noformat}
> camel_migration_guide_search query="cxf servlet" limit=3
> -> 1. Camel 4.21 Upgrade Guide / "Upgrading Camel 4.20 to 4.21", snippet
> "=== camel-docling ... === camel-grok ... === camel-jbang ..."
> camel_migration_guide_search query="camel-mail subject header" limit=3
> -> 1. Camel 4.14 Upgrade Guide / "Upgrading from 4.14.3 to 4.14.8", snippet
> "=== camel-jackson - potential breaking change ..."
> camel_migration_guide_search query="CXF SOAP web service migration Camel 2 to
> 4" limit=5
> -> 1. Camel 4.14 Upgrade Guide / "Upgrading Camel 4.13 to 4.14", snippet
> "=== camel-core ... ExecutorServiceManager ..."
> (the 2.x to 3.0 guide's "Migrating Camel applications" comes 4th)
> {noformat}
> Recomputing the score offline over the bundled guides gives the same order:
> for "cxf servlet" the top hit is the 4.20 to 4.21 section (2,667 lines, score
> 6 of 16) because "cxf" appears about 830 lines into it and "camel-servlet"
> about 1,060 lines in, in two unrelated entries; the 30-line snippet never
> reaches either.
> h3. Cause
> {{MigrationData.splitIntoSections}} splits each guide only at {{== }}
> headings, so an upgrade guide is one section per release (158 sections in
> total, up to 2,667 lines; the {{=== camel-xxx}} entries are not split).
> {{MigrationData.scoreSection}} gives +3/+2/+1 per query token found anywhere
> in the section, with no length normalisation, so the longest sections win any
> multi-word query. {{searchGuides}} keeps every section with score > 0, so
> there is never an empty answer.
> {{MigrationTools.camel_migration_guide_search}} has only {{query}} and
> {{limit}}, and the snippet is {{truncateSnippet(section.content(), 30)}}, the
> start of the section. {{MigrationData.getGuidesForVersion(int)}} already
> knows which guides apply to a source major version, but the search does not
> use it.
> Unchanged on main (only {{limit}} became optional in #27511).
> h3. Fix
> * Index at the {{=== }} level (one entry per component/topic, titled "4.21 /
> camel-cxf"), keeping the release heading as context. Offline, this alone
> moves the "camel-cxf" and "camel-mail - potential breaking change" entries to
> the top for the queries above.
> * Score with length normalisation (BM25 or similar) and drop words such as
> camel, migration, upgrade, version from the query.
> * Add optional {{fromVersion}} / {{toVersion}} arguments (e.g. 2.25 / 4.22):
> rank the major migration guides (2.x to 3.0, 3.x to 4.0) and the upgrade
> guides between the two versions first, and omit guides outside the range.
> * Return the lines around the match as the snippet, not the first 30 lines of
> the section.
> * Apply a minimum score and return no results (with a hint to rephrase or use
> {{camel_catalog_docs}}) instead of unrelated sections.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
