Federico Mariani created CAMEL-25536:
----------------------------------------
Summary: camel-jbang-mcp - camel_migration_guide_search ranks
whole release sections by word count and ignores the source version, so it
returns unrelated upgrade notes
Key: CAMEL-25536
URL: https://issues.apache.org/jira/browse/CAMEL-25536
Project: Camel
Issue Type: Bug
Components: camel-jbang-mcp
Affects Versions: 4.22.1
Reporter: Federico Mariani
{{camel_migration_guide_search}} returns whole release sections of the upgrade
guides, ranked by how many query words they contain anywhere, and shows the
first 30 lines of the section rather than the part that matched. A Camel 2 to 4
question is answered from the 4.21 or 4.14 upgrade notes, with a snippet about
an unrelated component.
In an evaluation of coding agents (Claude Code and IBM Bob) migrating a Camel 2
Spring XML application (camel-cxf SOAP endpoint on the servlet transport,
camel-mail) to Camel 4 YAML, 9 sessions made 13 successful calls to this tool
(camel-jbang-mcp 4.22.1). None returned a relevant section; the agents then
fell back to the 20 K component pages. One agent passed {{camelVersion:
4.22.1}} to the tool hoping to scope it; the tool has no such argument.
h3. Reproduction
{noformat}
camel_migration_guide_search query="cxf servlet" limit=3
-> 1. Camel 4.21 Upgrade Guide / "Upgrading Camel 4.20 to 4.21", snippet "===
camel-docling ... === camel-grok ... === camel-jbang ..."
camel_migration_guide_search query="camel-mail subject header" limit=3
-> 1. Camel 4.14 Upgrade Guide / "Upgrading from 4.14.3 to 4.14.8", snippet
"=== camel-jackson - potential breaking change ..."
camel_migration_guide_search query="CXF SOAP web service migration Camel 2 to
4" limit=5
-> 1. Camel 4.14 Upgrade Guide / "Upgrading Camel 4.13 to 4.14", snippet "===
camel-core ... ExecutorServiceManager ..."
(the 2.x to 3.0 guide's "Migrating Camel applications" comes 4th)
{noformat}
Recomputing the score offline over the bundled guides gives the same order: for
"cxf servlet" the top hit is the 4.20 to 4.21 section (2,667 lines, score 6 of
16) because "cxf" appears about 830 lines into it and "camel-servlet" about
1,060 lines in, in two unrelated entries; the 30-line snippet never reaches
either.
h3. Cause
{{MigrationData.splitIntoSections}} splits each guide only at {{== }} headings,
so an upgrade guide is one section per release (158 sections in total, up to
2,667 lines; the {{=== camel-xxx}} entries are not split).
{{MigrationData.scoreSection}} gives +3/+2/+1 per query token found anywhere in
the section, with no length normalisation, so the longest sections win any
multi-word query. {{searchGuides}} keeps every section with score > 0, so there
is never an empty answer. {{MigrationTools.camel_migration_guide_search}} has
only {{query}} and {{limit}}, and the snippet is
{{truncateSnippet(section.content(), 30)}}, the start of the section.
{{MigrationData.getGuidesForVersion(int)}} already knows which guides apply to
a source major version, but the search does not use it.
Unchanged on main (only {{limit}} became optional in #27511).
h3. Fix
* Index at the {{=== }} level (one entry per component/topic, titled "4.21 /
camel-cxf"), keeping the release heading as context. Offline, this alone moves
the "camel-cxf" and "camel-mail - potential breaking change" entries to the top
for the queries above.
* Score with length normalisation (BM25 or similar) and drop words such as
camel, migration, upgrade, version from the query.
* Add optional {{fromVersion}} / {{toVersion}} arguments (e.g. 2.25 / 4.22):
rank the major migration guides (2.x to 3.0, 3.x to 4.0) and the upgrade guides
between the two versions first, and omit guides outside the range.
* Return the lines around the match as the snippet, not the first 30 lines of
the section.
* Apply a minimum score and return no results (with a hint to rephrase or use
{{camel_catalog_docs}}) instead of unrelated sections.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
