[
https://issues.apache.org/jira/browse/CAMEL-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18120001#comment-18120001
]
Claus Ibsen commented on CAMEL-25075:
-------------------------------------
Related: CAMEL-25040 (the catalog answer omitting what the author needed) and
CAMEL-25074 (an option's consequence not stated). All three surfaced from the
same benchmark rung.
> camel-jbang-mcp - a repeated identical tool call should answer differently,
> so an agent can notice it is looping
> ----------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25075
> URL: https://issues.apache.org/jira/browse/CAMEL-25075
> Project: Camel
> Issue Type: Improvement
> Components: camel-jbang
> Reporter: Claus Ibsen
> Priority: Major
>
> An agent that asks the same tool the same question over and over gets the
> same answer each time, with nothing in the answer to tell it so. A local
> model does this, and it is the largest single cause of failure on one rung of
> the benchmark.
> Measured on the {{connect-service-sql}} example, whose first step writes a
> route that inserts a row with named parameters. Across 28 attempts in three
> 10-pass runs, the calls of a failing attempt look like this (s19-2, verbatim):
> {noformat}
> camel_get_files()
> camel_get_files(sql.camel.yaml)
> camel_get_files(application.properties)
> camel_get_files(orders/order-1001.json)
> camel_catalog_doc(sql)
> camel_catalog_doc(sql)
> ... 16 times in total, identical arguments
> {noformat}
> It never wrote a file, and the step failed with its whole budget spent. Of 10
> passes of that step, 8 ended pinned at the tool-call ceiling, and in most of
> them the majority of the calls were identical repeats of
> {{camel_catalog_doc(sql)}}. Raising the ceiling does not help: the same step
> was run at 12, 20 and 24 calls, and at 24 the extra calls were spent on more
> repeats.
> What it was hunting for was the SQL dialect of the database, which the
> catalog cannot supply, so the answer was never going to change. But a model
> has no way of noticing that the answer it just received is the one it already
> had.
> This is fixable outside the model. Some options, in rough order of how little
> they assume:
> * the server notices that a call repeats an earlier call of the same session
> with the same arguments, and answers with something different: that the
> answer is unchanged, how many times it has been asked, and what else is worth
> trying ({{camel_catalog_docs}} for the page, {{camel_catalog_sample}} for a
> shape, {{camel_get_files}} for what is on disk);
> * the repeat answer carries a short list of the tools not yet used in this
> session, since an agent going in circles is usually one that has not thought
> of the next tool;
> * a cheaper variant: the answer is returned as before with one line prepended
> saying it is a repeat.
> The point is not to refuse the call -- a legitimate re-read after an edit
> must still work -- but to make a repeat *look different* so it can break the
> loop.
> Filed from the local-model benchmark of the camel-jbang-mcp server; see
> CAMEL-25040 for the related finding that the catalog answer omitted what the
> author needed in the first place.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
