[
https://issues.apache.org/jira/browse/CAMEL-24796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18116317#comment-18116317
]
Claus Ibsen commented on CAMEL-24796:
-------------------------------------
Agreed design (2026-09-17):
* *Window policy* (LlmClient, so camel ask and the TUI agree):
OLLAMA_CONTEXT_LENGTH wins when set. Otherwise, if the model is already loaded
(/api/ps), adopt its context_length so the TUI never forces a reload against
another client (minimum 32k, a smaller loaded window is raised to 32k).
Otherwise request 64k when the model's KV cache for 64k fits next to its
weights in the machine's memory (computed from /api/show: layers, KV heads,
head dimension), else 32k. 64k is the maximum the panel asks for on its own.
* *Compaction budget*: min(allocated window, 64k). A 256k window loaded by
someone else is used as is, but the panel manages its own history as if the
window were 64k, so a cache loss (idle unload, other client, Ollama restart)
never means minutes of re-prefill.
* *Trigger on measured tokens*: the prompt size Ollama reports for the last
step of a turn, against the budget (half of it), with the character estimate
only as fallback when nothing was measured.
* *Compact once, big*: on a local endpoint compaction also shrinks the previous
turn's tool results and drops the oldest turns until the history is around a
quarter to a third of the budget, then stays quiet until the trigger fires
again. Hosted providers keep today's behaviour.
* *No confirm dialog*: an explicit /compact and an automatic compaction both
print one line in the conversation before the work, with the measured numbers:
"Compacting history: 24.8k -> about 8k tokens. The next reply prefills the
whole prompt again (about 12 s at 650 tok/s) because Ollama's cache resets."
Hosted providers get the short form (tokens saved per request). The panel title
shows the measured fill (ctx 34%), /context shows window, budget and fill.
* *Ollama tab*: the header shows the panel's window and compaction budget next
to what Ollama allocated and the model maximum, so the user sees 32k, 64k or
256k at a glance; the turns line colours against the budget rather than the raw
window.
_Claude Code on behalf of davsclaus_
> camel-jbang - TUI AI panel with Ollama: compact on measured tokens, compact
> harder, and raise the 32k default context
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-24796
> URL: https://issues.apache.org/jira/browse/CAMEL-24796
> Project: Camel
> Issue Type: Improvement
> Components: camel-jbang
> Reporter: Claus Ibsen
> Priority: Major
>
> Observed with the new Ollama tab (CAMEL-24794) while using the TUI AI panel
> (F8) against Ollama 0.33.3 serving qwen3.6:35b-a3b on an Apple M4 Pro, 64 GB.
> Six questions with tool calls produced 54 Ollama requests. The prompt grew
> from 4.5k tokens (system prompt plus core tools plus question) to 24.8k, 76%
> of the 32k window the client asks for, before the panel compacted. The
> compaction then freed only 3.8k tokens (down to 21.0k, 64%) and, because
> rewriting history invalidates Ollama's prompt cache, the next request
> re-evaluated all 21k tokens at 528 tok/s: about 40 s time to first token
> against 0.3 to 0.7 s for the cached steps before it.
> Three changes to the AI panel (AiPanel.compactHistoryAfterTurn /
> shouldCompactAfterTurn / compactHistory in camel-jbang-plugin-tui):
> 1. *Trigger on measured tokens, not an estimate.* The panel compacts when its
> character-based estimate of the history (chars / 4) passes
> LOCAL_HISTORY_BUDGET_TOKENS = 16000. Tool results are JSON and tokenize
> nearer 3 chars per token, so the real prompt was 24.8k when the estimate
> crossed 16k. Ollama reports the exact prompt size (prompt_eval_count, now in
> LlmClient.TokenUsage.inputTokens together with cached, prefill, generation,
> load and total) on every reply; use that against the num_ctx the client
> requested, and compact at about half the window.
> 2. *Compact big when compacting.* Today compaction shrinks tool results of
> older turns to 400 characters and drops turns only beyond 20. Locally every
> compaction costs a full re-prefill, so it must be worth it: keep the last two
> or three turns intact, reduce older turns to question plus final answer, and
> aim for the prompt to land around 25 to 30% of the window rather than 64%.
> Consider doing it once at the threshold rather than a little every turn. Show
> "compacted, the next reply prefills the whole prompt" in the panel and the
> measured fill (e.g. ctx 64%) in the panel title, so the climb is visible
> before it hurts.
> 3. *Tune the default context size.* LlmClient.OLLAMA_NUM_CTX = 32768
> (override with OLLAMA_CONTEXT_LENGTH) is low for a tool-calling panel: the
> static prefix alone is 4.5k and each question with tool calls adds 2 to 4k.
> Measured cost of a bigger window on this model: 22.47 GB loaded at 32k versus
> 23.57 GB at 262k (the hybrid attention of qwen3.5 MoE keeps the KV cache
> small), so 64k or 128k is affordable on typical developer machines; dense
> models cost more per token of context, so the default should probably depend
> on the model's max context (from /api/show) and available memory, capped
> around 64k or 128k, with the prefill speed in mind (a full re-prefill of 128k
> at 600 tok/s is 3.5 minutes). Whatever value is chosen must be the same for
> every Ollama request the TUI makes, because a differing num_ctx makes Ollama
> reload the model (20 s cold start observed).
> The Ollama tab (CTX column, turns line, CACHED column) and tui_get_ollama
> make the effect of these changes measurable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
