[
https://issues.apache.org/jira/browse/CAMEL-24796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen updated CAMEL-24796:
--------------------------------
Fix Version/s: 4.23.0
> camel-jbang - TUI AI panel with Ollama: compact on measured tokens, compact
> harder, and raise the 32k default context
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-24796
> URL: https://issues.apache.org/jira/browse/CAMEL-24796
> Project: Camel
> Issue Type: Improvement
> Components: camel-jbang
> Reporter: Claus Ibsen
> Assignee: Claus Ibsen
> Priority: Major
> Fix For: 4.23.0
>
>
> Observed with the new Ollama tab (CAMEL-24794) while using the TUI AI panel
> (F8) against Ollama 0.33.3 serving qwen3.6:35b-a3b on an Apple M4 Pro, 64 GB.
> Six questions with tool calls produced 54 Ollama requests. The prompt grew
> from 4.5k tokens (system prompt plus core tools plus question) to 24.8k, 76%
> of the 32k window the client asks for, before the panel compacted. The
> compaction then freed only 3.8k tokens (down to 21.0k, 64%) and, because
> rewriting history invalidates Ollama's prompt cache, the next request
> re-evaluated all 21k tokens at 528 tok/s: about 40 s time to first token
> against 0.3 to 0.7 s for the cached steps before it.
> Three changes to the AI panel (AiPanel.compactHistoryAfterTurn /
> shouldCompactAfterTurn / compactHistory in camel-jbang-plugin-tui):
> 1. *Trigger on measured tokens, not an estimate.* The panel compacts when its
> character-based estimate of the history (chars / 4) passes
> LOCAL_HISTORY_BUDGET_TOKENS = 16000. Tool results are JSON and tokenize
> nearer 3 chars per token, so the real prompt was 24.8k when the estimate
> crossed 16k. Ollama reports the exact prompt size (prompt_eval_count, now in
> LlmClient.TokenUsage.inputTokens together with cached, prefill, generation,
> load and total) on every reply; use that against the num_ctx the client
> requested, and compact at about half the window.
> 2. *Compact big when compacting.* Today compaction shrinks tool results of
> older turns to 400 characters and drops turns only beyond 20. Locally every
> compaction costs a full re-prefill, so it must be worth it: keep the last two
> or three turns intact, reduce older turns to question plus final answer, and
> aim for the prompt to land around 25 to 30% of the window rather than 64%.
> Consider doing it once at the threshold rather than a little every turn. Show
> "compacted, the next reply prefills the whole prompt" in the panel and the
> measured fill (e.g. ctx 64%) in the panel title, so the climb is visible
> before it hurts.
> 3. *Tune the default context size.* LlmClient.OLLAMA_NUM_CTX = 32768
> (override with OLLAMA_CONTEXT_LENGTH) is low for a tool-calling panel: the
> static prefix alone is 4.5k and each question with tool calls adds 2 to 4k.
> Measured cost of a bigger window on this model: 22.47 GB loaded at 32k versus
> 23.57 GB at 262k (the hybrid attention of qwen3.5 MoE keeps the KV cache
> small), so 64k or 128k is affordable on typical developer machines; dense
> models cost more per token of context, so the default should probably depend
> on the model's max context (from /api/show) and available memory, capped
> around 64k or 128k, with the prefill speed in mind (a full re-prefill of 128k
> at 600 tok/s is 3.5 minutes). Whatever value is chosen must be the same for
> every Ollama request the TUI makes, because a differing num_ctx makes Ollama
> reload the model (20 s cold start observed).
> The Ollama tab (CTX column, turns line, CACHED column) and tui_get_ollama
> make the effect of these changes measurable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
