GitHub user eugenegujing edited a discussion: Proposal: Making the workspace 
agent useful to a new user by seeing run errors, reading the docs, knowing the 
datasets

The agent can build and run workflows, but it cannot see the errors from a run 
the user started, cannot read Texera's own documentation, and does not know 
which datasets the user has. I want to propose three small, independent 
changes, each tracked as its own issue and PR. The first two are ready to 
start, and the third is a sketch I would like feedback on first. This follows 
up on #9246.

```mermaid
flowchart TB
  subgraph browser [Browser]
    canvas["Workflow canvas<br/>run state, errors, row counts"]
    chat[Agent chat]
  end
  subgraph service [Agent service]
    ctx[Context sent to the model]
    tools[Agent tools]
  end
  docs[("docs/<br/>bundled in the image")]
  files[("file-service<br/>the user's datasets")]

  canvas == "1. report on the user's last run" ==> chat
  chat -- "prompt" --> ctx
  ctx -- "LLM" --> tools
  tools == "2. searchDocs, readDoc" ==> docs
  tools -. "3. listDatasets, listDatasetFiles (later)" .-> files
```

The thick arrows are new in changes 1 and 2. The dotted arrow is the sketch in 
change 3. Everything else exists today.

## 1. Let the agent see the errors from the user's own run

Today the agent only sees an error when it ran the operator itself through its 
`executeOperator` tool. When the user clicks Run and the workflow fails, the 
agent has no idea. So "why did this fail?" is the one question a new user most 
wants to ask it, and the one it cannot answer. The system prompt already tells 
the model how to fix a failing operator in place. What is missing is the input.

The change is on the frontend side. When the user sends a message, the frontend 
attaches a report on the last run the user started from that browser: the fatal 
errors, the last error-level console message of each operator (long stack 
traces truncated), and each operator's state and input and output row counts, 
as suggested in the comments below. The report is attached once the run has 
finished, or while it is still running if an operator has already logged an 
error. A run the agent started itself is not reported. The agent service then 
shows the report under the matching operator in the dataflow section of the 
context, the same way compilation errors already appear there. Two short prompt 
rules ask the model to explain an error in plain language, name the operator, 
suggest the fix and ask before editing anything, and to read the row counts 
when the user asks why a result is empty.

```mermaid
sequenceDiagram
  actor U as User
  participant B as Browser
  participant CU as Computing unit
  participant A as Agent service
  participant M as LLM

  U->>B: Click Run
  B->>CU: Execute the workflow
  CU-->>B: Run state, row counts, errors (websocket)
  Note over B: Today this stays in the browser
  U->>B: "Why did this fail?"
  rect rgba(46, 160, 67, 0.15)
    B->>A: Prompt + report on the user's last run
    A->>M: Context with the error and row counts under each operator
  end
  M-->>A: Explanation and a suggested fix
  A-->>B: Answer
  B-->>U: Answer
```

Why not read the errors from the backend? Fatal errors are not persisted 
anywhere. They live in the computing unit's in-memory state and reach the 
browser over the websocket. The frontend already holds exactly this data, and 
the AI Fix tab from #8637 reads it. Attaching it to the prompt needs no backend 
or database change. If fatal errors get persisted later, the source can switch 
to a REST call without changing the agent side.

This touches the frontend agent and execution services, the agent-service types 
and context serialization, two prompt rules, and tests. It edits the same file 
as #9244, so it will land after that.

## 2. Give the agent the documentation

The agent has no way to look anything up in Texera's own documentation, so a 
question about how Texera works is answered from the model's general knowledge. 
Meanwhile, most of the questions @zyratlo's researcher asked were "how do I do 
X in the UI", and the answers exist in `docs/` and on texera.apache.org.

The change is to bundle `docs/` into the agent-service image at build time and 
build an in-memory keyword index over the pages on startup, split by section. 
Two read-only tools go on top of it. `searchDocs(query)` returns the best 
matching sections with a link to the published page, and `readDoc(path, 
section?)` returns the text. What the tools return is kept in its own section 
of the model's context for the rest of the task, with a size limit. One new 
prompt rule says: for questions about how Texera works, search the docs, read 
the page, answer with a link, and do not edit the workflow.

```mermaid
flowchart LR
  subgraph build [Build time]
    md["docs/*.md"] -- "COPY docs" --> image[agent-service image]
  end
  subgraph startup [Startup]
    index["In-memory keyword index<br/>one entry per section"]
  end
  subgraph question [A user question]
    q["How do I upload a dataset?"] --> search[searchDocs]
    search --> read[readDoc]
    read --> answer["Answer with a link<br/>to the published page"]
  end
  image --> index
  index --> search
  index --> read
```

I want to keep this small. No embeddings, no model download, no new dependency. 
A plain keyword index over about 500 KB of markdown is enough. Docs are bundled 
rather than fetched at runtime so the answers match the deployed version and 
work offline. When #9031 regenerates the operator reference pages, the index 
picks them up with no change.

One side effect is useful on its own. Running a set of new-user questions 
against the index also shows which questions have no page at all, and that list 
can go to #8773.

This is two new files (the index and the tools), one new section in the 
context, one new prompt rule, one `COPY docs` line in the Dockerfile, a 
`TEXERA_DOCS_DIR` override for local runs, and tests. #4932 is related, since 
llms.txt could be generated from the same index later.

## 3. A sketch for letting the agent see the user's datasets

The agent does not know which datasets the user has or what a dataset file path 
looks like, so it guesses paths and gets them wrong (#4588).

The idea is two read-only tools backed by file-service endpoints that already 
exist, `listDatasets()` and `listDatasetFiles(dataset, version?)`. The second 
returns each file's full path in the form the file scan operators accept, so 
the agent can fill it in directly. The tools call file-service with the user's 
own delegated token, so the user's access rules are enforced where they already 
are. This is discovery only. Reading a dataset from inside a Python UDF is the 
mount feature (#6866, #6896, #6897) and the questions raised in #9048.

There is prior work here. @aicam prototyped dataset tools (list, get, versions, 
and more) as part of a standalone MCP server on a fork branch in July, 
following the discussion in #5610, where the conclusion was that Texera's own 
actions should stay inline tools. The shape above is the inline version of that 
prototype's read-only subset. #5918 raised how the agent should call backend 
endpoints without duplicating the REST contract by hand, and #9395 now proposes 
one set of endpoints for every kind of resource. I would rather build these 
tools on that than add another hand-written client.

I am not proposing to start this yet. The agent service does not bind an agent 
to its owner (#5268) and is getting request authentication in #9043. A tool 
that lists a user's datasets should land after those, so I would rather agree 
on the shape now and open the issue once #9043 is in.

## What I am asking

- Any suggestions on the three changes, or other ideas for what the agent 
should be able to see or do for a new user?
- For 2, bundling docs into the image versus fetching from the website at 
runtime. I prefer bundling for the reasons above.
- For 3, any objection to the shape?

I will open one issue per change and link the PRs here.


GitHub link: https://github.com/apache/texera/discussions/9377

----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Reply via email to