GitHub user eugenegujing added a comment to the discussion: Proposal: Unify 
Backend Resource Management On Texera

I support this proposal. Following @aicam's suggestion in #9377, here is what 
the agent changes proposed there need to read from the backend, so it can be 
considered in the design.

1. **The outcome of a run.** To answer "why did my run fail?" or "why is my 
result empty?", the agent needs the errors, the console messages and the row 
counts per operator of the user's last execution. Today fatal errors live only 
in the computing unit's memory, console messages are stored but no endpoint 
returns them for a past execution, and only row counts can be read. #9423 works 
around this by sending them from the browser. Executing is specific to 
workflows, so I don't expect the common endpoints to cover it. I am listing it 
so the design knows the agent needs to read an execution's outcome, with the 
same access check as the workflow it belongs to.
2. **Datasets.** To answer "which file should I scan?", the agent needs to list 
the datasets the user can read and the files of a dataset version. This is 
change 3 in #9377. It is not started, and I can build it on the unified 
endpoints.
3. **The same access as the user.** The agent service already calls the backend 
with the user's token, so a resource the user cannot read should be unreadable 
for the agent too, with no separate rule for the agent.

Also, I don't think my open agent PRs (#9423, #9424) conflict with this effort. 
Neither changes backend code or adds a call to a backend endpoint.

GitHub link: 
https://github.com/apache/texera/discussions/9395#discussioncomment-19604876

----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Reply via email to