GitHub user eugenegujing added a comment to the discussion: Proposal: Unify Backend Resource Management On Texera
I support this proposal. Following @aicam's suggestion in #9377, here is what the agent changes proposed there need to read from the backend, so it can be considered in the design. 1. **The outcome of a run.** To answer "why did my run fail?" or "why is my result empty?", the agent needs the errors, the console messages and the row counts per operator of the user's last execution. Today fatal errors live only in the computing unit's memory, console messages are stored but no endpoint returns them for a past execution, and only row counts can be read. #9423 works around this by sending them from the browser. Executing is specific to workflows, so I don't expect the common endpoints to cover it. I am listing it so the design knows the agent needs to read an execution's outcome, with the same access check as the workflow it belongs to. 2. **Datasets.** To answer "which file should I scan?", the agent needs to list the datasets the user can read and the files of a dataset version. This is change 3 in #9377. It is not started, and I can build it on the unified endpoints. 3. **The same access as the user.** The agent service already calls the backend with the user's token, so a resource the user cannot read should be unreadable for the agent too, with no separate rule for the agent. Also, I don't think my open agent PRs (#9423, #9424) conflict with this effort. Neither changes backend code or adds a call to a backend endpoint. GitHub link: https://github.com/apache/texera/discussions/9395#discussioncomment-19604876 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
