GitHub user bobbairobot created a discussion: Proposal: Unify Resource
Management On Texera
## How Texera manages resources today
Texera manages six kinds of resources: workflows, datasets, models, computing
units, environments, and warehouses. By managing a resource I mean three
things: creating, reading, updating, and deleting it (CRUD), keeping its
versions, and controlling who can access it. Today each kind is managed by its
own code: workflows and warehouses in the Dashboard Service, datasets and
models in the File Service, computing units in the Computing Unit Managing
Service, and environments in the computing unit itself.
## Major Limitation: Resources Are Managed Ad-hoc
I think Texera's resource management has one major limitation: resources are
managed ad-hoc. Each kind was built on its own, with its own tables, endpoints,
and rules, and the codebase has no single definition of a resource that every
kind follows. As a result, the same three things are done differently for each
kind:
- **CRUD is defined per kind**: each kind has its own endpoints with its own
conventions. For example, a workflow is created with `POST /workflow/create`,
but an environment with `POST /pve/db`. A dataset is deleted with `DELETE
/dataset/{did}`, but a workflow with `POST /workflow/delete`.
- **Versions are defined per kind**: each versioned kind has its own version
logic. For example, the Dashboard Service stores a workflow version as a JSON
patch on every save that changes the workflow. The File Service instead stores
a dataset version as a lakeFS commit when the user asks for one, and keeps a
second copy of that code for model versions.
- **Access control is defined per kind**: each kind has its own way to record
who owns a resource and who can access it. For example, ownership is recorded
in a `workflow_of_user` table for workflows, in an `owner_uid` column for
datasets and models, and in a `uid` column for the other kinds. The rules
differ too: a user with read access can remove themselves from a shared
workflow, but not from a shared dataset, and environments and warehouses cannot
be shared at all.
The codebase itself shows how far the kinds have drifted apart.
`ResourceTables.scala` was added so that the access rules could be written
once, but its own comment says that it fits only datasets and models, because
workflows record ownership in a separate table and computing units have no
public flag.
## Consequences: Backend Code Is Duplicated, Hard to Maintain, and Hard to
Extend 
The current backend codebase suffers from three problems:
- **Duplicate code**: the same logic is written once per kind. For example,
`ModelResource.scala` repeats 31 of the 32 endpoints of
`DatasetResource.scala`, and the access rules are implemented three times: for
workflows, for datasets and models, and for computing units. The database
repeats itself in the same way: 17 of its 42 tables are per-kind copies.
- **Hard to add new resource kinds**: a new kind has to repeat the tables, the
endpoints, the version logic, and the access rules. For example, adding models
took 12 backend pull requests and 5 database migrations, and 5 of those pull
requests were refactors that made dataset code reusable for models. Down the
road I expect Texera to add more kinds, such as agents, conversations, and
operators, and each of them will pay this cost again.
- **Hard to develop new features on top of resources**: a feature that works
across kinds has to be connected to each kind separately. For example, search
maps the columns of each searchable kind by hand into one schema of 28 columns,
and the agent service can read and save only workflows, because that is the
only kind it has a client for. A new feature that works with every kind, such
as an agent or an MCP server, would need the same boilerplate for each kind.
## Proposal: Unify the Resource Management
I propose to unify resource management. By unify I mean: designing and
implementing common primitives that are shared across all kinds of resources.
Based on current Texera's position, I see three major primitives:
**P1: Resource CRUD**: every kind has an id, a name, and an owner, and every
kind needs the same operations: create, list, read, update, and delete.
**P2: Resource version CRUD**: a workflow version, a dataset version, and a
model version are the same concept, which is an immutable snapshot of a
resource. The `dataset_version` and `model_version` tables already have the
same columns.
**P3: Resource access control**: every kind needs the same decision, which is
whether a user may read or write a resource. The four `*_user_access` tables
already have the same three columns: the user, the resource, and the privilege.
[image: Screenshot 2026-11-09 at 4.51.03 PM.png\]
What stays specific to a kind is its own fields and actions, such as executing
a workflow or uploading a file to a dataset. Once the common parts exist in one
copy, a new kind defines only what is specific to it, and a feature built on
the common parts works for every kind.
## Implementation Proposal
I see at least two layers whose implementation needs to be refactored: the
database schema layer and the API definition layer. Each layer needs its own
detailed discussion, so below I only give my proposal for each.
### Layer 1: Database Schema Layer
At this layer, my proposal is to store the common parts once, under two
supertypes: `Resource` and `Resource Version`.
**Before**: each kind has its own tables for the common parts. Each kind keeps
its own relationships to `User`, ten `own` and `access` relationships in total,
and each versioned kind keeps its own version table.
[image: Animation of today's database schema: the plain diagram first, then
redundant access tables, redundant version tables, and what adding one kind
costs, highlighted one at a time\]
**After**: every kind is a subtype of `Resource`, which has a single `access`
relationship to `User` with the levels own, read, and write. Every version is a
subtype of `Resource Version`, keyed by `(rid, vno)`, so one `lineage` table
can link a version to the versions it was derived from. A new kind adds only
its own fields under `Resource`.
[image: Animation of the proposed database schema: the plain diagram first,
then one access relationship, one version model, easy to record resource
lineage, and easy to add new resource kinds, highlighted one at a time\]
### Layer 2: API Definition Layer
My proposal at this layer is to give every kind the same endpoints for the
three common parts. For each part, the table below lists the endpoints before
and after. In the After column, `{kind}` is the plural name of a kind, such as
`workflows` or `datasets`, `{rid}` is the id of a resource, `{vno}` is the
number of a version, and `{uid}` is the id of a user.
#### Resource CRUD
Before, each kind spells the same operation in its own way, and some kinds lack
an operation. After, every kind has the same five operations.
| Operation | Before | After |
| --- | --- | --- |
| Create | `POST /workflow/create`, `POST /dataset/create`, `POST
/model/create`, `POST /computing-unit/create`, `POST /pve/db`, `POST
/warehouse` | `POST /{kind}` |
| List | `GET /workflow/list`, `GET /dataset/list`, `GET /model/list`, `GET
/computing-unit`, `GET /pve/db`, `GET /warehouse/status` | `GET /{kind}` |
| Read | `GET /workflow/{wid}`, `GET /dataset/{did}`, `GET /model/{mid}`, `GET
/computing-unit/{cuid}`; none for environments and warehouses | `GET
/{kind}/{rid}` |
| Update (rename) | `POST /workflow/update/name`, `POST /dataset/update/name`,
`POST /model/update/name`, `PUT /computing-unit/{cuid}/rename/{name}`, `PUT
/pve/db/{veid}`; none for warehouses | `PATCH /{kind}/{rid}` |
| Delete | `POST /workflow/delete`, `DELETE /dataset/{did}`, `DELETE
/model/{mid}`, `DELETE /computing-unit/{cuid}/terminate`, `DELETE
/pve/db/{veid}`, `DELETE /warehouse/{whid}` | `DELETE /{kind}/{rid}` |
#### Resource Version CRUD
Before, workflow versions have their own root path and no endpoint to create
one, because a version is made whenever a workflow is saved, while dataset and
model versions put the operation in the path. After, every versioned kind has
the same three operations. A version is numbered within its resource and
immutable, so it has no update or delete.
| Operation | Before | After |
| --- | --- | --- |
| Create | `POST /dataset/{did}/version/create`, `POST
/model/{mid}/version/create`; none for workflows, where `POST
/workflow/persist` makes a version as a side effect | `POST
/{kind}/{rid}/versions` |
| List | `GET /version/{wid}`, `GET /dataset/{did}/version/list`, `GET
/model/{mid}/version/list` | `GET /{kind}/{rid}/versions` |
| Read | `GET /version/{wid}/{vid}`, `GET
/dataset/{did}/version/{dvid}/rootFileNodes`, `GET
/model/{mid}/version/{mvid}/rootFileNodes` | `GET /{kind}/{rid}/versions/{vno}`
|
#### Resource Access Control
Before, the same sharing endpoints are defined four times, in three services.
Environments and warehouses have none, and no endpoint checks access, so each
service checks it in its own code. After, every kind is shared and checked in
the same way.
| Operation | Before | After |
| --- | --- | --- |
| List | `GET /access/workflow/list/{wid}`, plus a copy each for datasets,
models, and computing units | `GET /{kind}/{rid}/access` |
| Grant or change | `PUT /access/workflow/grant/{wid}/{email}/{privilege}`,
plus the same three copies | `PUT /{kind}/{rid}/access/{uid}` |
| Revoke | `DELETE /access/workflow/revoke/{wid}/{email}`, plus the same three
copies | `DELETE /{kind}/{rid}/access/{uid}` |
| Check | No endpoint: each service checks access in its own code | `POST
/access/evaluation` |
GitHub link: https://github.com/apache/texera/discussions/9394
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]