GitHub user sshiv012 added a comment to the discussion: Proposal: operators 
that emit on more than one output port

@Xiao-zhen-Liu here's the Python UDF design for Phase 2, before I open any PRs. 
I have a working prototype with tests on a local branch.

## The API

```python
def process_tuple(self, tuple_, port):
    if tuple_["score"] >= 50:
        yield tuple_                       # port 0, same as today
    else:
        yield PortOutput(tuple_, port=1)   # port 1
```

A plain yield goes to port 0. `PortOutput` sends a row to another port. To send 
a row to both ports, yield it twice.

## Files that I believe would need changes

Each line shows what the file does today -> what changes. Line numbers are 
approximate.

**Compile time (Scala)**
1. `PythonUDFOpDescV2.scala`: only port 0 gets a schema, so a link from port 1 
fails with "schema is not available"  ->  every output port gets the same 
schema (one line, ~131).

**User API (pytexera)**

2. `core/models/port_output.py` (new): `PortOutput(value, port)`, a frozen 
dataclass. `value` is anything a UDF can yield today: a row, a dict, a Table or 
a DataFrame. It's exported from `pytexera` and `pyamber`.
3. `pytexera/udf/udf_operator.py`: the base classes users extend  -> return 
types and docstrings now allow `PortOutput`. No behavior change.

**Turning a yield into rows**

4. `core/models/operator.py`: `SourceOperator.on_finish` splits every yield 
into rows ->  it passes `PortOutput` through, so the port isn't lost (~223).
5. `core/models/table.py`: `all_output_to_tuple` turns a yield into rows  -> 
`all_output_to_port_tuples` turns it into `(row, port)` pairs, with port 0 when 
the yield isn't wrapped (~132–144).

**Running the UDF**

6. `core/runnables/data_processor.py`: checks every row against port 0's schema 
 -> checks each row against its own port's schema (~172).
7. `core/architecture/managers/tuple_processing_manager.py`: passes rows from 
the UDF thread to the main loop  ->  also passes the port, then resets it to 0 
(~32, ~49).
8. `core/runnables/main_loop.py`: counts every row on port 0 and sends it 
everywhere  -> counts, sends and stores each row on its own port (~376).

**Sending and storing**

9. `core/architecture/packaging/output_manager.py`: sends every row on every 
link, encodes it with port 0's schema, and stores it on every port  -> each row 
goes only to the links and the storage of its own port, encoded with that 
port's schema (~347–421). An unknown port raises `ValueError` (~204).

**Compatibility:** UDFs with one port behave exactly as before. UDFs with two 
or more ports fail on `main` today, so no working workflow changes.

## I need your help with the below

1. **Naming a port uses `PortOutput(row, port=1)`.** I ruled out three 
alternatives:
   - a `(row, 1)` tuple, which looks like data;
   - a `__port__` column, which mixes routing into the data;
   - `self.emit()`, which would be a second way to output rows.
2. **A plain yield goes to port 0 only, not to every port.** Sending to every 
port would copy each row and make splitting impossible. Scala treats a missing 
port as "all ports". Does anything rely on Python doing the same?
3. **Ports are numbered from 0, not named.** This matches input ports, and 
renaming a port won't break code.
4. **An unknown port is an error while the workflow runs, not a silent drop.** 
The compiler can't catch it because it never reads the code.
5. Should every port share the same columns, as in the prototype, or should 
each port declare its own? The compiler only reads settings, so columns per 
port would need a per-port setting in the form. For next PR. 
My proposal is to start with a per-port `kind` setting:
- `table` (the default): today's columns.
- `visualization`: `html-content: STRING`.
That covers "table on one port, chart on another". Full column lists per port 
can come later. Workflows saved before this change treat every port as `table`. 

Note: The first PR will assume all output ports to have the same schema. We can 
add future PRs to support different schema for different ports. 


GitHub link: 
https://github.com/apache/texera/discussions/8914#discussioncomment-19728384

----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Reply via email to