GitHub user Xiao-zhen-Liu added a comment to the discussion: Proposal:
operators that emit on more than one output port
Thanks for writing up the Phase 2 design before opening PRs, @sshiv012. Here
are my thoughts on the five points:
1. **`PortOutput(value, port=1)` on the existing generator methods:** Agreed.
Keeping one generator method per UDF base class with an explicit `PortOutput`
wrapper is cleaner than adding separate `*_multi_port` methods (which would
double the callback surface across the four UDF base classes and drop
`@abstractmethod` checks on 1-port UDFs) or yielding raw `(row, port)` pairs.
2. **Unnamed yields on multi-port UDFs:** Keep bare `yield value` for
1-output-port UDFs, and require `PortOutput(value, port=...)` on every output
(including port `0`) whenever the operator has 2 or more output ports, raising
a `ValueError` on a bare yield:
```python
def process_tuple(self, tuple_: Tuple, port: int) ->
Iterator[Optional[UDFOutput]]:
if tuple_["latency_ms"] <= 1000:
yield PortOutput(tuple_, port=0)
else:
yield PortOutput(tuple_, port=1)
```
On the JVM side (`DataProcessor.outputOneTuple` ->
`OutputManager.getSingleOutputPortIdentity`), an output with `None` port
asserts `ports.size == 1` rather than broadcasting. Requiring `PortOutput` on
multi-port Python UDFs matches that rule, reads symmetrically across ports, and
turns a forgotten wrapper on a same-schema split into an immediate error
instead of silently routing every row to port `0`.
3. **Zero-based `int` port indices:** Agreed. This matches the input `port:
int` parameter and survives port renames. Please also reject `bool` values
since `isinstance(True, int)` is `True` in Python.
4. **Runtime error on unknown ports:** Agreed. Please include the invalid port
index and the operator's valid output port indices in the `ValueError` message.
5. **Per-port schemas (and visualization helper):** Using one shared schema
across all output ports in the first Phase 2 PR is a good first step so routing
lands on its own. For the second Phase 2 PR, rather than an interim `kind`-only
setting that still forces all `table` ports to share operator-level columns
(which cannot express cases like parsed rows on port `0`, reject rows with an
`error` column on port `1`, and summary counts on port `2`), let us put the
full output schema on each output port directly: `kind` (`table` with
`SET_SNAPSHOT` vs `visualization` with `html-content: STRING` and
`OutputMode.SINGLE_SNAPSHOT` in `PythonUDFOpDescV2.operatorInfo`), plus
per-port `retainInputColumns` and `outputColumns` for `table` ports (falling
back to the operator-level fields for saved workflows without per-port
schemas). For `visualization` ports, a small helper such as
`PortOutput.html(html, port=1)` (wrapping `{"html-content": html}`) would also
avoid making users write the internal
`"html-content"` column name by hand.
One implementation note for `OutputManager` (`output_manager.py`): `emit_ecm`
and `emit_state` also call `tuple_to_frame` when flushing buffered batches, so
please pass each link's `from_port_id` there as well once ports can have
different schemas.
@Yicong-Huang could you also share your thoughts on this Python UDF multi-port
API and per-port schema design before @sshiv012 opens the Phase 2 PRs?
GitHub link:
https://github.com/apache/texera/discussions/8914#discussioncomment-19787161
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]