GitHub user aglinxinyuan edited a discussion: Control blocks in Texera: design
and roadmap
Loops are now on Texera's main branch (#5700). Each existing thread covers one
detail, such as `$name` in properties (#8643) or the control-variable port
(#8915). This thread is for the big picture: what a control block is, where it
stands, and what to build next.
## What and why
An operator normally runs once, with its settings fixed before the run. Some
tasks need part of a workflow to run again with a new value each time, for
example trying K = 2 to 10 to pick the best K, or processing every file in a
folder.
A control block is that part of the workflow. A start operator and an end
operator enclose a body of ordinary operators, and short Python code on the two
decides how the body runs. Loops are the first kind.
```mermaid
flowchart LR
D[Data] --> LS
subgraph blk [Control block: a loop]
LS["Loop Start<br/>K = 2"] --> A["Body operators<br/>a setting uses $K"]
--> LE["Loop End<br/>K += 1<br/>while K <= 10"]
LE -. next iteration .-> LS
end
LE --> R["Results of<br/>all iterations"]
```
## Key ideas
- **The body stays on the canvas.** Users build and inspect it like any other
part of the workflow.
- **Control variables** such as `K` or a file path change between iterations. A
property refers to one as `$K`, so existing operators work unchanged.
- **Values travel with the data.** Each iteration's variables move along the
same edges as its rows, using Texera's state passing (#2802). They reach every
operator in the body before its first row. The reasons are in #7429.
```mermaid
sequenceDiagram
participant LS as Loop Start
participant OP as Operator using $K
participant LE as Loop End
participant C as Coordinator
LS->>OP: the rows, with K = 2
OP->>LE: the result rows, with K = 2
Note over LE: Update: K = 3<br/>Condition: 3 <= 10 is true
LE->>C: run the block again
LE->>LS: save K = 3 for Loop Start
C->>LS: start the next iteration
```
Today each iteration runs the block again on new workers, and a workflow with a
loop runs in Materialized mode.
## Roadmap
```mermaid
flowchart LR
A["<b>On main</b><br/>Loop Start and Loop End<br/>$name in Java and Scala"]
B["<b>In progress</b><br/>Control variables in Python<br/>Control-variable
port<br/>Try/catch block"]
C["<b>To discuss</b><br/>Stopping on body results<br/>More blocks, such as
while<br/>Faster iterations<br/>User docs"]
A --> B --> C
classDef done fill:#dafbe1,stroke:#1a7f37,color:#1f2328
classDef wip fill:#fff8c5,stroke:#9a6700,color:#1f2328
classDef todo fill:#f6f8fa,stroke:#59636e,color:#1f2328
class A done
class B wip
class C todo
```
- On main: loops (#5700) and `$name` in Java and Scala operators (#8639, #8750,
#8640)
- In progress: Python (#8642, #8671), the port (#8915), try/catch (#7387)
Feature details can stay in their own threads.
GitHub link: https://github.com/apache/texera/discussions/9263
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]