GitHub user aglinxinyuan edited a discussion: Control blocks in Texera: design 
and roadmap

Loops are now on Texera's main branch (#5700). Each existing thread covers one 
detail, such as `$name` in properties (#8643) or the control-variable port 
(#8915). This thread is for the big picture: what a control block is, where it 
stands, and what to build next.

## What and why

An operator normally runs once, with its settings fixed before the run. Some 
tasks need part of a workflow to run again with a new value each time, for 
example trying K = 2 to 10 to pick the best K, or processing every file in a 
folder.

A control block is that part of the workflow. A start operator and an end 
operator enclose a body of ordinary operators, and short Python code on the two 
decides how the body runs. Loops are the first kind.

```mermaid
flowchart LR
  D[Data] --> LS
  subgraph blk [Control block: a loop]
    LS["Loop Start<br/>K = 2"] --> A["Body operators<br/>a setting uses $K"] 
--> LE["Loop End<br/>K += 1<br/>while K <= 10"]
    LE -. next iteration .-> LS
  end
  LE --> R["Results of<br/>all iterations"]
```

## Key ideas

- **The body stays on the canvas.** Users build and inspect it like any other 
part of the workflow.
- **Control variables** such as `K` or a file path change between iterations. A 
property refers to one as `$K`, so existing operators work unchanged.
- **Values travel with the data.** Each iteration's variables move along the 
same edges as its rows, using Texera's state passing (#2802). They reach every 
operator in the body before its first row. The reasons are in #7429.

```mermaid
sequenceDiagram
  participant LS as Loop Start
  participant OP as Operator using $K
  participant LE as Loop End
  participant C as Coordinator
  LS->>OP: the rows, with K = 2
  OP->>LE: the result rows, with K = 2
  Note over LE: Update: K = 3<br/>Condition: 3 <= 10 is true
  LE->>C: run the block again
  LE->>LS: save K = 3 for Loop Start
  C->>LS: start the next iteration
```

Today each iteration runs the block again on new workers, and a workflow with a 
loop runs in Materialized mode.

## Roadmap

```mermaid
flowchart LR
  A["<b>On main</b><br/>Loop Start and Loop End<br/>$name in Java and Scala"]
  B["<b>In progress</b><br/>Control variables in Python<br/>Control-variable 
port<br/>Try/catch block"]
  C["<b>To discuss</b><br/>Stopping on body results<br/>More blocks, such as 
while<br/>Faster iterations<br/>User docs"]
  A --> B --> C
  classDef done fill:#dafbe1,stroke:#1a7f37,color:#1f2328
  classDef wip fill:#fff8c5,stroke:#9a6700,color:#1f2328
  classDef todo fill:#f6f8fa,stroke:#59636e,color:#1f2328
  class A done
  class B wip
  class C todo
```

- On main: loops (#5700) and `$name` in Java and Scala operators (#8639, #8750, 
#8640)
- In progress: Python (#8642, #8671), the port (#8915), try/catch (#7387)

Feature details can stay in their own threads.


GitHub link: https://github.com/apache/texera/discussions/9263

----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Reply via email to