Just a quick reply You're right that you use the Beam coders to interpret the bytes. You just need an implementation of them in rust (and in every language building Beam components).
For the Prism runner (written in Go, default for Python and Go) we use the Go SDK coder implementations, because they were already present. But not every thing makes sense to use directly from the SDK within a runner context. https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/prism/internal/coders.go For example, for timers, it made more sense to reimplement certain portions within the engine portion of prism, than to route towards SDK constructs. https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/prism/internal/engine/timers.go#L135 I wrote up the flow that Prism uses for managing bundles here. There are flowcharts that provide the broad strokes, and I hope they are useful to someone building their own runner. https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/prism/internal/README.md Finally, while the Beam Pipeline Protos and runner APIs dictate how things communicate between runners and SDKs. The rest is up to you. I have a languishing "hobby" Go SDK that uses a different approach to coders to make them easier to deal with and manipulate, vs just the bytestream approach. https://github.com/lostluck/beam-go/blob/main/coders/coders.go It mostly wraps a byte buffer, and then allows callers to pop or push values to it. But this ends up playing very well for the garbage collector in Go. Rust has different constraints and problems to solve, so don't feel constrained by how the other languages do it. Hope this helps, and let me know if you have questions. Robert Burke On Tue, May 19, 2026, 5:29 AM Ganesh Sivakumar <[email protected]> wrote: > Hey Everyone, > > I am working on a new Rust based portable Beam Runner and I'm at pipeline > execution phase where the Rust runner side needs to > communicate with the worker sdk harness(Java) and execute the stages( > stages are nothing but a set of fused transforms, formed using the greedy > fusion approach from Beam Java utils, rewritten in Rust for the runner) > > For the stage to run on a worker, the runner needs to register the stage > information with the worker and then send a run request via grpc channels. > The worker will execute the transforms in the stage and send the output > back to runner via data grpc channel in the form of Elements [1] Elements > contain the output data as raw bytes which the runner needs to decode to > get the actual data like String, Int or POJO. Typically other runners do it > with Beam's defined coders for encoding and decoding. But for Rust there > isn't Beam coders implementation. Curious if anyone previously worked on > coders for cross languages, or similar things, and how did you implement in > general. > > Thanks, > Ganesh. > > [1] - > https://github.com/apache/beam/blob/55eb624e5cd00e546ab19fc411281a0e5f596142/model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto#L724 >
