Hi all, I've been working on an Apache Beam SDK in Rust, building on the ideas from the 2023 hackathon and design notes [1, 2], and I wanted to share the design doc and current progress on the implementation with the list:
- Design doc <https://docs.google.com/document/d/1M4ZsHNobSt526__rtcwp7K0ULvL9sAKnKBt39CcA7Vw/view> . - Code (branch `feat/rust-sdk` <https://github.com/prodriguezdefino/beam-pabs/tree/feat/rust-sdk/sdks/rust> ). The doc goes through the main design choices (targeting portability with Prism and Dataflow Runner v2, static typing and coders, the `DoFn` trait, and how the worker locates user code) and what is currently implemented and minimally validated on the branch. First of all, thanks for your attention =) I'd love to hear if there is interest in the community in having this contributed to Beam, and get general thoughts on the approach. Section 6 of the doc also lists a few open topics, if anyone wants to chime in: - worker code registration, - `DoFn` ergonomics, - whether it should live in the main repo or a separate one, - and a proposed stacked-PR plan for review. Any comments on the doc or replies on this thread are very welcome! Thanks, Pablo [1] https://lists.apache.org/thread/xg9xq0btp8k1wh2v1gpqyfhwpsyxq4ds [2] https://lists.apache.org/thread/gwnn0l9qq3x8svjbwmsr8cykq8600vhz
