Thanks Byron. This sounds great. I wonder if there is interest in Swift SDK from folks currently subscribed to the +user <user@beam.apache.org> list.
On Wed, Aug 16, 2023 at 6:53 PM Byron Ellis via dev <d...@beam.apache.org> wrote: > Hello everyone, > > A couple of months ago I decided that I wanted to really understand how > the Beam FnApi works and how it interacts with the Portable Runner. For me > at least that usually means I need to write some code so I can see things > happening in a debugger and to really prove to myself I understood what was > going on I decided I couldn't use an existing SDK language to do it since > there would be the temptation to read some code and convince myself that I > actually understood what was going on. > > One thing led to another and it turns out that to get a minimal FnApi > integration going you end up writing a fair bit of an SDK. So I decided to > take things to a point where I had an SDK that could execute a word count > example via a portable runner backend. I've now reached that point and > would like to submit my prototype SDK to the list for feedback. > > It's currently living in a branch on my fork here: > > https://github.com/byronellis/beam/tree/swift-sdk/sdks/swift > > At the moment it runs via the most recent XCode Beta using Swift 5.9 on > Intel Macs, but should also work using beta builds of 5.9 for Linux running > on Intel hardware. I haven't had a chance to try it on ARM hardware and > make sure all of the endian checks are complete. The > "IntegrationTests.swift" file contains a word count example that reads some > local files (as well as a missing file to exercise DLQ functionality) and > output counts through two separate group by operations to get it past the > "map reduce" size of pipeline. I've tested it against the Python Portable > Runner. Since my goal was to learn FnApi there is no Direct Runner at this > time. > > I've shown it to a couple of folks already and incorporated some of that > feedback already (for example pardo was originally called dofn when > defining pipelines). In general I've tried to make the API as "Swift-y" as > possible, hence the heavy reliance on closures and while there aren't yet > composite PTransforms there's the beginnings of what would be needed for a > SwiftUI-like declarative API for creating them. > > There are of course a ton of missing bits still to be implemented, like > counters, metrics, windowing, state, timers, etc. > This should be fine and we can get the code documented without these features. I think support for composites and adding an external transform (see, Java <https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/External.java>, Python <https://github.com/apache/beam/blob/c7b7921185686da573f76ce7320817c32375c7d0/sdks/python/apache_beam/transforms/external.py#L556>, Go <https://github.com/apache/beam/blob/c7b7921185686da573f76ce7320817c32375c7d0/sdks/go/pkg/beam/xlang.go#L155>, TypeScript <https://github.com/apache/beam/blob/master/sdks/typescript/src/apache_beam/transforms/external.ts>) to add support for multi-lang will bring in a lot of features (for example, I/O connectors) for free. > > Any and all feedback welcome and happy to submit a PR if folks are > interested, though the "Swift Way" would be to have it in its own repo so > that it can easily be used from the Swift Package Manager. > +1 for creating a PR (may be as a draft initially). Also it'll be easier to comment on a PR :) - Cham [1] [2] [3] > > Best, > B > > >
