Sure, we could definitely include things as a submodule for stuff like
testing multi-language, though I think there's actually a cleaner way just
using the Swift package manager's test facilities to access the swift sdk
repo.
That would also be consistent with the user-side experience and let us
On Wed, Sep 20, 2023 at 9:54 AM Byron Ellis wrote:
> Hi all,
>
> I've chatted with a couple of people offline about this and my impression
> is that folks are generally amenable to a separate repo to match the target
> community? I have no idea what the next steps would be though other than
> gue
Hi all,
I've chatted with a couple of people offline about this and my impression
is that folks are generally amenable to a separate repo to match the target
community? I have no idea what the next steps would be though other than
guessing that there's probably some sort of PMC thing involved? Sho
Hi all,
I've been on vacation, but mostly working on getting External Transform
support going (which in turn basically requires Schema support as well). It
also looks like macros landed in Swift 5.9 for Linux so we'll be able to
use those to do some compile-time automation. In particular, this let
After a couple of iterations (thanks rebo!) we've also gotten the Swift SDK
working with the new Prism runner. The fact that it doesn't do fusion
caught a couple of configuration bugs (e.g. that the grpc message receiver
buffer should be fairly large). It would seem that at the moment Prism and
the
Okay, after a brief detour through "get this working in the Flink Portable
Runner" I think I have something pretty workable.
PInput and POutput can actually be structs rather than protocols, which
simplifies things quite a bit. It also allows us to use them with property
wrappers for a SwiftUI-lik
Okay, I tried a couple of different things.
Implicitly passing the timestamp and window during iteration did not go
well. While physically possible it introduces an invisible side effect into
loop iteration which confused me when I tried to use it and I implemented
it. Also, I'm pretty sure there'
Gotcha, I think there's a fairly easy solution to link input and output
streams Let me try it out... might even be possible to have both
element and stream-wise closure pardos. Definitely possible to have that at
the DoFn level (called SerializableFn in the SDK because I want to
use @DoFn as a
On Thu, Aug 24, 2023 at 12:58 PM Chamikara Jayalath
wrote:
>
>
> On Thu, Aug 24, 2023 at 12:27 PM Robert Bradshaw
> wrote:
>
>> I would like to figure out a way to get the stream-y interface to work,
>> as I think it's more natural overall.
>>
>> One hypothesis is that if any elements are carrie
On Thu, Aug 24, 2023 at 12:27 PM Robert Bradshaw
wrote:
> I would like to figure out a way to get the stream-y interface to work, as
> I think it's more natural overall.
>
> One hypothesis is that if any elements are carried over loop iterations,
> there will likely be some that are carried over
I would like to figure out a way to get the stream-y interface to work, as
I think it's more natural overall.
One hypothesis is that if any elements are carried over loop iterations,
there will likely be some that are carried over beyond the loop (after all
the callee doesn't know when the loop is
Ah, that is a good point—being element-wise would make managing windows and
time stamps easier for the user. Fortunately it’s a fairly easy change to
make and maybe even less typing for the user. I was originally thinking
side inputs and metrics would happen outside the loop, but I think you want
a
Ah, I see.
Yeah, I've thought about using an iterable for the whole bundle rather than
start/finish bundle callbacks, but one of the questions is how that would
impact implicit passing of the timestamp (and other) metadata from
input elements to output elements. (You can of course attach the metad
Oh, I also forgot to mention that I included element-wise collection
operations like "map" that eliminate the need for pardo in many cases. the
groupBy command is actually a map + groupByKey under the hood. That was to
be more consistent with Swift's collection protocol (and is also why
PCollection
Indeed :-) Yeah, I went back and forth on the pardo syntax quite a bit
before settling on where I ended up. Ultimately I decided to go with
something that felt more Swift-y than anything else which means that rather
than dealing with a single element like you do in the other SDKs you're
dealing wit
For everyone who is interested, here's the draft PR:
https://github.com/apache/beam/pull/28062
I haven't had a chance to test it on my M1 machine yet though (there's a
good chance there are a few places that need to properly address
endianness. Specifically timestamps in windowed values and lengt
Thanks Cham,
Definitely happy to open a draft PR so folks can comment---there's not as
much code as it looks like since most of the LOC is just generated
protobuf. As for the support, I definitely want to add external transforms
and may actually add that support before adding the ability to make
c
Thanks Byron. This sounds great. I wonder if there is interest in Swift SDK
from folks currently subscribed to the +user list.
On Wed, Aug 16, 2023 at 6:53 PM Byron Ellis via dev
wrote:
> Hello everyone,
>
> A couple of months ago I decided that I wanted to really understand how
> the Beam FnAp
Hello everyone,
A couple of months ago I decided that I wanted to really understand how the
Beam FnApi works and how it interacts with the Portable Runner. For me at
least that usually means I need to write some code so I can see things
happening in a debugger and to really prove to myself I under
19 matches
Mail list logo
