Hey everyone,
During PR reviews, we saw that there are multiple valid alternatives for a
dynamic destinations configuration with regards to naming and positioning.
@Robert Bradshaw put together a doc [1] enumerating
some of these options -- we hope to get some feedback on what looks best to
you!
Big thanks to everyone for the ongoing discussions in this thread and on
the doc!
The implementation to enable portable dynamic destinations is now underway
- GitHub tracker: https://github.com/apache/beam/issues/32365
Best,
Ahmed
On Wed, Apr 3, 2024 at 1:00 PM Robert Bradshaw via dev
wrote:
>
On Wed, Apr 3, 2024 at 4:15 AM Kenneth Knowles wrote:
> Let me summarize the most recent proposal on-list to frame my question
> about this last suggestion. It looks like this:
>
> 1. user has an element, call it `data`
> 2. user maps `data` to an arbitrary metadata row, call it `dest`
> 3. we ca
Let me summarize the most recent proposal on-list to frame my question
about this last suggestion. It looks like this:
1. user has an element, call it `data`
2. user maps `data` to an arbitrary metadata row, call it `dest`
3. we can do things like shuffle on `dest` because it isn't too big
4. we m
I do suspect that over time we'll find more and more cases we can't
express, and will be asked to extend this little templating in more
directions. To head that off - could we easily just reuse an existing
language (SQL, LUA, something of the form?) instead of creating something
new?
On Tue, Apr 2
I really like this proposal. I think it has narrowed down and solved the
essential problem of not shuffling excess redundant data, and also provides
the vast majority of the functionality that a lambda would, with
significantly better debugability and usability too, since the dynamic
destination pa
On Wed, Mar 27, 2024 at 10:20 AM Reuven Lax wrote:
> Can the prefix still be generated programmatically at graph creation time?
>
Yes. It's just a property of the transform passed by the user at
configuration time.
> On Wed, Mar 27, 2024 at 9:40 AM Robert Bradshaw
> wrote:
>
>> On Wed, Mar 27
Can the prefix still be generated programmatically at graph creation time?
On Wed, Mar 27, 2024 at 9:40 AM Robert Bradshaw wrote:
> On Wed, Mar 27, 2024 at 9:12 AM Reuven Lax wrote:
>
>> This does seem like the best compromise, though I think there will still
>> end up being performance issues.
On Wed, Mar 27, 2024 at 9:12 AM Reuven Lax wrote:
> This does seem like the best compromise, though I think there will still
> end up being performance issues. A common pattern I've seen is that there
> is a long common prefix to the dynamic destination followed the dynamic
> component. e.g. the
> This does seem like the best compromise, though I think there will still
end up being performance issues. A common pattern I've seen is that there
is a long common prefix to the dynamic destination followed the dynamic
component. e.g. the destination might be
long/common/path/to/destination/files
This does seem like the best compromise, though I think there will still
end up being performance issues. A common pattern I've seen is that there
is a long common prefix to the dynamic destination followed the dynamic
component. e.g. the destination might be
long/common/path/to/destination/files/.
Thanks for putting this together, it will be a really useful feature to
have.
I am in favor of the string-pattern approaches. I think we need to support
both the {record=..., dest_info=...} and the elide-fields approaches, as
the former is nicer when one has a fixed representation for the
output r
12 matches
Mail list logo
