Thanks for the update Steffen. I'll try to take a look at it asap.
Cheers,
Till
On Fri, Aug 20, 2021 at 1:34 PM Hausmann, Steffen wrote:
> Hi Till,
>
> I've updated the wiki page as per the discussion on flip-177. I hope it
> makes more sense now.
>
> Cheers, Steffen
>
> On 16.07.21, 18:28, "T
Hi Till,
I've updated the wiki page as per the discussion on flip-177. I hope it makes
more sense now.
Cheers, Steffen
On 16.07.21, 18:28, "Till Rohrmann" wrote:
CAUTION: This email originated from outside of the organization. Do not
click links or open attachments unless you can confir
Hi Guowei,
That's a good question. FLIP-171 was mainly motivated by at-least-once
sinks (Kinesis, DynamoDB, Timestream, Firehose). The idea is to expand that
interface in the future to also support exactly-once.
I'm currently drafting FLIP-177 (the FLIP that enables async pattern in the
first pla
Hi,
I'm very sorry to participate in this discussion so late. But I have a
little question. I understand the goal of this FLIP is to make `Writer`
support asynchronous. But my question is: why not let `Committer` support
asynchronization? If only `Writer` supports asynchronization, ExactlyOnce
is i
Sure, thanks for the pointers.
Cheers,
Till
On Fri, Jul 16, 2021 at 6:19 PM Hausmann, Steffen
wrote:
> Hi Till,
>
> You are right, I’ve left out some implementation details, which have
> actually changed a couple of time as part of the ongoing discussion. You
> can find our current prototype he
Hi Till,
You are right, I’ve left out some implementation details, which have actually
changed a couple of time as part of the ongoing discussion. You can find our
current prototype here [1] and a sample implementation of the KPL free Kinesis
sink here [2].
I plan to update the FLIP. But I thi
Hi Steffen,
I've taken another look at the FLIP and I stumbled across a couple of
inconsistencies. I think it is mainly because of the lacking code. For
example, it is not fully clear to me based on the current FLIP how we
ensure that there are no in-flight requests when
AsyncSinkWriter.snapshotSt
Thanks for addressing this issue :)
Best, Piotrek
wt., 29 cze 2021 o 17:58 Hausmann, Steffen napisał(a):
> Hey Poitr,
>
> I've just adapted the FLIP and changed the signature for the
> `submitRequestEntries` method:
>
> protected abstract void submitRequestEntries(List
> requestEntries, ResultF
Hey Poitr,
I've just adapted the FLIP and changed the signature for the
`submitRequestEntries` method:
protected abstract void submitRequestEntries(List
requestEntries, ResultFuture requestResult);
In addition, we are likely to use an AtomicLong to track the number of
outstanding requests, as
Hi Piotr,
I’m happy to take your guidance on this. I need to think through your proposals
and I’ll follow-up on Monday with some more context so that we can close the
discussion on these details. But for now, I’ll close the vote.
Thanks, Steffen
From: Piotr Nowojski
Date: Friday, 25. June 202
Hey,
I've just synced with Arvid about a couple of more remarks from my side and
he shared mine concerns.
1. I would very strongly recommend ditching `CompletableFuture ` from
the `protected abstract CompletableFuture
submitRequestEntries(List requestEntries);` in favor of
something like
`org.a
Adding the InterruptException to the write method would make it explicit
that the write call can block but must react to interruptions (e.g. when
Flink wants to cancel the operation). I think this makes the contract a bit
clearer.
I think starting simple and then extending the API as we see the ne
Hey,
Agreed on starting with a blocking `write`. I've adapted the FLIP accordingly.
For now I've chosen to add the `InterruptedException` to the `write` method
signature as I'm not fully understanding the implications of swallowing the
exception. Depending on the details of the code that is ca
Hi,
Thanks Steffen for the explanations. I think it makes sense to me.
Re Arvid/Steffen:
- Keep in mind that even if we choose to provide a non blocking API using
the `isAvailable()`/`getAvailableFuture()` method, we would still need to
support blocking inside the sinks. For example at the very
Hi Piotr,
to pick up this discussion thread again:
- This FLIP is about providing some base implementation for FLIP-143 sinks
that make adding new implementations easier, similar to the
SourceReaderBase.
- The whole availability topic will most likely be a separate FLIP. The
basic issue just poppe
Hi Danny,
to add I'd propose to use the flink-connector-base package which has the
rough equivalent on source-side SourceReaderBase [1]. Since it's such a
handy base implementation, I'd like to see it directly in the main flink
repository.
For the actual connectors, I'm currently working on a pro
Hi Danny,
Right now, I'd expect the core of the Async Sink (without third party
dependencies) to live in its own submodule. For instance
`flink-connector-async` as part of `flink-connectors`.
I'm currently planning to implement three different sinks to verify that the
design of the sink if fle
Hey Steffen,
I have a few questions regarding the FLIP:
1. Where do you expect the core code to live, would it be in an existing module
(say flink-clients) or would you introduce a new module?
2. Which destination implementations do you intend to ship with this FLIP? I
see an example with Kines
Hey Piotrek,
Thanks for your comments on the FLIP. I'll address your second question first,
as I think it's more central to this FLIP. Just looking at the AWS ecosystem,
there are several sinks with overlapping functionality. I've chosen AWS sinks
here because I'm most familiar with those, but
Hi Steffen,
Thanks for writing down the proposal. Back when the new Sink API was being
discussed, I was proposing to add our usual `CompletableFuture
isAvailable()` pattern to make sinks non-blocking. You can see the
discussion starting here [1], and continuing for a couple of more posts
until her
20 matches
Mail list logo
