Many thanks Fabian for your prompt replies much appreciated
Thanks
Declan
On Wed, Oct 6, 2021 at 8:38 AM Fabian Paul wrote:
> Hi Declan,
>
> As far as I know the FileSink does not buffer records but writes the
> records to temporary files which are bucketed later. For the Elasticsearch
> sink
>
Hi Declan,
As far as I know the FileSink does not buffer records but writes the records to
temporary files which are bucketed later. For the Elasticsearch sink
you are right it buffers the records before flushing them to ElasticSearch but
you can control the flushing behaviour based on a given
Hi Fabian
I am currently using the streaming file sink to local disk though
potentially this sink could change to be Elastic Search.
Declan
On Mon, Oct 4, 2021 at 1:16 PM Fabian Paul wrote:
> Hi Declan,
>
> I forgot to ask which sink you are using. I do not think it is generally
> applicable t
Hi Declan,
I forgot to ask which sink you are using. I do not think it is generally
applicable that all sinks buffer records and only send them periodically. It
depends a lot on the connector and what kind of capabilities the external
system you are writing to offers.
The amount of buffered da
Hi Fabian
Primarily more a case of understanding how many records are likely to be
buffered by the sink still awaiting processing. So we are streaming event
records to a sink for downstream processing in as close to real time as
possible but wondered how many might be buffered by Flink and if that
Hi Declan,
Thanks for reaching out, we always welcome new users to Apache Flink community
:)
Your first question is a bit tricky. I am still trying to understand the
motivation behind. In general there is no generic way to access the records
which one of the operator currently processes.
Are
Hi Guys
I've just recently started using Apache Flink to evaluate its suitability
for a project I'm working on.
First impressions are that the project is great, well documented and has
lots of examples and guidance showcasing the multitude of things that it
can do. Challenging knowing where to s
