Many thanks Fabian for your prompt replies much appreciated
Thanks
Declan
On Wed, Oct 6, 2021 at 8:38 AM Fabian Paul wrote:
> Hi Declan,
>
> As far as I know the FileSink does not buffer records but writes the
> records to temporary files which are bucketed later. For the Elasticsearch
> sink
>
Hi Declan,
As far as I know the FileSink does not buffer records but writes the records to
temporary files which are bucketed later. For the Elasticsearch sink
you are right it buffers the records before flushing them to ElasticSearch but
you can control the flushing behaviour based on a given
Hi Fabian
I am currently using the streaming file sink to local disk though
potentially this sink could change to be Elastic Search.
Declan
On Mon, Oct 4, 2021 at 1:16 PM Fabian Paul wrote:
> Hi Declan,
>
> I forgot to ask which sink you are using. I do not think it is generally
> applicable t
Hi Declan,
I forgot to ask which sink you are using. I do not think it is generally
applicable that all sinks buffer records and only send them periodically. It
depends a lot on the connector and what kind of capabilities the external
system you are writing to offers.
The amount of buffered da
Hi Fabian
Primarily more a case of understanding how many records are likely to be
buffered by the sink still awaiting processing. So we are streaming event
records to a sink for downstream processing in as close to real time as
possible but wondered how many might be buffered by Flink and if that
Hi Declan,
Thanks for reaching out, we always welcome new users to Apache Flink community
:)
Your first question is a bit tricky. I am still trying to understand the
motivation behind. In general there is no generic way to access the records
which one of the operator currently processes.
Are
Hi Guys
I've just recently started using Apache Flink to evaluate its suitability
for a project I'm working on.
First impressions are that the project is great, well documented and has
lots of examples and guidance showcasing the multitude of things that it
can do. Challenging knowing where to s
Hi,
it is true that there is no dedicated machine learning library for
Flink. Flink is a general data processing framework. It allows to
embedded any available algorithm library within user-defined functions.
Flink's focus is on stream processing. There are not many dedicated
stream processi
Hello
I am a complete newbie and I need help. I am evaluating the usage of flink
for my academic study and reading the documentation. I have a bit of
experience in Apache Spark. I am asking this question, based on my
experience in Apache Spark.
In spark, there is a machine learning library embed
Hi Renato,
First of all to do anything together on the two streams you probably want
to union them. This means that you need to have a common type. If this is
the case you are lucky and you don't need anything else. Otherwise I
suggest using the Either type provided by Flink as a simple wrapper.
Hi all,
I have two streams in which I need to keep counts of different metrics that
will have to be shared by both of the streams. So they will be sharing some
state once they have finished processing the stream. My question is if I
should do this as a sink aggregating what I need at the end or by
11 matches
Mail list logo
