I would need to restart it to be sure (and when it starts to be stuck, the
web interface doesn't give the backpressure anymore), but it seems so. I
put a text file as the output and it took 5h to complete:
aggregates.writeAsText("hdfs:///user/christophe/flinkHBase");
What is weird is that I have a
Do the backpressure metrics indicate that the sink function is blocking?
2016-06-13 16:58 GMT+02:00 Christophe Salperwyck <
christophe.salperw...@gmail.com>:
> To continue, I implemented the ws.apply(new SummaryStatistics(), new
> YourFoldFunction(), new YourWindowFunction());
>
> It works fine w
To continue, I implemented the ws.apply(new SummaryStatistics(), new
YourFoldFunction(), new YourWindowFunction());
It works fine when there is no sink, but when I put an HBase sink it seems
that the sink, somehow, blocks the flow. The sink writes very little data
into HBase and when I limit my in
Thanks!
On Mon, Jun 13, 2016 at 12:34 PM, Christophe Salperwyck
wrote:
> Hi,
> I vote on this issue and I agree this would be nice to have.
>
> Thx!
> Christophe
>
> 2016-06-13 12:26 GMT+02:00 Aljoscha Krettek :
>>
>> Hi,
>> I'm afraid this is currently a shortcoming in the API. There is this ope
Hi,
I vote on this issue and I agree this would be nice to have.
Thx!
Christophe
2016-06-13 12:26 GMT+02:00 Aljoscha Krettek :
> Hi,
> I'm afraid this is currently a shortcoming in the API. There is this open
> Jira issue to track it: https://issues.apache.org/jira/browse/FLINK-3869.
> We can't
Hi Max,
In fact the Put would be the output of my WindowFunction. I saw Aljoscha
replied, seems I will need to create another intermediate class to handle
that. But it is fine.
Thx for help!
Christophe
2016-06-13 12:25 GMT+02:00 Maximilian Michels :
> Hi Christophe,
>
> A fold function has two
Hi,
I'm afraid this is currently a shortcoming in the API. There is this open
Jira issue to track it: https://issues.apache.org/jira/browse/FLINK-3869.
We can't fix it before Flink 2.0, though, because we have to keep the API
stable on the Flink 1.x release line.
Cheers,
Aljoscha
On Mon, 13 Jun 2
Hi Christophe,
A fold function has two inputs: The state and a record to update the
state with. So you can update the SummaryStatistics (state) with each
Put (input).
Cheers,
Max
On Mon, Jun 13, 2016 at 11:04 AM, Christophe Salperwyck
wrote:
> Thanks for the feedback and sorry that I can't try
Thanks for the feedback and sorry that I can't try all this straight away.
Is there a way to have a different function than:
WindowFunction()
I would like to return a HBase Put and not a SummaryStatistics. So
something like this:
WindowFunction()
Christophe
2016-06-09 17:47 GMT+02:00 Fabian Hue
OK, this indicates that the operator following the source is a bottleneck.
If that's the WindowOperator, it makes sense to try the refactoring of the
WindowFunction.
Alternatively, you can try to run that operator with a higher parallelism.
2016-06-09 17:39 GMT+02:00 Christophe Salperwyck <
chris
Hi Fabian,
Thanks for the help, I will try that. The backpressure was on the source
(HBase).
Christophe
2016-06-09 16:38 GMT+02:00 Fabian Hueske :
> Hi Christophe,
>
> where does the backpressure appear? In front of the sink operator or
> before the window operator?
>
> In any case, I think you
Hi Christophe,
where does the backpressure appear? In front of the sink operator or before
the window operator?
In any case, I think you can improve your WindowFunction if you convert
parts of it into a FoldFunction.
The FoldFunction would take care of the statistics computation and the
WindowFun
Hi,
I am writing a program to read timeseries from HBase and do some daily
aggregations (Flink streaming). For now I am just computing some average so
not very consuming but my HBase read get slower and slower (I have few
billions of points to read). The back pressure is almost all the time close
13 matches
Mail list logo
