I would need to restart it to be sure (and when it starts to be stuck, the
web interface doesn't give the backpressure anymore), but it seems so. I
put a text file as the output and it took 5h to complete:
aggregates.writeAsText("hdfs:///user/christophe/flinkHBase");
What is weird is that I have a
Do the backpressure metrics indicate that the sink function is blocking?
2016-06-13 16:58 GMT+02:00 Christophe Salperwyck <
christophe.salperw...@gmail.com>:
> To continue, I implemented the ws.apply(new SummaryStatistics(), new
> YourFoldFunction(), new YourWindowFunction());
>
> It works fine w
To continue, I implemented the ws.apply(new SummaryStatistics(), new
YourFoldFunction(), new YourWindowFunction());
It works fine when there is no sink, but when I put an HBase sink it seems
that the sink, somehow, blocks the flow. The sink writes very little data
into HBase and when I limit my in
Thanks!
On Mon, Jun 13, 2016 at 12:34 PM, Christophe Salperwyck
wrote:
> Hi,
> I vote on this issue and I agree this would be nice to have.
>
> Thx!
> Christophe
>
> 2016-06-13 12:26 GMT+02:00 Aljoscha Krettek :
>>
>> Hi,
>> I'm afraid this is currently a shortcoming in the API. There is this ope
Hi,
I vote on this issue and I agree this would be nice to have.
Thx!
Christophe
2016-06-13 12:26 GMT+02:00 Aljoscha Krettek :
> Hi,
> I'm afraid this is currently a shortcoming in the API. There is this open
> Jira issue to track it: https://issues.apache.org/jira/browse/FLINK-3869.
> We can't
Hi Max,
In fact the Put would be the output of my WindowFunction. I saw Aljoscha
replied, seems I will need to create another intermediate class to handle
that. But it is fine.
Thx for help!
Christophe
2016-06-13 12:25 GMT+02:00 Maximilian Michels :
> Hi Christophe,
>
> A fold function has two
Hi,
I'm afraid this is currently a shortcoming in the API. There is this open
Jira issue to track it: https://issues.apache.org/jira/browse/FLINK-3869.
We can't fix it before Flink 2.0, though, because we have to keep the API
stable on the Flink 1.x release line.
Cheers,
Aljoscha
On Mon, 13 Jun 2
Hi Christophe,
A fold function has two inputs: The state and a record to update the
state with. So you can update the SummaryStatistics (state) with each
Put (input).
Cheers,
Max
On Mon, Jun 13, 2016 at 11:04 AM, Christophe Salperwyck
wrote:
> Thanks for the feedback and sorry that I can't try
Thanks for the feedback and sorry that I can't try all this straight away.
Is there a way to have a different function than:
WindowFunction()
I would like to return a HBase Put and not a SummaryStatistics. So
something like this:
WindowFunction()
Christophe
2016-06-09 17:47 GMT+02:00 Fabian Hue
OK, this indicates that the operator following the source is a bottleneck.
If that's the WindowOperator, it makes sense to try the refactoring of the
WindowFunction.
Alternatively, you can try to run that operator with a higher parallelism.
2016-06-09 17:39 GMT+02:00 Christophe Salperwyck <
chris
Hi Fabian,
Thanks for the help, I will try that. The backpressure was on the source
(HBase).
Christophe
2016-06-09 16:38 GMT+02:00 Fabian Hueske :
> Hi Christophe,
>
> where does the backpressure appear? In front of the sink operator or
> before the window operator?
>
> In any case, I think you
Hi Christophe,
where does the backpressure appear? In front of the sink operator or before
the window operator?
In any case, I think you can improve your WindowFunction if you convert
parts of it into a FoldFunction.
The FoldFunction would take care of the statistics computation and the
WindowFun
12 matches
Mail list logo
