I am trying out Flink. I think Flink must not run out of memory. What should I
do? Same code run in Hadoop. Why Flink ran out of memory?
I'm curios to learn more, maybe we can discuss this over a coffee in the
next days ;)
2015-04-27 12:16 GMT+02:00 Kirschnick, Johannes <
johannes.kirschn...@tu-berlin.de>:
> Hi,
>
> thanks for dedicating some of your weekend time to this problem.
> Your solution looks quite neat, thanks ..
>
> For
Vasia Kalavri created FLINK-1951:
Summary: NullPointerException in DeltaIteration when no
ForwardedFileds
Key: FLINK-1951
URL: https://issues.apache.org/jira/browse/FLINK-1951
Project: Flink
Will do, thanks!
On 27 April 2015 at 11:06, Fabian Hueske wrote:
> No, haven't looked at it since my last mail :-(
> Both plans (with and without forward fields annotation) look good except
> for the suspicious pipeline breaker.
>
> @Vasia Could you open a JIRA and assign it to me?
> I'll have a
Robert Metzger created FLINK-1950:
-
Summary: Increase default heap cutoff ratio from 20% to 30% and
move default value to ConfigConstants
Key: FLINK-1950
URL: https://issues.apache.org/jira/browse/FLINK-1950
Thanks, Stephan.
So the follow-up question -
Is it possible to cast an operator from optimizers/common API to Java
API? ( as Java to Common is possible via translateToDataFlow())
For eg -
org.apache.flink.api.common.operators.base.JoinOperatorBase;
to
org.apache.flink.api.java.operators.JoinOperat
Robert Metzger created FLINK-1949:
-
Summary: YARNSessionFIFOITCase sometimes fails to detect when the
detached session finishes
Key: FLINK-1949
URL: https://issues.apache.org/jira/browse/FLINK-1949
Pr
Hi!
It seems you are actually working on the Optimizer's representation of the
program. The optimizer (and the runtime as well) think about Flink programs
in terms of data flows, not data sets any more (that is only in the
Java/Scala API).
The Java API operations get translated to the data flow o
Hi
For a given Operator, it is easily possible to find the input and output
TypeInformation,which gives the datatype and other information of the
dataset. For e.g.
if(operator instanceof SingleInputOperator){
inputTypes.add(((SingleInputOperator)operator).getOperatorInfo().getInputType());
}
For
Gyula Fora created FLINK-1948:
-
Summary: Add manual task slot handling for streaming jobs
Key: FLINK-1948
URL: https://issues.apache.org/jira/browse/FLINK-1948
Project: Flink
Issue Type: Improvem
I am not sure if I got everything right.
Let me first explain how a Reduce operator is executed in Flink at the
moment:
Assume a data set is randomly distributed across four machines and not
sorted.
When a Reduce function is applied on this data set, each of the four
machines run a Combine operat
Hi,
thanks for dedicating some of your weekend time to this problem.
Your solution looks quite neat, thanks ..
For your information, what I'm trying to do is multiply matrix blocks (dataset
one) with individual elements of a vector (dataset two, of row id and some
value).
In this case "one" n
Some form of tree aggregation is useful in many cases, and IMO a good
addition to the system.
Kostas
On Mon, Apr 27, 2015 at 11:04 AM, Andra Lungu wrote:
> Hi Fabian,
>
> After a quick look at the current behaviour of Flink's combinable reduce, I
> saw that it does something like this:
>
> http
No, haven't looked at it since my last mail :-(
Both plans (with and without forward fields annotation) look good except
for the suspicious pipeline breaker.
@Vasia Could you open a JIRA and assign it to me?
I'll have a closer look and try to figure out what's going on.
2015-04-27 10:34 GMT+02:0
Hi Fabian,
After a quick look at the current behaviour of Flink's combinable reduce, I
saw that it does something like this:
https://docs.google.com/drawings/d/1WfGJq1ZNQ-F0EQZ2TwEYS_861xc3fSdfJL9Z4VdXBQU/edit?usp=sharing
It basically iterates over all the key groups two by two and if it finds
tw
Hi,
Andra is working on a modified reduce operator, which would internally
create an aggregation tree.
This is work related to her thesis and we want to use it in graph
computations for skewed inputs.
It might or might not be a good idea to add it as a Flink operator and we
will need to evaluate t
I think Fabian looked into this a while back...
@Fabian, do you have any insights what causes this?
On Sat, Apr 25, 2015 at 7:46 PM, Vasiliki Kalavri wrote:
> Hi,
>
> I actually ran into this problem again with a different algorithm :/
> Same exception and it looks like getMatchFor() in Compac
Till Rohrmann created FLINK-1947:
Summary: Make Avro and Tachyon test logging less verbose
Key: FLINK-1947
URL: https://issues.apache.org/jira/browse/FLINK-1947
Project: Flink
Issue Type: Imp
Till Rohrmann created FLINK-1946:
Summary: Make yarn tests logging less verbose
Key: FLINK-1946
URL: https://issues.apache.org/jira/browse/FLINK-1946
Project: Flink
Issue Type: Improvement
Hi Andra,
is there a JIRA for the new runtime operator?
Adding a new operator is a lot of work and touches many core parts of the
system.
It would be good to start a discussion about that early in the process to
make sure that the design is aligned with the system.
Otherwise, duplicated work migh
Till Rohrmann created FLINK-1945:
Summary: Make python tests less verbose
Key: FLINK-1945
URL: https://issues.apache.org/jira/browse/FLINK-1945
Project: Flink
Issue Type: Improvement
That's a good solution. In order to deal with ranges which overlap two
intervals you have to create multiple "coarse-grained" join keys. One key
for each interval contained in the range.
Cheers,
Till
On Apr 26, 2015 11:22 PM, "Alexander Alexandrov" <
alexander.s.alexand...@gmail.com> wrote:
> I t
22 matches
Mail list logo
