Robert or Stephan know the Travis setup quite well.
They might know, if we can give a bit more than 80MB. But at some point
there will be a hard limit.
Once we have dynamic memory management (most of) such problems should be
solved.
2015-03-30 23:46 GMT+02:00 Andra Lungu :
> Oh! In that case, who
Oh! In that case, who should I refer to? :D
[It's kind of ugly to split this kind of test. I mean if a person is
counting the degrees, then that's the result that should be tested - at
least in my opinion]
In any case, thanks for the help :)
On Mon, Mar 30, 2015 at 11:37 PM, Fabian Hueske wrote:
Well, each combiner, reducer, join, coGroup, and solutionset needs a share
of memory (maps & filters don't).
In your case it was pretty much at the edge, the hash joins require 33
buffers and got 32. So one memory-consuming operator less might fix it.
I did not look in detail at the other job, but
Hi Fabian,
I'll see what I can do :).
I am just a bit shocked. If this set of coGroups and joins was too much for
a test case, how come the following worked?
https://github.com/andralungu/flink/commit/f60b022de056ac259459b68eee6ff0ae9993f0f8
400 lines of complex computations :) And I have an eve
Hi Andra,
I found the cause for the exception. Your test case is simply too complex
for our testing environment.
We restrict the TM memory for testcases to 80MB in order to execute
multiple tests in parallel on Travis.
I counted the memory consumers in your job and got:
- 2 Combine
- 4 GroupReduc
Sure,
It was in the first mail but that was sent a while ago :)
This is the code:
https://github.com/andralungu/gelly-partitioning/tree/alphaSplit
I also added the log4j file in case it helps!
The error is totally reproducible. 2 out of 2 people got the same.
Steps to reproduce:
1). Clone the co
Hmm, that is really weird.
Can you point me to a branch in your repository and the test case that
gives the error?
Then I have a look at it and try to figure out what's going wrong.
Cheers, Fabian
2015-03-30 10:43 GMT+02:00 Andra Lungu :
> Hello,
>
> I went on and did some further debugging on
Hello,
I went on and did some further debugging on this issue. Even though the
exception said that the problem comes from here:
4837 [Join(Join at* weighEdges(NodeSplitting.java:117)*) (1/4)] ERROR
org.apache.flink.runtime.operators.RegularPactTask - Error in task code:
Join(Join at weighEdges(No
Opps! Sorry! Did not know the mailing list does not support attachments :)
https://gist.github.com/andralungu/fba36d77f79189daa183
On Fri, Mar 27, 2015 at 12:02 AM, Andra Lungu wrote:
> Hi Fabian,
>
> I uploaded a file with my execution plan.
>
> On Thu, Mar 26, 2015 at 11:50 PM, Fabian Hueske
Hi Fabian,
I uploaded a file with my execution plan.
On Thu, Mar 26, 2015 at 11:50 PM, Fabian Hueske wrote:
> Hi Andra,
>
> the error is independent of the size of the data set. A HashTable needs at
> least 33 memory pages to operate.
> Since you have 820MB of managed memory and the size of a m
Hi Andra,
the error is independent of the size of the data set. A HashTable needs at
least 33 memory pages to operate.
Since you have 820MB of managed memory and the size of a memory page is
32KB, there should be more than 25k pages available.
Can you post the execution plan of the program you ex
For 20 edges and 5 nodes, that should be more thank enough.
On Thu, Mar 26, 2015 at 11:24 PM, Andra Lungu wrote:
> Sure,
>
> 3470 [main] INFO org.apache.flink.runtime.taskmanager.TaskManager -
> Using 820 MB for Flink managed memory.
>
> On Thu, Mar 26, 2015 at 4:48 PM, Robert Metzger
> wrote
Sure,
3470 [main] INFO org.apache.flink.runtime.taskmanager.TaskManager - Using
820 MB for Flink managed memory.
On Thu, Mar 26, 2015 at 4:48 PM, Robert Metzger wrote:
> Hi,
>
> during startup, Flink will log something like:
> 16:48:09,669 INFO org.apache.flink.runtime.taskmanager.TaskManage
Hi,
during startup, Flink will log something like:
16:48:09,669 INFO org.apache.flink.runtime.taskmanager.TaskManager
- Using 1193 MB for Flink managed memory.
Can you tell us how much memory Flink is managing in your case?
On Thu, Mar 26, 2015 at 4:46 PM, Andra Lungu wrote:
> Hello ev
Hi Stephan,
I tried the solution with DeltaIteration#setSolutionSetUnManaged(),
unfortunatelly the error is still there... Even when I try to run it with
just one iteration... Also, I am not sure that the job can be broken into
subparts in this particular case.
Any other suggestions would be appr
This is actually a problem of the number of memory segments available to
the hash table for the solution set.
For complex pipelines, memory currently gets too fragmented.
There are two workarounds, until we do the dynamic memory management, or
break it into shorter pipelines: Break the job up int
Hi Andra,
have you tried increasing the number of network buffers in your cluster?
You can control by the configuration value:
taskmanager.network.numberOfBuffers: #numberBuffers
Greets,
Till
On Mon, Feb 9, 2015 at 9:56 AM, Andra Lungu wrote:
> Hello everyone,
>
> I am implementing a graph a
17 matches
Mail list logo
