Thanks, the off-heap solution is indeed faster. 15s instead of 45s for the
amounts of memory I allocate.
On Fri, Oct 2, 2015 at 6:09 PM, Stephan Ewen wrote:
> Yeah, registration is fast, JVM heatup is what takes time.
>
> You can try two things:
>
> - Use the off-heap memory variant and see if
Yeah, registration is fast, JVM heatup is what takes time.
You can try two things:
- Use the off-heap memory variant and see if that allocates the memory
faster. Just add the entry "taskmanager.memory.off-heap: true" to the
config.
- Or start the system in "streaming" mode. Then, it will not
Looking into the logs of each TM it only took about 5 seconds per TM to go
from "Trying to register" to "Successful registration".
On Fri, Oct 2, 2015 at 5:50 PM, Robert Schmidtke
wrote:
> I recently switched from running Flink on YARN to running Flink Standalone
> and I realized I had to add a
I recently switched from running Flink on YARN to running Flink Standalone
and I realized I had to add a sleep after ./start-cluster.sh (well, my
Slurm adaptation of it). I did not have to explicitly wait before since
Flink would wait until all YARN containers became available, so to be
honest I do
Is that a new observation that it takes so long, or has it always taken so
long?
On Fri, Oct 2, 2015 at 5:40 PM, Robert Schmidtke
wrote:
> I figured the JM would be waiting for the TMs. Each of my nodes has 64G of
> memory available.
>
> On Fri, Oct 2, 2015 at 5:38 PM, Maximilian Michels wrote:
I figured the JM would be waiting for the TMs. Each of my nodes has 64G of
memory available.
On Fri, Oct 2, 2015 at 5:38 PM, Maximilian Michels wrote:
> Hi Robert,
>
> During startup, the task manager allocates the entire managed memory.
>
> From the log:
> 17:03:33,554 INFO org.apache.flink.ru
Hi Robert,
During startup, the task manager allocates the entire managed memory.
>From the log:
17:03:33,554 INFO org.apache.flink.runtime.taskmanager.TaskManager
- Using 0.7 of the currently free heap space for Flink
managed heap memory (34395 MB).
It seems like you are allocating al
Yes, they're both set to the same value. These are the JVM Options as
reported by the TM:
17:36:12,137 INFO org.apache.flink.runtime.taskmanager.TaskManager
- JVM Options:
17:36:12,137 INFO org.apache.flink.runtime.taskmanager.TaskManager
- -Xms51355M
17:36:12,137 INFO org.apache
The delay you see happens when the TaskManager allocates the memory for its
memory manager. Allocating that much in a JVM can take a bit, although 40
seconds looks a lot to me...
How do you start the JVM? Are Xmx and Xms set to the same value? If not,
the JVM incrementally grows through multiple g
