Re: managedMemoryInMB failure

2020-01-08 Thread Xintong Song
Hi Fanbin, > On YARN setups, this value is automatically configured to the size of the > TaskManager's YARN container, minus a certain tolerance value. > If I understand correctly, you are running Flink standalone cluster both in docker and on EMR? If that is the case, then this sentence has noth

Re: managedMemoryInMB failure

2020-01-08 Thread Fanbin Bu
Xintong, Thanks for looking into this. I changed docker setting of #CPUs to a lower number and it works now. I was using the same code and same flink version. The reason that it works on EMR is that I'm using a machine with large memory. According to the doc: *JVM heap size for the TaskManagers, w

Re: managedMemoryInMB failure

2020-01-07 Thread Xintong Song
Hi Fanbin, The blink planner batch sql operators requires managed memory, and the amount of managed memory needed depends on your job. The failure is because the slot, according to your cluster configurations, does not have enough managed memory to fulfill the requests. To fix the problem, you wo

managedMemoryInMB failure

2020-01-07 Thread Fanbin Bu
Hi, with Flink 1.9 running in docker mode, I have a batch job and got the following error message. However, it works totally fine with the same code on EMR. I checked the log and here is the only difference: managedMemoryInMB=138 . (the working ones has 0 value) did anybody see this before? Than