Hi Stephan, thanks for your support. I was able to track the problem a few days ago. Unirest was the one to blame, I was using it on some mapfuncionts to connect to external services and for some reason it was using insane amounts of virtual memory.
Paulo Cezar On Mon, Dec 19, 2016 at 11:30 AM Stephan Ewen <se...@apache.org> wrote: > Hi Paulo! > > Hmm, interesting. The high discrepancy between virtual and physical memory > usually means that the process either maps large files into memory, or that > it pre-allocates a lot of memory without immediately using it. > Neither of these things are done by Flink. > > Could this be an effect of either the Docker environment (mapping certain > kernel spaces / libraries / whatever) or a result of one of the libraries > (gRPC or so)? > > Stephan > > > On Mon, Dec 19, 2016 at 12:32 PM, Paulo Cezar <paulo.ce...@gogeo.io> > wrote: > > - Are you using RocksDB? > > No. > > > - What is your flink configuration, especially around memory settings? > > I'm using default config with 2GB for jobmanager and 5GB for taskmanagers. > I'm starting flink via "./bin/yarn-session.sh -d -n 5 -jm 2048 -tm 5120 -s > 4 -nm 'Flink'" > > - What do you use for TaskManager heap size? Any manual value, or do you > let Flink/Yarn set it automatically based on container size? > > No manual values here. YARN config is pretty much default with maximum > allocation of 12GB of physical memory and ratio between virtual memory to > physical memory 2.1 (via yarn.nodemanager.vmem-pmem-ratio). > > > - Do you use any libraries or connectors in your program? > > I'm using flink-connector-kafka-0.10_2.11, a MongoDB client, a gRPC > client and some http libraries like unirest and Apache HttpClient. > > - Also, can you tell us what OS you are running on? > > My YARN cluster runs on Docker containers (docker version 1.12) with > images based on Ubuntu 14.04. Host OS is Ubuntu 14.04.4 LTS (GNU/Linux > 3.19.0-65-generic x86_64). > > >
