Ignite-yarn[Ignition] uses almost all memory and vcores available in YARN cluster

[Murugar]

Hi,

Ignite-yarn[Ignition] uses almost all memory and vcores available in YARN
cluster. The allocated memory to the Ignition server was 40GB and 16 VCores. 

*When we checked the actual usage with yarn top command the Ignition server
is using 380 GB and 140 VCores!!! * When we checked the cache, it is having
only less number of records - 3000 records.

*Information about the application:*
We load data into Ignite Cache, process them periodically and delete
processed records periodically. 

We are running Ignite-yarn in Cloudera cluster with 5 data nodes with below
cluster.properties

IGNITE_NODE_COUNT=4
IGNITE_RUN_CPU_PER_NODE=4
IGNITE_MEMORY_PER_NODE=10096
IGNITE_VERSION=2.3.0
IGNITE_PATH=/apache-ignite-fabric-2.3.0-bin.zip

IGNITE_RELEASES_DIR=/releases
IGNITE_WORKING_DIR=/work
IGNITE_XML_CONFIG=/ignite-config.xml
IGNITE_USERS_LIBS=/libs
IGNITE_LOCAL_WORK_DIR=/home/user
IGNITE_JVM_OPTS="-server -Xms10g -Xmx10g -XX:+AlwaysPreTouch -XX:+UseG1GC
-XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC"


Please find our Ignite-config.xml below

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans";
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
           xmlns:util="http://www.springframework.org/schema/util";
           xsi:schemaLocation="
        http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
        http://www.springframework.org/schema/util
        http://www.springframework.org/schema/util/spring-util-2.0.xsd";>
<bean class="org.apache.ignite.configuration.IgniteConfiguration">
                <property name="clientMode" value="false"/>
                <property name="peerClassLoadingEnabled" value="true"/>
                <property name="networkTimeout" value="100000"/>
                <property name="networkSendRetryCount" value="5"/>
                <property name="discoverySpi">
            <bean
class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                <property name="ipFinder">
                    <bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                        <property name="addresses">
                          <list>
                                                               
<value>ipaddress:47500</value>
                            </list>
                        </property>
                    </bean>
                </property>
                                <property name="networkTimeout"
value="10000000"/>
                                <property name="joinTimeout"
value="10000000"/>
                                <property name="maxAckTimeout"
value="10000000"/>
                                <property name="reconnectCount" value="50"/>
                                <property name="socketTimeout"
value="10000000"/>
            </bean>
        </property>
        </bean>
</beans>

Cache Configuration:

var userGridConf: CacheConfiguration[Long, User] = new
CacheConfiguration[Long, User](userCacheName)
   userGridConf.setCacheMode(CacheMode.REPLICATED)
    userGridConf.setEagerTtl(false)
    userGridConf.setOnheapCacheEnabled(false)
    userGridConf.setBackups(0)
    userGridConf.setIndexedTypes(classOf[Long], classOf[User])
val cache = ignite.getOrCreateCache(userGridConf)
cache.size(CachePeekMode.OFFHEAP)

Could you please let me know what is wrong with the above configurations?

Thanks in Advance!!!




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/