Hi All, We upgraded solr(cloud) from 6.6 version to 8.8.2.
Below are job details: 1. Indexing every 24 hour 2. There are two in place update runs 15 min apart each running every 30 minutes. In-place update details: updatechain: inplace-update-chain number of segment forced to 1 Index 1. Total documents 983768 occupying 1.87 GB of data Implementation Details: 1. Customized plugin code with block join implementation. Memory: 30 GB of Physical Memory 12 GB of Heap 16 core cpu Cache usage: Changed the cache for queryResultCache, documentCache from LRUcache to CaffeineCache , due to performance issue. GC Parameters: -XX:+AggressiveOpts -XX:+HeapDumpOnOutOfMemoryError -XX:+ParallelRefProcEnabled -XX:+PerfDisableSharedMem -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -XX:+UseGCLogFileRotation -XX:+UseLargePages -XX:-OmitStackTraceInFastThrow -XX:ConcGCThreads=8 -XX:G1HeapWastePercent=2 -XX:G1MaxNewSizePercent=75 -XX:G1MixedGCLiveThresholdPercent=90 -XX:G1OldCSetRegionThresholdPercent=15 -XX:G1ReservePercent=5 -XX:GCLogFileSize=20M -XX:HeapDumpPath=/data/java_69904(date).hprof -XX:InitiatingHeapOccupancyPercent=80 -XX:MaxGCPauseMillis=1000 -XX:MaxPermSize=250M -XX:NumberOfGCLogFiles=9 -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983 /var/log/DP/solr -XX:ParallelGCThreads=13 -Xloggc:/var/log/DP/solr/solr_gc.log -Xms12g -Xmx12g -Xss256k -Xss256k -verbose:gc It was observed after deploying the migrated code base during negligible traffic and in-place update in the span of 2 hours that complete 12 GB of heap usage resulted in dead nodes. Attached the GC snapshot where deep GC is observed. Please suggest how to avoid the high heap usage ? is there any specific aspects to handle the inplacee updates while solr upgrade ? Any pointers to resolve this issue is highly appreciated -- Regards, Sachin
