The command line that we use contains the workaround -XX:CompileCommand=exclude,com.github.benmanes.caffeine.cache.BoundedLocalCache::put
and the stack is different so I don’t think it’s related. Thanks anyway for trying to help. — Ing. Andrea Vettori Sistemi Informativi B2BIres s.r.l. > On 21 Oct 2023, at 12:13, Ishan Chattopadhyaya <ichattopadhy...@gmail.com> > wrote: > > Could it be related to > https://issues.apache.org/jira/plugins/servlet/mobile#issue/SOLR-16463 ? > > On Sat, 21 Oct, 2023, 2:03 pm Ing. Andrea Vettori, <a.vett...@b2bires.com> > wrote: > >> Hello, we’re using two SOLR servers (same hw, same version of solr and >> java, same solr config). The SOLR version is 9.3 and JVM is Adoptium JDK >> 17.0.8.1 on Linux. >> They both were running fine since a couple years (we upgraded from SOLR 8 >> to 9 with full reindexing some time ago). >> >> Yesterday one of the server died with JVM crash with the following reason >> (I have the full JVM trace if needed). >> Once restarted the server ran fine and received data updates every 15 >> minutes, and responded to queries during the day. >> Today the server died around the same time with the same JVM trace. >> >> The time it died two times is early in the morning when we upload a lot of >> data. Then during the day the updates are less heavy in terms of size. >> >> One strange thing is that only one of the server died, the other one is >> running fine and it’s receiving the same data. >> >> Another thing to note is that in solrconfig we still had the “old” caches >> of SOLR 8 configured. Two days ago we changed the configuration to use >> CaffeineCache on one of the four cores (the biggest one). Not sure if it’s >> related but the time is suspicious… but why would it crash only on one of >> the servers since they’re both identical in configuration, version and >> hardware? Anyway I replaced solrconfig with the old configuration to see >> what happens tomorrow. >> >> I searched for bug fixes in Solr 9.4 and Lucene 9.8 and did not find >> anything that seems related to this. >> Any hint ? >> >> Thanks >> >> Current thread (0x00007f3f28002d10): ConcurrentGCThread "G1 Conc#4" >> [stack: 0x00007f3e8df6e000,0x00007f3e8e06e000] [id=25851] >> >> Stack: [0x00007f3e8df6e000,0x00007f3e8e06e000], sp=0x00007f3e8e06cc20, >> free space=1019k >> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native >> code) >> V [libjvm.so+0x74efc8] void >> OopOopIterateDispatch<G1RebuildRemSetClosure>::Table::oop_oop_iterate<InstanceKlass, >> narrowOop>(G1RebuildRemSetClosure*, oopDesc*, Klass*)+0x98 >> V [libjvm.so+0x750c09] >> G1RebuildRemSetTask::G1RebuildRemSetHeapRegionClosure::do_heap_region(HeapRegion*)+0x579 >> V [libjvm.so+0x7dc2d6] >> HeapRegionManager::par_iterate(HeapRegionClosure*, HeapRegionClaimer*, >> unsigned int) const+0x96 >> V [libjvm.so+0x74d799] G1RebuildRemSetTask::work(unsigned int)+0x69 >> V [libjvm.so+0xf17f2f] GangWorker::loop()+0x5f >> V [libjvm.so+0xf17f8f] >> V [libjvm.so+0xe68d40] Thread::call_run()+0xc0 >> V [libjvm.so+0xc1e681] thread_native_entry(Thread*)+0xe1 >> >> >> siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: >> 0x0000000000000003 >> >> Register to memory mapping: >> >> RAX=0x00000006a0b79a98 is an oop: java.lang.String >> {0x00000006a0b79a98} - klass: 'java/lang/String' >> - string: "cmesmedia_44_2021_12" >> RBX=0x00000006e0382a18 is an oop: org.apache.lucene.document.StoredField >> {0x00000006e0382a18} - klass: 'org/apache/lucene/document/StoredField' >> - ---- fields (total size 4 words): >> - protected final 'type' 'Lorg/apache/lucene/index/IndexableFieldType;' >> @12 a 'org/apache/lucene/document/FieldType'{0x0000000691241d58} (d22483ab) >> - protected final 'name' 'Ljava/lang/String;' @16 >> "cmesmedia_44_2021_12"{0x00000006a0b79a98} (d416f353) >> - protected 'fieldsData' 'Ljava/lang/Object;' @20 a >> 'java/lang/Float'{0x00000006e0382a38} = 55.000000 (dc070547) >> - protected 'tokenStream' 'Lorg/apache/lucene/analysis/TokenStream;' @24 >> NULL (0) >> RCX=0x0000000000000003 is an unknown value >> RDX=0x00000000408fb0b0 is an unknown value >> RSP=0x00007f3e8e06cc20 points into unknown readable memory: >> 0x0000000000301c00 | 00 1c 30 00 00 00 00 00 >> RBP=0x00007f3e8e06cc60 points into unknown readable memory: >> 0x00007f3e8e06cd50 | 50 cd 06 8e 3e 7f 00 00 >> RSI=0x0000000000200000 is an unknown value >> RDI=0x00007f3d881d70d0 points into unknown readable memory: >> 0x00000000003803ff | ff 03 38 00 00 00 00 00 >> R8 =0x000000000000006d is an unknown value >> R9 =0x0000000000000044 is an unknown value >> R10=0x0000000000301c00 is an unknown value >> R11=0x0000000000000180 is an unknown value >> R12=0x00000006e0382a28 is pointing into object: >> org.apache.lucene.document.StoredField >> {0x00000006e0382a18} - klass: 'org/apache/lucene/document/StoredField' >> - ---- fields (total size 4 words): >> - protected final 'type' 'Lorg/apache/lucene/index/IndexableFieldType;' >> @12 a 'org/apache/lucene/document/FieldType'{0x0000000691241d58} (d22483ab) >> - protected final 'name' 'Ljava/lang/String;' @16 >> "cmesmedia_44_2021_12"{0x00000006a0b79a98} (d416f353) >> - protected 'fieldsData' 'Ljava/lang/Object;' @20 a >> 'java/lang/Float'{0x00000006e0382a38} = 55.000000 (dc070547) >> - protected 'tokenStream' 'Lorg/apache/lucene/analysis/TokenStream;' @24 >> NULL (0) >> R13=0x00000006e0382a34 is pointing into object: >> org.apache.lucene.document.StoredField >> {0x00000006e0382a18} - klass: 'org/apache/lucene/document/StoredField' >> - ---- fields (total size 4 words): >> - protected final 'type' 'Lorg/apache/lucene/index/IndexableFieldType;' >> @12 a 'org/apache/lucene/document/FieldType'{0x0000000691241d58} (d22483ab) >> - protected final 'name' 'Ljava/lang/String;' @16 >> "cmesmedia_44_2021_12"{0x00000006a0b79a98} (d416f353) >> - protected 'fieldsData' 'Ljava/lang/Object;' @20 a >> 'java/lang/Float'{0x00000006e0382a38} = 55.000000 (dc070547) >> - protected 'tokenStream' 'Lorg/apache/lucene/analysis/TokenStream;' @24 >> NULL (0) >> R14=0x00007f3e30786358 is pointing into metadata >> R15=0x00007f3e8e06cdc8 points into unknown readable memory: >> 0x00007f3f64941f68 | 68 1f 94 64 3f 7f 00 00 >> >> >> — >> Ing. Andrea Vettori >> Sistemi Informativi >> B2BIres s.r.l. >> >>
