Hi, We did a performance test for ActiveMQ and came across a problem. I hope someone help us understand what was wrong with my ActiveMQ. Let me explain it here.
<problem> Transaction time became long suddenly from 50 millsec to 60 sec in about 60 minutes after starting a performance test. ActiveMQ kept cpu and memory usage in a normal condition during the performance test. However, iowait rose 10% in 60 minutes. <environment> Our ActiveMQ version 5.14.3 We emulated 1,000 IoT clients and sent data to ActiveMQ at one minute interval. Payload size sent by IoT client was about 2 Kbytes. KahaDB size is enough big. (100GB) <KahaDB log> There was an error and out of memory in kahaDB log. -------------------------- 2017-06-16 04:39:19,615 [eckpoint Worker] WARN MessageDatabase - Failed to load next journal location: null 2017-06-16 04:39:19,615 [eckpoint Worker] WARN MessageDatabase - Failed to load next journal location: null 2017-06-16 04:45:08,527 [eckpoint Worker] WARN MessageDatabase - Failed to load next journal location: null 2017-06-16 04:45:08,527 [eckpoint Worker] WARN MessageDatabase - Failed to load next journal location: null 2017-06-16 04:57:16,863 [eckpoint Worker] WARN MessageDatabase - Failed to load next journal location: null 2017-06-16 04:57:16,863 [eckpoint Worker] WARN MessageDatabase - Failed to load next journal location: null 2017-06-16 05:10:08,083 [eckpoint Worker] WARN MessageDatabase - Failed to load next journal location: null 2017-06-16 05:10:08,083 [eckpoint Worker] WARN MessageDatabase - Failed to load next journal location: null 2017-06-16 05:22:49,960 [eckpoint Worker] WARN MessageDatabase - Failed to load next journal location: null 2017-06-16 05:22:49,960 [eckpoint Worker] WARN MessageDatabase - Failed to load next journal location: null 2017-06-16 08:19:45,059 [eckpoint Worker] ERROR MessageDatabase - Checkpoint failed java.lang.OutOfMemoryError: GC overhead limit exceeded 2017-06-16 08:19:45,059 [eckpoint Worker] ERROR MessageDatabase - Checkpoint failed java.lang.OutOfMemoryError: GC overhead limit exceeded -------------------------- Thanks. Hidekazu -- View this message in context: http://activemq.2283324.n4.nabble.com/AcativeMQ-becomes-slow-suddenly-in-60-minutes-after-starting-a-performance-test-tp4728103.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
