Glad to hear it What was the problem? On Mon, 9 Dec 2019 at 16:44, Coscend@OM <om.insig...@coscend.com> wrote:
> Dear Maxim, > > > > Thank you for the pointers. These hints directed us to a resolution. > > > > Sincerely, > > > > Hemant K. Sabat > > www.Coscend.com <http://www.coscend.com/> > > ------------------------------------------------------------------ > > *Real-time, Interactive Video Collaboration, Tele-healthcare, > Tele-education, Telepresence Services, on the fly…* > > ------------------------------------------------------------------ > > CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail > Messages from Coscend Communications Solutions' posted at: > http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html > <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html> > > > > > > > > > > > > *From:* Maxim Solodovnik [mailto:solomax...@gmail.com] > *Sent:* Monday, December 9, 2019 10:28 AM > *To:* Openmeetings user-list <user@openmeetings.apache.org>; > om.insig...@coscend.com > *Subject:* Re: OM5: Tomcat not shutting down due to OM5 method > > > > conf/server.xml should have this line: > https://github.com/apache/openmeetings/blob/master/openmeetings-server/src/main/assembly/conf/server.xml#L22 > > to be able to perform shutdown > > You can check Tomcat documentation and contact users@tomcat.a.o mailing > list :) > > > > On Sun, 8 Dec 2019 at 06:16, Coscend@OM <om.insig...@coscend.com> wrote: > > Dear Maxim, > > > > è The error "Could not contact [localhost:8005] " most probably means: > you have disabled AJP. This is why shutdown.sh can do nothing > > > > Our server.xml setting: > > <Connector port="5443" > protocol="org.apache.coyote.http11.Http11AprProtocol" > > maxThreads="150" SSLEnabled="true" > > > <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" > /> > > <SSLHostConfig> > > <Certificate certificateKeyFile="/etc/….key" > > certificateFile="/etc/...cer" > > type="EC" /> > > </SSLHostConfig> > > </Connector> > > … > > <Connector port="8009" protocol="AJP/1.3" redirectPort="<5443>" /> > > > > Question: There is a line in the log. After this line, there are several > errors. > > > > 07-Dec-2019 16:55:45.082 WARNING [main] > org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The > web application [Openmeetings] registered the JDBC driver > [com.mysql.cj.jdbc.Driver] but failed to unregister it when the web > application was stopped. To prevent a memory leak, the JDBC Driver has been > forcibly unregistered. > > > > > > ---------- > > Here is the OM log: > > > > 07-Dec-2019 16:55:44.938 INFO [main] > org.apache.catalina.core.StandardServer.await A valid shutdown command was > received via the shutdown port. Stopping the Server instance. > > 07-Dec-2019 16:55:44.938 INFO [main] > org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler > ["https-openssl-apr-5443"] > > 07-Dec-2019 16:55:44.945 INFO [main] > org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler > ["ajp-nio-8009"] > > 07-Dec-2019 16:55:44.950 INFO [main] > org.apache.catalina.core.StandardService.stopInternal Stopping service > [Catalina] > > [34mINFO [0;39m 12-07 16:55:44.983 [36mo.a.w.Application:539 [main][0;39m > - [OpenmeetingsApplication] destroy: Wicket Kendo UI initializer > > [34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m > - [OpenmeetingsApplication] destroy: > org.wicketstuff.dashboard.DashboardContextInitializer@10c541dc > > [34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m > - [OpenmeetingsApplication] destroy: Wicket jQuery UI initializer (plugins) > > [34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m > - [OpenmeetingsApplication] destroy: Wicket jQuery UI initializer > > [34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m > - [OpenmeetingsApplication] destroy: Wicket Kendo UI initializer > (theme-default) > > [34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m > - [OpenmeetingsApplication] destroy: Wicket core library initializer > > [34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m > - [OpenmeetingsApplication] destroy: DevUtils DebugBar Initializer > > [34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m > - [OpenmeetingsApplication] destroy: Wicket extensions initializer > > 07-Dec-2019 16:55:44.984 INFO [main] > com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12.2] > [172.17.0.1]:5701 is SHUTTING_DOWN > > 07-Dec-2019 16:55:44.990 INFO [hz.server-1.cached.thread-4] > com.hazelcast.internal.partition.impl.MigrationManager.null > [172.17.0.1]:5701 [dev] [3.12.2] Shutdown request of Member > [172.17.0.1]:5701 - 7d3e2187-936e-49b1-a0b7-cdb0f2a604d6 this is handled > > 07-Dec-2019 16:55:44.996 INFO [main] com.hazelcast.instance.Node.null > [172.17.0.1]:5701 [dev] [3.12.2] Shutting down connection manager... > > 07-Dec-2019 16:55:44.997 INFO [main] com.hazelcast.instance.Node.null > [172.17.0.1]:5701 [dev] [3.12.2] Shutting down node engine... > > 07-Dec-2019 16:55:45.011 INFO [main] > com.hazelcast.instance.NodeExtension.null [172.17.0.1]:5701 [dev] [3.12.2] > Destroying node NodeExtension. > > 07-Dec-2019 16:55:45.011 INFO [main] com.hazelcast.instance.Node.null > [172.17.0.1]:5701 [dev] [3.12.2] Hazelcast Shutdown is completed in 22 ms. > > 07-Dec-2019 16:55:45.011 INFO [main] > com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12.2] > [172.17.0.1]:5701 is SHUTDOWN > > 07-Dec-2019 16:55:45.082 WARNING [main] > org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The > web application [Openmeetings] registered the JDBC driver > [com.mysql.cj.jdbc.Driver] but failed to unregister it when the web > application was stopped. To prevent a memory leak, the JDBC Driver has been > forcibly unregistered. > > 07-Dec-2019 16:55:45.083 WARNING [main] > org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The > web application [Openmeetings] appears to have started a thread named > [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to > create a memory leak. Stack trace of thread: > > java.base@11/java.lang.Thread.sleep(Native Method) > > > io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:720) > > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:510) > > > io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > > java.base@11/java.lang.Thread.run(Thread.java:834) > > 07-Dec-2019 16:55:45.083 WARNING [main] > org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The > web application [Openmeetings] appears to have started a thread named > [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but > has failed to stop it. This is very likely to create a memory leak. Stack > trace of thread: > > java.base@11/java.lang.Object.wait(Native Method) > > > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555) > > > > Thank you. > > > > Sincerely, > > > > Hemant K. Sabat > > www.Coscend.com <http://www.coscend.com/> > > ------------------------------------------------------------------ > > *Real-time, Interactive Video Collaboration, Tele-healthcare, > Tele-education, Telepresence Services, on the fly…* > > ------------------------------------------------------------------ > > CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail > Messages from Coscend Communications Solutions' posted at: > http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html > <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html> > > > > > > > > *From:* Maxim Solodovnik [mailto:solomax...@gmail.com] > *Sent:* Tuesday, November 26, 2019 9:00 PM > *To:* Openmeetings user-list <user@openmeetings.apache.org>; > om.insig...@coscend.com > *Subject:* Re: OM5: Tomcat not shutting down due to OM5 method > > > > Hello Hemant, > > > > Vanilla OM5 doen't have MySQL driver > > It also not uses port 443 > > > > I would suggest to test OM with no modifications (before ANY > customizations) > > > > The error "Could not contact [localhost:8005] " most probably means: you > have disabled AJP > > This is why shutdown.sh can do nothing > > you can still use `kill -9` in this case > > > > On Tue, 26 Nov 2019 at 21:59, Coscend@OM <om.insig...@coscend.com> wrote: > > Dear Maxim, > > > > Vanilla Tomcat9: startup and shutdown fine. > > > > Vanilla OM5 > > User: when the binary is unarchived, it has a local usergroup as the > default owner. Is this correct? > > > > (binary from OpenMeetings Web site): no configuration changes except SSL > certificate input. > > > > > > Startup fine > > Shutdown: > > netstat gives no port binding on 443 and no pid > > echo $CATALINA_PID gives blank > > BUT > > ps aux gives tomcat pid running > > > > Below are CLI log and OM5 / Catalina log. > > > > > > If we run shutdown command again, CLI gives “Tomcat may not be running”: > > > > ------------------------------------------------------- > > CLI log > > > > org.apache.catalina.startup.Catalina stopServer > > SEVERE: Could not contact [localhost:8005] (base port [8005] and offset > [0]). Tomcat may not be running. > > Nov 26, 2019 8:39:11 AM org.apache.catalina.startup.Catalina stopServer > > SEVERE: Error stopping Catalina > > java.net.ConnectException: Connection refused (Connection refused) > > at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) > > at > java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) > > at > java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) > > at > java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) > > at > java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) > > at java.base/java.net.Socket.connect(Socket.java:591) > > at java.base/java.net.Socket.connect(Socket.java:540) > > at java.base/java.net.Socket.<init>(Socket.java:436) > > at java.base/java.net.Socket.<init>(Socket.java:213) > > at > org.apache.catalina.startup.Catalina.stopServer(Catalina.java:513) > > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > > at > org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:390) > > at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:480) > > > > > > OM5 log > > > > 26-Nov-2019 08:25:49.733 INFO [main] > com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12] > [172.17.0.1]:5701 is SHUTDOWN > > 26-Nov-2019 08:25:49.790 WARNING [main] > org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The > web application [Openmeetings] registered the JDBC driver > [com.mysql.cj.jdbc.Driver] but failed to unregister it when the web > application was stopped. To prevent a memory leak, the JDBC Driver has been > forcibly unregistered. > > 26-Nov-2019 08:25:49.791 WARNING [main] > org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The > web application [Openmeetings] appears to have started a thread named > [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-6] but > has failed to stop it. This is very likely to create a memory leak. Stack > trace of thread: > > java.base@11/java.lang.Object.wait(Native Method) > > > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555) > > 26-Nov-2019 08:25:49.792 WARNING [main] > org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The > web application [Openmeetings] appears to have started a thread named > [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-9] but > has failed to stop it. This is very likely to create a memory leak. Stack > trace of thread: > > java.base@11/java.lang.Object.wait(Native Method) > > > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555) > > 26-Nov-2019 08:25:49.793 WARNING [main] > org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The > web application [Openmeetings] appears to have started a thread named > [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-10] > but has failed to stop it. This is very likely to create a memory leak. > Stack trace of thread: > > java.base@11/java.lang.Object.wait(Native Method) > > > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555) > > 26-Nov-2019 08:25:49.793 WARNING [main] > org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The > web application [Openmeetings] appears to have started a thread named > [mysql-cj-abandoned-connection-cleanup] but has failed to stop it. This is > very likely to create a memory leak. Stack trace of thread: > > java.base@11/java.lang.Object.wait(Native Method) > > java.base@11/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155) > > > com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:85) > > > java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > > java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > java.base@11/java.lang.Thread.run(Thread.java:834) > > 26-Nov-2019 08:25:49.794 WARNING [main] > org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The > web application [Openmeetings] appears to have started a thread named > [pool-5-thread-1] but has failed to stop it. This is very likely to create > a memory leak. Stack trace of thread: > > java.base@11/jdk.internal.misc.Unsafe.park(Native Method) > > > java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) > > > java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) > > > java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170) > > > java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) > > > java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) > > > java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) > > > java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > java.base@11/java.lang.Thread.run(Thread.java:834) > > 26-Nov-2019 08:25:49.795 WARNING [main] > org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The > web application [Openmeetings] appears to have started a thread named > [pool-1-thread-1] but has failed to stop it. This is very likely to create > a memory leak. Stack trace of thread: > > java.base@11/jdk.internal.misc.Unsafe.park(Native Method) > > > java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) > > > java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) > > > java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170) > > > java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) > > > java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) > > > java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) > > > java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > java.base@11/java.lang.Thread.run(Thread.java:834) > > 26-Nov-2019 08:25:49.795 WARNING [main] > org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The > web application [Openmeetings] appears to have started a thread named > [nioEventLoopGroup-6-1] but has failed to stop it. This is very likely to > create a memory leak. Stack trace of thread: > > java.base@11/java.lang.Thread.sleep(Native Method) > > > io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719) > > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505) > > > io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) > > > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > > java.base@11/java.lang.Thread.run(Thread.java:834) > > 26-Nov-2019 08:25:49.796 WARNING [main] > org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The > web application [Openmeetings] appears to have started a thread named > [pool-21-thread-1] but has failed to stop it. This is very likely to create > a memory leak. Stack trace of thread: > > java.base@11/jdk.internal.misc.Unsafe.park(Native Method) > > > java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) > > > java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) > > > java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170) > > > java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) > > > java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) > > > java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) > > > java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > java.base@11/java.lang.Thread.run(Thread.java:834) > > 26-Nov-2019 08:25:49.796 WARNING [main] > org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The > web application [Openmeetings] appears to have started a thread named > [nioEventLoopGroup-6-2] but has failed to stop it. This is very likely to > create a memory leak. Stack trace of thread: > > java.base@11/java.lang.Thread.sleep(Native Method) > > > io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719) > > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505) > > > io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) > > > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > > java.base@11/java.lang.Thread.run(Thread.java:834) > > 26-Nov-2019 08:25:49.797 WARNING [main] > org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The > web application [Openmeetings] appears to have started a thread named > [nioEventLoopGroup-6-3] but has failed to stop it. This is very likely to > create a memory leak. Stack trace of thread: > > java.base@11/java.lang.Thread.sleep(Native Method) > > > io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719) > > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505) > > > io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) > > > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > > java.base@11/java.lang.Thread.run(Thread.java:834) > > > > …..REPEATS SEVERAL TIMES > > > > 26-Nov-2019 08:25:49.814 WARNING [main] > org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The > web application [Openmeetings] appears to have started a thread named > [nioEventLoopGroup-6-30] but has failed to stop it. This is very likely to > create a memory leak. Stack trace of thread: > > java.base@11/java.lang.Thread.sleep(Native Method) > > > io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719) > > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505) > > > io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) > > > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > > java.base@11/java.lang.Thread.run(Thread.java:834) > > 26-Nov-2019 08:25:49.815 SEVERE [main] > org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks > The web application [Openmeetings] created a ThreadLocal with key of type > [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a > value of type [io.netty.util.internal.InternalThreadLocalMap] (value > [io.netty.util.internal.InternalThreadLocalMap@560478b8]) but failed to > remove it when the web application was stopped. Threads are going to be > renewed over time to try and avoid a probable memory leak. > > 26-Nov-2019 08:25:49.816 SEVERE [main] > org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks > The web application [Openmeetings] created a ThreadLocal with key of type > [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@71da8156]) and a > value of type [org.apache.wicket.ThreadContext] (value > [org.apache.wicket.ThreadContext@12b21dea]) but failed to remove it when > the web application was stopped. Threads are going to be renewed over time > to try and avoid a probable memory leak. > > 26-Nov-2019 08:25:49.816 SEVERE [main] > org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks > The web application [Openmeetings] created a ThreadLocal with key of type > [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@71da8156]) and a > value of type [org.apache.wicket.ThreadContext] (value > [org.apache.wicket.ThreadContext@e41cc5d]) but failed to remove it when > the web application was stopped. Threads are going to be renewed over time > to try and avoid a probable memory leak… > > > > …REPEATS SEVERAL TIMES > > > > 26-Nov-2019 08:25:49.818 SEVERE [main] > org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks > The web application [Openmeetings] created a ThreadLocal with key of type > [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@71da8156]) and a > value of type [org.apache.wicket.ThreadContext] (value > [org.apache.wicket.ThreadContext@4e2fb8d6]) but failed to remove it when > the web application was stopped. Threads are going to be renewed over time > to try and avoid a probable memory leak. > > 26-Nov-2019 08:25:49.818 SEVERE [main] > org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks > The web application [Openmeetings] created a ThreadLocal with key of type > [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a > value of type [io.netty.util.internal.InternalThreadLocalMap] (value > [io.netty.util.internal.InternalThreadLocalMap@1c5e617f]) but failed to > remove it when the web application was stopped. Threads are going to be > renewed over time to try and avoid a probable memory leak. > > 26-Nov-2019 08:25:49.818 SEVERE [main] > org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks > The web application [Openmeetings] created a ThreadLocal with key of type > [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a > value of type [io.netty.util.internal.InternalThreadLocalMap] (value > [io.netty.util.internal.InternalThreadLocalMap@5147f2c6]) but failed to > remove it when the web application was stopped. Threads are going to be > renewed over time to try and avoid a probable memory leak. > > 26-Nov-2019 08:25:49.818 SEVERE [main] > org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks > The web application [Openmeetings] created a ThreadLocal with key of type > [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a > value of type [io.netty.util.internal.InternalThreadLocalMap] (value > [io.netty.util.internal.InternalThreadLocalMap@ca5a0d3]) but failed to > remove it when the web application was stopped. Threads are going to be > renewed over time to try and avoid a probable memory leak. > > 26-Nov-2019 08:25:49.823 INFO [main] > org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler > ["https-openssl-apr-443"] > > 26-Nov-2019 08:25:49.874 INFO [main] > org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler > ["ajp-nio-8009"] > > 26-Nov-2019 08:25:49.879 INFO [main] > org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler > ["https-openssl-apr-443"] > > 26-Nov-2019 08:25:49.881 INFO [main] > org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler > ["ajp-nio-8009"] > > 26-Nov-2019 08:25:49.913 INFO [mysql-cj-abandoned-connection-cleanup] > org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading > Illegal access: this web application instance has been stopped already. > Could not load []. The following stack trace is thrown for debugging > purposes as well as to attempt to terminate the thread which caused the > illegal access. > > java.lang.IllegalStateException: Illegal access: this > web application instance has been stopped already. Could not load []. The > following stack trace is thrown for debugging purposes as well as to > attempt to terminate the thread which caused the illegal access. > > at > org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1385) > > at > org.apache.catalina.loader.WebappClassLoaderBase.getResource(WebappClassLoaderBase.java:1038) > > at > com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.checkThreadContextClassLoader(AbandonedConnectionCleanupThread.java:117) > > at > com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:84) > > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > at > java.base/java.lang.Thread.run(Thread.java:834) > > > > > > > > Thank you. > > > > Sincerely, > > > > Hemant K. Sabat > > www.Coscend.com <http://www.coscend.com/> > > ------------------------------------------------------------------ > > *Real-time, Interactive Video Collaboration, Tele-healthcare, > Tele-education, Telepresence Services, on the fly…* > > ------------------------------------------------------------------ > > CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail > Messages from Coscend Communications Solutions' posted at: > http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html > <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html> > > > > > > > > > > *From:* Maxim Solodovnik [mailto:solomax...@gmail.com] > *Sent:* Monday, November 25, 2019 10:21 AM > *To:* Openmeetings user-list <user@openmeetings.apache.org>; > om.insig...@coscend.com > *Subject:* Re: OM5: Tomcat not shutting down due to OM5 method > > > > Hello Hemant, > > > > dev@ mailing list is for discussing OM/Om community development (i.e. > discussing patch, the way to solve something, etc.) > > NOT for contacting OM devs to report an issue > > > > I don't have CentOS8 so I need detailed description of > > what do you mean by "Environment is CentOS 8 (using docker) and MariaDB > 10.3.x"? (OM is java application, it doesn't require docker to be able to > run) > > > > So could you run "vanilla" OM5 (without any other services, proxies, 3rd > parties etc.) and check it it works for you as expected? i.e. both > startup.sh and shutdown.sh run as correct user do it's job? > > > > If yes - add changes one by one and continue checking > > If not - compare with "vanilla" Tomcat9 - what is different :) > > baby steps :) > > > > On Fri, 22 Nov 2019 at 13:49, Coscend@OM <om.insig...@coscend.com> wrote: > > Further, if we do not run OM5, Tomcat shuts down properly. Environment is > CentOS 8 (using docker) and MariaDB 10.3.x > > > > *From:* Coscend@OM [mailto:om.insig...@coscend.com] > *Sent:* Friday, November 22, 2019 12:12 PM > *To:* user@openmeetings.apache.org > *Subject:* RE: OM5: Tomcat not shutting down due to OM5 method > > > > Dear Maxim, > > > > I am sending this to users, but we think one possibility (not > confirmation) is this error is related to OM5 code [1]. There is a method > in OM5 that is triggering memory leak while shutting down Tomcat. The most > common reason is something hanging or delaying in Web application > initializations [2]. This prevents Tomcat to properly shutdown [1, 2]. > > > > Or, is due to something else? (We are using OM5 binary from OM Web site. > There are no customization or configuration changes.) > > > > Here is the relevant log. > > > > The key line in the log is: “The web application [Openmeetings] appears > to have started a thread named [nioEventLoopGroup-2-1] but has failed to > stop it. This is very likely to create a memory leak.” > > ------------------------ > > 21-Nov-2019 23:24:30.024 WARNING [main] > > org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads > The web application [Openmeetings] appears to have started a thread named > [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to > create a memory leak. Stack trace of thread: > java.base@11/java.lang.Thread.sleep(Native Method) > > > > > io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719) > > > > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505) > > > > > io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) > > > > > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > > java.base@11/java.lang.Thread.run(Thread.java:834) > > > > 21-Nov-2019 23:24:30.025 WARNING [main] > > org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads > The web application [Openmeetings] appears to have started a thread named > [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but > has failed to stop it. This is very likely to create a memory leak. Stack > trace of thread:java.base@11/java.lang.Object.wait(Native Method) > > > > > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555) > > > > > > [1] > https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shutdown-port-8005 > > > > [2] > https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listening?noredirect=1 > > > > > > Thank you. > > > > Sincerely, > > > > Hemant K. Sabat > > www.Coscend.com <http://www.coscend.com/> > > ------------------------------------------------------------------ > > *Real-time, Interactive Video Collaboration, Tele-healthcare, > Tele-education, Telepresence Services, on the fly…* > > ------------------------------------------------------------------ > > CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail > Messages from Coscend Communications Solutions' posted at: > http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html > <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html> > > > > > > > > *From:* Maxim Solodovnik [mailto:solomax...@gmail.com > <solomax...@gmail.com>] > *Sent:* Friday, November 22, 2019 11:36 AM > *To:* dev <d...@openmeetings.apache.org>; om.insig...@coscend.com > *Subject:* Re: OM5: Tomcat not shutting down due to OM5 method > > > > Please send these messages to users@ > > It is not about OM/OM_community development > > > > shutdown works for me as expected > > please provide additional details (if any) > > > > On Fri, 22 Nov 2019 at 13:04, Coscend@OM <om.insig...@coscend.com> wrote: > > Dear OM Community, > > > > While shutting down OM5-Tomcat via shutdown.sh, Tomcat is not shutting > down. > These links state some method in Webapp could be preventing the shutdown > process of Tomcat: > > > https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shu > tdown-port-8005 > > > https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listen > ing?noredirect=1 > < > https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-liste > ning?noredirect=1&lq=1> &lq=1 > > > > Could anyone provide any guidance what we need to modify in OM5 code or any > other place to properly shutdown Tomcat? > > > > Here is the log snippet. There are several more such errors while shutting > down. > > > > ------------------------------ > > > > 21-Nov-2019 23:24:29.843 INFO [main] > org.apache.catalina.core.StandardServer.await A valid shutdown command was > received via the shutdown port. Stopping the Server instance. > > 21-Nov-2019 23:24:29.844 INFO [main] > org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler > ["http-nio-5080"] > > 21-Nov-2019 23:24:29.850 INFO [main] > org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler > ["ajp-nio-8009"] > > 21-Nov-2019 23:24:29.856 INFO [main] > org.apache.catalina.core.StandardService.stopInternal Stopping service > [Catalina] > > [34mINFO [0;39m 11-21 23:24:29.884 [36mo.a.w.Application:539 [main][0;39m - > [OpenmeetingsApplication] destroy: Wicket jQuery UI initializer > > [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m - > [OpenmeetingsApplication] destroy: Wicket extensions initializer > > [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m - > [OpenmeetingsApplication] destroy: Wicket Kendo UI initializer > > [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m - > [OpenmeetingsApplication] destroy: Wicket core library initializer > > [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m - > [OpenmeetingsApplication] destroy: DevUtils DebugBar Initializer > > [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m - > [OpenmeetingsApplication] destroy: > org.wicketstuff.dashboard.DashboardContextInitializer@72690a1 > > [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m - > [OpenmeetingsApplication] destroy: Wicket jQuery UI initializer (plugins) > > [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m - > [OpenmeetingsApplication] destroy: Wicket Kendo UI initializer > (theme-default) > > 21-Nov-2019 23:24:29.886 INFO [main] > com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12] > [172.17.0.1]:5701 is SHUTTING_DOWN > > 21-Nov-2019 23:24:29.891 INFO [hz.server-1.cached.thread-2] > com.hazelcast.internal.partition.impl.MigrationManager.null > [172.17.0.1]:5701 [dev] [3.12] Shutdown request of Member [172.17.0.1]:5701 > - 799babea-5e78-4cf9-a099-f4b1fb09b9c9 this is handled > > 21-Nov-2019 23:24:29.965 INFO [main] com.hazelcast.instance.Node.null > [172.17.0.1]:5701 [dev] [3.12] Shutting down connection manager... > > 21-Nov-2019 23:24:29.965 INFO [main] com.hazelcast.instance.Node.null > [172.17.0.1]:5701 [dev] [3.12] Shutting down node engine... > > 21-Nov-2019 23:24:29.979 INFO [main] > com.hazelcast.instance.NodeExtension.null [172.17.0.1]:5701 [dev] [3.12] > Destroying node NodeExtension. > > 21-Nov-2019 23:24:29.980 INFO [main] com.hazelcast.instance.Node.null > [172.17.0.1]:5701 [dev] [3.12] Hazelcast Shutdown is completed in 90 ms. > > 21-Nov-2019 23:24:29.980 INFO [main] > com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12] > [172.17.0.1]:5701 is SHUTDOWN > > 21-Nov-2019 23:24:30.023 WARNING [main] > org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The > web > application [Openmeetings] registered the JDBC driver > [com.mysql.cj.jdbc.Driver] but failed to unregister it when the web > application was stopped. To prevent a memory leak, the JDBC Driver has been > forcibly unregistered. > > 21-Nov-2019 23:24:30.024 WARNING [main] > org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The > web application [Openmeetings] appears to have started a thread named > [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to > create a memory leak. Stack trace of thread: > > java.base@11/java.lang.Thread.sleep(Native Method) > > > io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThr > eadEventExecutor.java:719) > > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505) > > > io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventEx > ecutor.java:905) > > > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable > .java:30) > > java.base@11/java.lang.Thread.run(Thread.java:834) > > 21-Nov-2019 23:24:30.025 WARNING [main] > org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The > web application [Openmeetings] appears to have started a thread named > [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but > has failed to stop it. This is very likely to create a memory leak. Stack > trace of thread: > > java.base@11/java.lang.Object.wait(Native Method) > > > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555 > ) > > > > > > Thank you. > > > > Sincerely, > > > > Hemant K. Sabat > > <http://www.coscend.com/> www.Coscend.com > > ------------------------------------------------------------------ > > Real-time, Interactive Video Collaboration, Tele-healthcare, > Tele-education, > Telepresence Services, on the fly. > > ------------------------------------------------------------------ > > CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail > Messages from Coscend Communications Solutions' posted at: > <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html> > http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html > > > > > -- > > WBR > Maxim aka solomax > > > > > -- > > WBR > Maxim aka solomax > > > > > -- > > WBR > Maxim aka solomax > > > > > -- > > WBR > Maxim aka solomax > -- WBR Maxim aka solomax
