2020-04-13 13:28:52 UTC - Alex Yaroslavsky: @Alex Yaroslavsky has joined the
channel
----
2020-04-13 22:31:52 UTC - Addison Higham: :thinking_face: have an issue with
pulsar functions worker JVM not exiting on an exception, relevant logs:
```22:20:10.304 [main] INFO
org.apache.pulsar.functions.runtime.JavaInstanceStarter - RuntimeSpawner quit,
shutting down JavaInstance
22:20:10.306 [main] INFO org.apache.pulsar.client.impl.PulsarClientImpl -
Client closing. URL: <pulsar+ssl://pulsar-prod-broker.pulsar:6651/>
22:20:10.306 [pulsar-client-io-1-1] INFO
org.apache.pulsar.client.impl.ProducerImpl -
[<persistent://canvas-cdc/prod-iad/cluster-11>] [pulsar-prod-iad-321-272]
Closed Producer
22:20:10.311 [pulsar-client-io-1-1] INFO
org.apache.pulsar.client.impl.ClientCnx - [id: 0xeee86625, L:/10.9.41.128:39674
! R:/10.9.62.180:6651] Disconnected
22:20:10.316 [pulsar-client-io-1-1] INFO
org.apache.pulsar.client.impl.ClientCnx - [id: 0xebe3b52e, L:/10.9.41.128:37308
! R:pulsar-prod-broker.pulsar.svc.cluster.local/10.9.52.60:6651] Disconnected
22:20:59.210 [aws-java-sdk-NettyEventLoop-1-2] WARN
io.netty.channel.AbstractChannelHandlerContext - An exception
'java.lang.NoClassDefFoundError: io/netty/buffer/ByteBufUtil$HexUtil' [enable
DEBUG level for full stacktrace] was thrown by a user handler's
exceptionCaught() method while handling the following exception:
java.lang.NoClassDefFoundError:
io/netty/channel/AbstractChannel$AbstractUnsafe$8
at
io.netty.channel.AbstractChannel$AbstractUnsafe.deregister(AbstractChannel.java:810)
~[impulse-uberjar.jar:0.2.3]
at
io.netty.channel.AbstractChannel$AbstractUnsafe.fireChannelInactiveAndDeregister(AbstractChannel.java:770)
~[impulse-uberjar.jar:0.2.3]
at
io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:753)
~[impulse-uberjar.jar:0.2.3]
at
io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:608)
~[impulse-uberjar.jar:0.2.3]
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:105)
[impulse-uberjar.jar:0.2.3]
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:171)
[impulse-uberjar.jar:0.2.3]
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
[impulse-uberjar.jar:0.2.3]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
[impulse-uberjar.jar:0.2.3]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
[impulse-uberjar.jar:0.2.3]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
[impulse-uberjar.jar:0.2.3]
at
io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
[impulse-uberjar.jar:0.2.3]
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[impulse-uberjar.jar:0.2.3]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
Caused by: java.lang.ClassNotFoundException:
io.netty.channel.AbstractChannel$AbstractUnsafe$8
at
<http://java.net|java.net>.URLClassLoader.findClass(URLClassLoader.java:382)
~[?:1.8.0_232]
at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_232]
at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_232]
... 13 more```
----
2020-04-13 22:32:21 UTC - Addison Higham: oh, hrm, maybe this is my issue, as
it is aws-sdk netty not closing
----
2020-04-13 22:32:37 UTC - Matteo Merli: It seems a netty version mismatch issue
----
2020-04-13 22:33:28 UTC - Matteo Merli: or, even more likely, 2 versions of
netty in classpath at same time
----
2020-04-13 22:33:59 UTC - Addison Higham: yeah, hrm, only surfaces in exception
handling, but I guess the biggest question: should the function runtime just
forcefully close at a certain point?
----
2020-04-13 22:35:45 UTC - Matteo Merli: you mean, having a close() hook?
----
2020-04-13 22:40:25 UTC - Addison Higham: could be implemented that way, just
specifically thinking of when running functions via k8s workers. Each function
has its own JVM, `JavaInstanceMain` kicks off `JavaInstanceStarter` and the
starter closes everything down and the run loop for the function stops. It
seems like `JavaInstanceStarter` (or perhaps `JavaInstanceMain`) at least in
the case of k8s workers could have a timeout and then `System.exit()` or
something?
----
2020-04-13 22:50:51 UTC - Matteo Merli: I see, so you mean that with that
exception the JVM is not exiting since the background threads are not stopped?
In K8S that should be taken care by K8S itself, there's are shutdown timeout
after which the process it's SIGKILLed, though in any case we could improve that
----
2020-04-14 07:38:00 UTC - Anshul Dhyani: @Anshul Dhyani has joined the channel
----
