Hi Justin Thanks for responding. We are using Artemis (version 2.13.0) to send "notifications" to interested consumers when data is updated/added in our databases. The use case is for a third party to get these "notifications" but as they are external to our cloud deployment they need to consume via a public interface. We are evaluating Artemis REST interface and JMS via http tunnel. I don;t have a stack trace but tried modifying code to add some debug statements (see log messages marked as SKV) . I have attached snippets from two logs; one for local server that works while the other is from the cloud server. I observed a noticeable delay from when it gets the first request before sending a "bogusresponse" debug statement. I had tried with HTTP with TLS ("sslEnabled=true") as well and it fails in the same manner. To make things simpler, I used plain HTTP transport.
Thanks Sunil On Thu, 26 May 2022 at 22:54, Justin Bertram <jbert...@apache.org> wrote: > > I tried running a sample JMS program using Netty Http tunnelling to > connect to the Artemis service via the public interface but the connection > fails... > > Can you provide the source code and configuration for this sample JMS > program? > > > Server side I am seeing some error after some time (not immediately): > AMQ212037: Connection failure to /10.5.170.231:48460 has been detected: > io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpRequest > cannot be cast to io.netty.buffer.ByteBuf [code=GENERIC_EXCEPTION]... > > Is there a stack-trace associated with this? If so, could you provide it? > > > Are there any issues using Netty http tunnelling via such proxies? > > As far as I can tell this is not a normal use-case. In fact, you're the > first person I've heard of trying something like this. Therefore, it's hard > to say if there are "any issues." Given your experience I would say that > there are. > > Aside from the specific problems you're facing, can you elaborate on why > you're attempting to connect your JMS application(s) via HTTP through an > istio proxy? Messaging connections are stateful unlike HTTP connections > (which are stateless). It's technically possible to use HTTP, but it's > certainly not optimal. > > > Justin > > On Wed, May 25, 2022 at 2:37 AM Sunil Varma <sunil.k.va...@gmail.com> > wrote: > > > Hello all! > > I am trying to connect to our Artemis service running in the cloud as a > > kubernetes service. > > We have istio ingress to expose HTTP and HTTPS routes from outside the > > cluster to services within the cluster and some istio routing rules to > > forward traffic to the appropriate services. > > > > I tried running a sample JMS program using Netty Http tunnelling to > connect > > to the Artemis service via the public interface but the connection fails > > (ActiveMQConnectionFactory.createConnection) . I have verified the Http > > POST request reaches Artemis and it does send responses (200) but the > > connection always fails. > > Server side I am seeing some error after some time (not immediately): > > AMQ212037: Connection failure to /10.5.170.231:48460 has been detected: > > > io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpRequest > > cannot be cast to io.netty.buffer.ByteBuf [code=GENERIC_EXCEPTION] > > > > Using kubernetes port forwarding the same JMS program using http tunnel > > works. The only difference I think is that the public interface goes via > > the istio Envoy proxy. > > Are there any issues using Netty http tunnelling via such proxies? > > Any pointers on how to debug this issue? > > Any help would be highly appreciated as I am kind of lost :). > > > > Thanks > > Sunil > > >
2022-05-22 14:16:36,383 INFO SKV:org.apache.activemq.artemis.core.protocol.ProtocolHandler$ProtocolDecoder;Switching to HTTP protocol! 2022-05-22 14:16:36,384 INFO SKV:org.apache.activemq.artemis.core.remoting.impl.netty.HttpAcceptorHandler;channelRead received POST request.ByteBuf:Capacity=7 2022-05-22 14:16:36,387 INFO SKV:org.apache.activemq.artemis.core.protocol.ProtocolHandler$ProtocolDecoder;NON HTTP;http=false;magic1=65;magic2=82;protocolToUse=CORE 2022-05-22 14:16:36,388 INFO SKV:org.apache.activemq.artemis.core.remoting.impl.netty.HttpAcceptorHandler;channelRead received POST request.ByteBuf:Capacity=21 2022-05-22 14:16:36,388 INFO SKV:org.apache.activemq.artemis.core.remoting.impl.netty.HttpAcceptorHandler$ResponseRunner;SENT HTTP RESPONSE;code=200 2022-05-22 14:16:36,389 INFO SKV:org.apache.activemq.artemis.core.remoting.impl.netty.HttpAcceptorHandler$ResponseRunner;SENT HTTP RESPONSE;code=200 2022-05-22 14:16:36,389 INFO SKV:org.apache.activemq.artemis.core.remoting.impl.netty.HttpAcceptorHandler;channelRead received POST request.ByteBuf:Capacity=21 2022-05-22 14:16:36,389 INFO SKV:org.apache.activemq.artemis.core.remoting.impl.netty.HttpAcceptorHandler;channelRead received POST request.ByteBuf:Capacity=18 2022-05-22 14:16:36,390 INFO SKV:org.apache.activemq.artemis.core.remoting.impl.netty.HttpAcceptorHandler$ResponseRunner;SENT HTTP RESPONSE;code=200 2022-05-22 14:16:36,449 INFO SKV:org.apache.activemq.artemis.core.remoting.impl.netty.HttpAcceptorHandler$ResponseRunner;SENT HTTP RESPONSE;code=200 2022-05-22 14:16:36,450 INFO SKV:org.apache.activemq.artemis.core.remoting.impl.netty.HttpAcceptorHandler;channelRead received POST request.ByteBuf:Capacity=2980 2022-05-22 14:16:36,489 INFO SKV:org.apache.activemq.artemis.core.remoting.impl.netty.HttpAcceptorHandler;channelRead received POST request.ByteBuf:Capacity=35 2022-05-22 14:16:36,490 INFO SKV:org.apache.activemq.artemis.core.remoting.impl.netty.HttpAcceptorHandler$ResponseRunner;SENT HTTP RESPONSE;code=200 2022-05-22 14:16:36,554 INFO SKV:org.apache.activemq.artemis.core.remoting.impl.netty.HttpAcceptorHandler;channelRead received POST request.ByteBuf:Capacity=13 2022-05-22 14:16:36,555 INFO SKV:org.apache.activemq.artemis.core.remoting.impl.netty.HttpAcceptorHandler$ResponseRunner;SENT HTTP RESPONSE;code=200
2022-05-22 14:10:02,982 INFO SKV:org.apache.activemq.artemis.core.protocol.ProtocolHandler$ProtocolDecoder;Switching to HTTP protocol! 2022-05-22 14:10:03,017 INFO SKV:org.apache.activemq.artemis.core.remoting.impl.netty.HttpAcceptorHandler;channelRead received POST request.ByteBuf:Capacity=7 2022-05-22 14:10:18,007 INFO SKV:org.apache.activemq.artemis.core.remoting.impl.netty.HttpAcceptorHandler$ResponseRunner;BOGUSRESPONSE!!!SENT HTTP RESPONSE 2022-05-22 14:10:18,044 INFO SKV:org.apache.activemq.artemis.core.protocol.ProtocolHandler$ProtocolDecoder;Switching to HTTP protocol! 2022-05-22 14:10:18,047 INFO SKV:org.apache.activemq.artemis.core.protocol.ProtocolHandler$ProtocolDecoder;NON HTTP;http=false;magic1=0;magic2=0;protocolToUse=HORNETQ 2022-05-22 14:10:18,049 INFO SKV:org.apache.activemq.artemis.core.remoting.impl.netty.HttpAcceptorHandler;channelRead received POST request.ByteBuf:Capacity=21 2022-05-22 14:10:18,049 INFO SKV:org.apache.activemq.artemis.core.remoting.impl.netty.HttpAcceptorHandler$ResponseRunner;SENT HTTP RESPONSE;code=200 2022-05-22 14:10:18,053 INFO SKV:org.apache.activemq.artemis.core.protocol.ProtocolHandler$ProtocolDecoder;NON HTTP;http=false;magic1=65;magic2=82;protocolToUse=CORE 2022-05-22 14:10:18,054 INFO SKV:org.apache.activemq.artemis.core.remoting.impl.netty.HttpAcceptorHandler;channelRead received POST request.ByteBuf:Capacity=18 2022-05-22 14:10:18,054 INFO SKV:org.apache.activemq.artemis.core.remoting.impl.netty.HttpAcceptorHandler$ResponseRunner;SENT HTTP RESPONSE;code=200 2022-05-22 14:10:18,058 INFO SKV:org.apache.activemq.artemis.core.remoting.impl.netty.HttpAcceptorHandler;channelRead received POST request.ByteBuf:Capacity=21 2022-05-22 14:10:18,058 INFO SKV:org.apache.activemq.artemis.core.remoting.impl.netty.HttpAcceptorHandler$ResponseRunner;SENT HTTP RESPONSE;code=200 2022-05-22 14:10:18,062 INFO SKV:org.apache.activemq.artemis.core.remoting.impl.netty.HttpAcceptorHandler;HttpAcceptorHandler_channelRead;method=GET 2022-05-22 14:10:18,074 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure to /10.5.43.204:43146 has been detected: io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpRequest cannot be cast to io.netty.buffer.ByteBuf [code=GENERIC_EXCEPTION] 2022-05-22 14:10:18,547 INFO SKV:org.apache.activemq.artemis.core.remoting.impl.netty.HttpAcceptorHandler;channelRead received POST request.ByteBuf:Capacity=2980 2022-05-22 14:10:18,547 INFO SKV:org.apache.activemq.artemis.core.remoting.impl.netty.HttpAcceptorHandler$ResponseRunner;SENT HTTP RESPONSE;code=200 2022-05-22 14:10:18,673 INFO SKV:org.apache.activemq.artemis.core.protocol.ProtocolHandler$ProtocolDecoder;Switching to HTTP protocol! 2022-05-22 14:10:18,675 INFO SKV:org.apache.activemq.artemis.core.remoting.impl.netty.HttpAcceptorHandler;HttpAcceptorHandler_channelRead;method=GET 2022-05-22 14:11:18,661 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure to /10.5.170.231:38374 has been detected: AMQ229014: Did not receive data from /10.5.170.231:38374 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT] 2022-05-22 14:11:18,662 WARN AMQ222061: Client connection failed, clearing up resources for session e8a5dc90-d9d8-11ec-8afb-0e39de063e6d
<?xml version="1.0"?> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <configuration xmlns="urn:activemq" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xi="http://www.w3.org/2001/XInclude" xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd"> <core xmlns="urn:activemq:core" xsi:schemaLocation="urn:activemq:core "> <name>dsmessagingienergy-master-0</name><persistence-enabled>true</persistence-enabled><!-- this could be ASYNCIO, MAPPED, NIO ASYNCIO: Linux Libaio MAPPED: mmap files NIO: Plain Java Files --><journal-type>ASYNCIO</journal-type><paging-directory>data/paging</paging-directory><bindings-directory>data/bindings</bindings-directory><journal-directory>data/journal</journal-directory><large-messages-directory>data/large-messages</large-messages-directory><journal-datasync>true</journal-datasync><journal-min-files>2</journal-min-files><journal-pool-files>10</journal-pool-files><journal-device-block-size>4096</journal-device-block-size><journal-file-size>10M</journal-file-size><!-- This value was determined through a calculation. Your system could perform 9.62 writes per millisecond on the current journal configuration. That translates as a sync write every 104000 nanoseconds. Note: If you specify 0 the system will perform writes directly to the disk. We recommend this to be 0 if you are using journalType=MAPPED and journal-datasync=false. --><journal-buffer-timeout>104000</journal-buffer-timeout><!-- When using ASYNCIO, this will determine the writing queue depth for libaio. --><journal-max-io>4096</journal-max-io><!-- You can verify the network health of a particular NIC by specifying the <network-check-NIC> element. <network-check-NIC>theNicName</network-check-NIC> --><!-- Use this to use an HTTP server to validate the network <network-check-URL-list>http://www.apache.org</network-check-URL-list> --><!-- <network-check-period>10000</network-check-period> --><!-- <network-check-timeout>1000</network-check-timeout> --><!-- this is a comma separated list, no spaces, just DNS or IPs it should accept IPV6 Warning: Make sure you understand your network topology as this is meant to validate if your network is valid. Using IPs that could eventually disappear or be partially visible may defeat the purpose. You can use a list of multiple IPs, and if any successful ping will make the server OK to continue running --><!-- <network-check-list>10.0.0.1</network-check-list> --><!-- use this to customize the ping used for ipv4 addresses --><!-- <network-check-ping-command>ping -c 1 -t %d %s</network-check-ping-command> --><!-- use this to customize the ping used for ipv6 addresses --><!-- <network-check-ping6-command>ping6 -c 1 %2$s</network-check-ping6-command> --><!-- how often we are looking for how many bytes are being used on the disk in ms --><disk-scan-period>5000</disk-scan-period><!-- once the disk hits this limit the system will block, or close the connection in certain protocols that won't support flow control. --><max-disk-usage>90</max-disk-usage><!-- should the broker detect dead locks and other issues --><critical-analyzer>true</critical-analyzer><critical-analyzer-timeout>120000</critical-analyzer-timeout><critical-analyzer-check-period>60000</critical-analyzer-check-period><critical-analyzer-policy>HALT</critical-analyzer-policy><page-sync-timeout>648000</page-sync-timeout><!-- the system will enter into page mode once you hit this limit. This is an estimate in bytes of how much the messages are using in memory The system will use half of the available memory (-Xmx) by default for the global-max-size. You may specify a different value here if you need to customize it to your needs. <global-max-size>100Mb</global-max-size> --><acceptors> <!-- useEpoll means: it will use Netty epoll if you are on a system (Linux) that supports it --> <!-- amqpCredits: The number of credits sent to AMQP producers --> <!-- amqpLowCredits: The server will send the # credits specified at amqpCredits at this low mark --> <!-- amqpDuplicateDetection: If you are not using duplicate detection, set this to false as duplicate detection requires applicationProperties to be parsed on the server. --> <!-- amqpMinLargeMessageSize: Determines how many bytes are considered large, so we start using files to hold their data. default: 102400, -1 would mean to disable large mesasge control --> <!-- Note: If an acceptor needs to be compatible with HornetQ and/or Artemis 1.x clients add "anycastPrefix=jms.queue.;multicastPrefix=jms.topic." to the acceptor url. See https://issues.apache.org/jira/browse/ARTEMIS-1644 for more information. --> <!-- Acceptor for every supported protocol --> <acceptor name="artemis">tcp://0.0.0.0:61616?handshake-timeout=0;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;amqpMinLargeMessageSize=102400;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true</acceptor> <!-- AMQP Acceptor. Listens on default AMQP port for AMQP traffic.--> <acceptor name="amqp">tcp://0.0.0.0:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpMinLargeMessageSize=102400;amqpDuplicateDetection=true</acceptor> <!-- STOMP Acceptor. --> <acceptor name="stomp">tcp://0.0.0.0:61613?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=STOMP;useEpoll=true</acceptor> <!-- HornetQ Compatibility Acceptor. Enables HornetQ Core and STOMP for legacy HornetQ clients. --> <acceptor name="hornetq">tcp://0.0.0.0:5445?anycastPrefix=jms.queue.;multicastPrefix=jms.topic.;protocols=HORNETQ,STOMP;useEpoll=true</acceptor> <!-- MQTT Acceptor --> <acceptor name="mqtt">tcp://0.0.0.0:1883?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=MQTT;useEpoll=true</acceptor> <acceptor name="netty-http-acceptor">tcp://0.0.0.0:8080?sslEnabled=false</acceptor> </acceptors><security-enabled>true</security-enabled><security-settings> <security-setting match="#"> <permission type="createNonDurableQueue" roles="amq,dsis-users,dsis-admins,dsis-ds-write,dsis-carto-write,dsis-ms-write,dsis-noti-write,dsis-plugin-write,dsis-port-write,dsis-pref-write,dsis-search-write,dsis-security-write,dsis-status-write,dsis-tr-write,dsis-unit-write,dsis-ut-write,dsis-vdb-write"/> <permission type="deleteNonDurableQueue" roles="amq,dsis-users,dsis-admins,dsis-ds-write,dsis-carto-write,dsis-ms-write,dsis-noti-write,dsis-plugin-write,dsis-port-write,dsis-pref-write,dsis-search-write,dsis-security-write,dsis-status-write,dsis-tr-write,dsis-unit-write,dsis-ut-write,dsis-vdb-write"/> <permission type="createDurableQueue" roles="amq,dsis-users,dsis-admins,dsis-ds-write,dsis-carto-write,dsis-ms-write,dsis-noti-write,dsis-plugin-write,dsis-port-write,dsis-pref-write,dsis-search-write,dsis-security-write,dsis-status-write,dsis-tr-write,dsis-unit-write,dsis-ut-write,dsis-vdb-write"/> <permission type="deleteDurableQueue" roles="amq,dsis-users,dsis-admins,dsis-ds-write,dsis-carto-write,dsis-ms-write,dsis-noti-write,dsis-plugin-write,dsis-port-write,dsis-pref-write,dsis-search-write,dsis-security-write,dsis-status-write,dsis-tr-write,dsis-unit-write,dsis-ut-write,dsis-vdb-write"/> <permission type="createAddress" roles="amq,dsis-users,dsis-admins,dsis-ds-write,dsis-carto-write,dsis-ms-write,dsis-noti-write,dsis-plugin-write,dsis-port-write,dsis-pref-write,dsis-search-write,dsis-security-write,dsis-status-write,dsis-tr-write,dsis-unit-write,dsis-ut-write,dsis-vdb-write"/> <permission type="deleteAddress" roles="amq,dsis-users,dsis-admins,dsis-ds-write,dsis-carto-write,dsis-ms-write,dsis-noti-write,dsis-plugin-write,dsis-port-write,dsis-pref-write,dsis-search-write,dsis-security-write,dsis-status-write,dsis-tr-write,dsis-unit-write,dsis-ut-write,dsis-vdb-write"/> <permission type="consume" roles="amq,dsis-users,dsis-admins,dsis-ds-write,dsis-carto-write,dsis-ms-write,dsis-noti-write,dsis-plugin-write,dsis-port-write,dsis-pref-write,dsis-search-write,dsis-security-write,dsis-status-write,dsis-tr-write,dsis-unit-write,dsis-ut-write,dsis-vdb-write"/> <permission type="browse" roles="amq,dsis-users,dsis-admins,dsis-ds-write,dsis-carto-write,dsis-ms-write,dsis-noti-write,dsis-plugin-write,dsis-port-write,dsis-pref-write,dsis-search-write,dsis-security-write,dsis-status-write,dsis-tr-write,dsis-unit-write,dsis-ut-write,dsis-vdb-write"/> <permission type="send" roles="amq,dsis-users,dsis-admins,dsis-ds-write,dsis-carto-write,dsis-ms-write,dsis-noti-write,dsis-plugin-write,dsis-port-write,dsis-pref-write,dsis-search-write,dsis-security-write,dsis-status-write,dsis-tr-write,dsis-unit-write,dsis-ut-write,dsis-vdb-write"/> <!-- we need this otherwise ./artemis data imp wouldn't work --> <permission type="manage" roles="amq,dsis-users,dsis-admins,dsis-ds-write,dsis-carto-write,dsis-ms-write,dsis-noti-write,dsis-plugin-write,dsis-port-write,dsis-pref-write,dsis-search-write,dsis-security-write,dsis-status-write,dsis-tr-write,dsis-unit-write,dsis-ut-write,dsis-vdb-write"/> </security-setting> </security-settings><address-settings> <!-- if you define auto-create on certain queues, management has to be auto-create --> <address-setting match="activemq.management#"> <dead-letter-address>DLQ</dead-letter-address> <expiry-address>ExpiryQueue</expiry-address> <redelivery-delay>0</redelivery-delay> <!-- with -1 only the global-max-size is in use for limiting --> <max-size-bytes>-1</max-size-bytes> <message-counter-history-day-limit>10</message-counter-history-day-limit> <address-full-policy>PAGE</address-full-policy> <auto-create-queues>true</auto-create-queues> <auto-create-addresses>true</auto-create-addresses> <auto-create-jms-queues>true</auto-create-jms-queues> <auto-create-jms-topics>true</auto-create-jms-topics> </address-setting> <!--default for catch all--> <address-setting match="#"> <dead-letter-address>DLQ</dead-letter-address> <expiry-address>ExpiryQueue</expiry-address> <redelivery-delay>0</redelivery-delay> <!-- with -1 only the global-max-size is in use for limiting --> <max-size-bytes>-1</max-size-bytes> <message-counter-history-day-limit>10</message-counter-history-day-limit> <address-full-policy>PAGE</address-full-policy> <auto-create-queues>true</auto-create-queues> <auto-create-addresses>true</auto-create-addresses> <auto-create-jms-queues>true</auto-create-jms-queues> <auto-create-jms-topics>true</auto-create-jms-topics> </address-setting> </address-settings><cluster-user>clusterUser</cluster-user><cluster-password>secret</cluster-password><connectors> <connector name="dsmessagingienergy-master-0">tcp://dsmessagingienergy-master-0.dsmessagingienergy-master.dsis-system-ienergy.svc.cluster.local:61616</connector> </connectors><cluster-connections> <cluster-connection name="dsmessaging-dsmessaging"> <address>jms</address> <connector-ref>dsmessagingienergy-master-0</connector-ref> <retry-interval>500</retry-interval> <retry-interval-multiplier>1.1</retry-interval-multiplier> <max-retry-interval>5000</max-retry-interval> <initial-connect-attempts>-1</initial-connect-attempts> <reconnect-attempts>-1</reconnect-attempts> <message-load-balancing>ON_DEMAND</message-load-balancing> <max-hops>1</max-hops> <static-connectors> <connector-ref>dsmessagingienergy-master-0</connector-ref> </static-connectors> </cluster-connection> </cluster-connections><addresses> <address name="DLQ"> <anycast> <queue name="DLQ"/> </anycast> </address><address name="ExpiryQueue"> <anycast> <queue name="ExpiryQueue"/> </anycast> </address><address name="DSIS_DataChanges"> <multicast> <queue name="DSIS_DataChanges.recent"/> </multicast> </address> <address name="dsis.test"> <multicast/> </address> </addresses><!-- Uncomment the following if you want to use the Standard LoggingActiveMQServerPlugin pluging to log in events <broker-plugins> <broker-plugin class-name="org.apache.activemq.artemis.core.server.plugin.impl.LoggingActiveMQServerPlugin"> <property key="LOG_ALL_EVENTS" value="true"/> <property key="LOG_CONNECTION_EVENTS" value="true"/> <property key="LOG_SESSION_EVENTS" value="true"/> <property key="LOG_CONSUMER_EVENTS" value="true"/> <property key="LOG_DELIVERING_EVENTS" value="true"/> <property key="LOG_SENDING_EVENTS" value="true"/> <property key="LOG_INTERNAL_EVENTS" value="true"/> </broker-plugin> </broker-plugins> --><ha-policy> <replication> <master> <group-name>dsmessagingienergy-0</group-name> <!--we need this for auto failback--> <check-for-live-server>false</check-for-live-server> </master> </replication> </ha-policy> </core><core xmlns="urn:activemq:core"> <jmx-management-enabled>true</jmx-management-enabled> </core> </configuration>