By "in Java" I assume you more specifically mean 'using the same Core client as the existing senders/receivers' ?
On Mon, 25 Jul 2022 at 11:10, Jan Šmucr <jan.sm...@aimtecglobal.com> wrote: > > > For the broker, it would ordinarily treat the large message as a > > chunked stream if doing e.g core->core, or amqp->amqp...but in this > > case having to do core->amqp means the broker will necessarily have to > > convert the message and in doing so it will load the entire message. > > I think this is exactly what we needed to know to drop Python support and > create the Lambda in Java. Thank you! 😊 > > Jan > > From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com> > Sent: pondělí 25. července 2022 12:05 > To: users@activemq.apache.org<mailto:users@activemq.apache.org> > Subject: Re: Artemis - Large Message - Core => AMQP > > Are you using StreamMessage because you think it will enable > 'steaming'? If so I wouldnt bother, the Stream name isnt related, its > just a JMS message type name meaning there is a list/sequence of > things in it. Previously you just seemed to want to send text/bytes. > > For the Core client side you can see > https://activemq.apache.org/components/artemis/documentation/latest/large-messages.html > > For the broker, it would ordinarily treat the large message as a > chunked stream if doing e.g core->core, or amqp->amqp...but in this > case having to do core->amqp means the broker will necessarily have to > convert the message and in doing so it will load the entire message. > > (It seems like its that last step currently not handling your sent > Core messages, because you are sending raw Core payloads that dont > match the JMS style Core message structure the brokers core->AMQP > converter is geared around, meaning it falls back to one that tries to > convert it as a string, which presumably fails due to starting to read > your bare payload as a size indicator) > > On Mon, 25 Jul 2022 at 06:40, Jan Šmucr <jan.sm...@aimtecglobal.com> wrote: > > > > Thank you for the feedback. > > So what would this be the proper way of streaming files (if the code below > > is correct)? > > > > final String body = "foo"; > > > > try (Connection connection = createConnection()) { > > Session session = connection.createSession(false, > > Session.AUTO_ACKNOWLEDGE); > > MessageProducer producer = > > session.createProducer(session.createQueue(getQueueName())); > > StreamMessage message = session.createStreamMessage(); > > message.writeBytes(body.getBytes()); > > producer.send(message); > > } > > > > AmqpClient client = createAmqpClient(); > > AmqpConnection amqpConnection = addConnection(client.connect()); > > AmqpSession amqpSession = amqpConnection.createSession(); > > AmqpReceiver receiver = amqpSession.createReceiver(getQueueName()); > > receiver.flow(100); > > AmqpMessage amqpMessage = receiver.receive(1, TimeUnit.SECONDS); > > Assert.assertNotNull(amqpMessage); > > Assert.assertEquals(body, new String(((Binary) ((AmqpSequence) > > amqpMessage.getWrappedMessage().getBody()).getValue().get(0)).getArray())); > > > > We want to stream the payloads due to the fact that these can take up to > > hundreds of megabytes. > > > > Thank you. > > Jan > > > > From: Clebert Suconic<mailto:clebert.suco...@gmail.com> > > Sent: sobota 23. července 2022 20:50 > > To: users@activemq.apache.org<mailto:users@activemq.apache.org> > > Subject: Re: Artemis - Large Message - Core => AMQP > > > > We could look at enhancing the converters. But the best would be to change > > your producer to have the exact format the converter would have. > > > > On Sat, Jul 23, 2022 at 2:41 PM Clebert Suconic <clebert.suco...@gmail.com> > > wrote: > > > > > Change your message in a way is compatible ? > > > > > > On Fri, Jul 22, 2022 at 3:36 PM Jan Šmucr <jan.sm...@aimtecglobal.com> > > > wrote: > > > > > >> The reason for this is that there's a whole infrastructure built using > > >> the core protocol, and now we need to connect a Python-based Lambda > > >> capable > > >> of receiving large messages. Is there any other, core-compatible method? > > >> > > >> Jan > > >> > > >> Dne 22. 7. 2022 21:21 napsal uživatel Clebert Suconic < > > >> clebert.suco...@gmail.com>: > > >> If you expect conversions to happen, you have to send messages from > > >> qpid-ms or core-jms client. > > >> > > >> > > >> On the case: > > >> > > >> > > >> ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory(); > > >> Connection conn = factory.createConnection(...); > > >> Session session = conn.createSession(...) > > >> Producer producer = session.createProducer(....); > > >> producer.send(session.createTextMessage(YOUR-LARGE-BODY-GOES-HERE); > > >> > > >> > > >> > > >> Then the conversion from Core to AMQP should happen fine when you > > >> receive the message on the other side at your Python client. > > >> > > >> > > >> > > >> If you really must use the Core-API, then you will need to hack your > > >> way in and find what the JMS client would do.. but at that point it's > > >> just a non standard way of doing this... You would need to have a very > > >> good reason to not do the right thing on this case. > > >> > > >> On Fri, Jul 22, 2022 at 3:00 PM Clebert Suconic > > >> <clebert.suco...@gmail.com> wrote: > > >> > > > >> > If you’re not using qpid JMS it will not work. > > >> > > > >> > On Fri, Jul 22, 2022 at 8:06 AM Andy Yar <andyya...@gmail.com> wrote: > > >> >> > > >> >> There is another attempt employing AMQP test structures. It produces > > >> the same incorrect message body. > > >> >> > > >> >> On Fri, Jul 22, 2022 at 12:21 PM Andy Yar <andyya...@gmail.com> wrote: > > >> >>> > > >> >>> Hello, > > >> >>> Using the LargeServerMessageImpl was just an attempt providing a > > >> seemingly easy way to introduce a kind of Large Message into the test. > > >> I'm > > >> not familiar with the code base at all - so I simply tried to somehow > > >> provide the requested test replicating my "sending via Core -> receiving > > >> via Qpid" issue. The same applies to the explicit AMQP conversion - the > > >> encountered error simply pointed on it. Is there an intended "test way" > > >> to > > >> model the message transfer among different clients/protocols? > > >> >>> > > >> >>> Anyway, I've changed the test to employ > > >> ClientLargeMessageImpl/ClientMessageImpl instead of the server-related > > >> LargeServerMessageImpl with a very similar result. Does that > > >> implementation > > >> fit the simulated use case (sending via Core -> receiving via Qpid AMQP) > > >> better or is the explicit AMQP conversion misused? > > >> >>> > > >> >>> Please, see the attached patch. > > >> >>> > > >> >>> On Mon, Jul 18, 2022 at 3:50 PM Clebert Suconic < > > >> clebert.suco...@gmail.com> wrote: > > >> >>>> > > >> >>>> AMQP to Core and Core to AMQP conversions expect you using JMS > > >> clients > > >> >>>> to generate your message. There are certain caveats that both > > >> >>>> clients > > >> >>>> will add to the messages. > > >> >>>> > > >> >>>> > > >> >>>> Usually I have seen more users doing weird conversions in AMQP when > > >> >>>> they use a C++ or non Java Client.. but on your case you used a Core > > >> >>>> Client, using a non standard way to send a message (ServerMessage > > >> from > > >> >>>> the client)... and it's not clear the message is correctly set for a > > >> >>>> conversion to work. > > >> >>>> > > >> >>>> > > >> >>>> I need you to provide some explanation on what you're doing.. if > > >> >>>> this > > >> >>>> was a way to hack a bug your saw or if this is just the way you're > > >> >>>> using it. > > >> >>>> > > >> >>>> > > >> >>>> if this is how you're actually using, I would suggest you either use > > >> >>>> the proper APIs or if you're doing some hack for performance > > >> >>>> consideration you have to set the message in a better way the > > >> >>>> converters would understand... > > >> >>>> > > >> >>>> On Mon, Jul 18, 2022 at 9:38 AM Clebert Suconic > > >> >>>> <clebert.suco...@gmail.com> wrote: > > >> >>>> > > > >> >>>> > Are you using the LargeServerMessageImpl on your client, or that > > >> was > > >> >>>> > just a "hack" to reproduce your issue. > > >> >>>> > > > >> >>>> > > > >> >>>> > LargeServerMessageImpl was not meant to be used on the client. But > > >> if > > >> >>>> > you're doing that just to show something you faced in production > > >> it's > > >> >>>> > ok.. but using the LargeServerMessageImpl to send a Client Message > > >> is > > >> >>>> > a big not for me. > > >> >>>> > > > >> >>>> > The only reason we do that is for server to server transfer. > > >> >>>> > > > >> >>>> > On Mon, Jul 18, 2022 at 7:39 AM Andy Yar <andyya...@gmail.com> > > >> wrote: > > >> >>>> > > > > >> >>>> > > Hello, > > >> >>>> > > Yes, the 2.23.1 test instance was freshly created. > > >> >>>> > > > > >> >>>> > > I've attached a test as a part of ARTEMIS-3897 > > >> >>>> > > <https://issues.apache.org/jira/browse/ARTEMIS-3897>. I hope it > > >> helps. > > >> >>>> > > > > >> >>>> > > > > >> >>>> > > On Sat, Jul 16, 2022 at 5:21 PM Clebert Suconic < > > >> clebert.suco...@gmail.com> > > >> >>>> > > wrote: > > >> >>>> > > > > >> >>>> > > > Did you start with fresh data on 2.23.1. > > >> >>>> > > > > > >> >>>> > > > If you did please provide a self enclosing test reproducing > > >> your issue. > > >> >>>> > > > > > >> >>>> > > > On Fri, Jul 15, 2022 at 4:29 AM Andy Yar <andyya...@gmail.com> > > >> wrote: > > >> >>>> > > > > > >> >>>> > > > > A quick test using 2.23.1 results in the same error payload > > >> being > > >> >>>> > > > received: > > >> >>>> > > > > > > >> >>>> > > > > "Message(address='test', durable=True, priority=4, > > >> >>>> > > > > annotations=AnnotationDict({symbol('x-opt-jms-dest'): > > >> byte(0), > > >> >>>> > > > > symbol('x-opt-jms-msg-type'): byte(0)}), > > >> >>>> > > > > properties={'JMSXDeliveryCount': None, '_AMQ_LARGE_SIZE': > > >> 67}, > > >> >>>> > > > > body='Conversion to AMQP error: Error reading in > > >> simpleString, > > >> >>>> > > > > length=1330464032 is greater than readableBytes=62')" > > >> >>>> > > > > > > >> >>>> > > > > Best regards > > >> >>>> > > > > > > >> >>>> > > > > On Thu, Jul 14, 2022 at 9:18 PM Clebert Suconic > > >> >>>> > > > > <clebert.suco...@gmail.com> wrote: > > >> >>>> > > > > > > > >> >>>> > > > > > There's been a few fixes in AMQP Large message. > > >> >>>> > > > > > > > >> >>>> > > > > > > > >> >>>> > > > > > More prominently a fix with the JDBC implementation > > >> between 2.17 and > > >> >>>> > > > > HEAD. > > >> >>>> > > > > > > > >> >>>> > > > > > > > >> >>>> > > > > > I would definitely recommend you to upgrade. > > >> >>>> > > > > > > > >> >>>> > > > > > On Thu, Jul 14, 2022 at 10:26 AM Justin Bertram < > > >> jbert...@apache.org> > > >> >>>> > > > > wrote: > > >> >>>> > > > > > > > > >> >>>> > > > > > > I would expect this to work. Can you try this on the > > >> latest release > > >> >>>> > > > > (i.e. > > >> >>>> > > > > > > 2.23.1) [1]? > > >> >>>> > > > > > > > > >> >>>> > > > > > > If it still doesn't work please open a Jira with all the > > >> details > > >> >>>> > > > > necessary > > >> >>>> > > > > > > to reproduce the problem. An actual test-case would be > > >> ideal. > > >> >>>> > > > > > > > > >> >>>> > > > > > > Thanks! > > >> >>>> > > > > > > > > >> >>>> > > > > > > > > >> >>>> > > > > > > Justin > > >> >>>> > > > > > > > > >> >>>> > > > > > > [1] > > >> https://activemq.apache.org/components/artemis/download/ > > >> >>>> > > > > > > > > >> >>>> > > > > > > On Thu, Jul 14, 2022 at 9:20 AM Andy Yar < > > >> andyya...@gmail.com> > > >> >>>> > > > wrote: > > >> >>>> > > > > > > > > >> >>>> > > > > > > > Hello, > > >> >>>> > > > > > > > Sending a message as a Large Message via Core protocol > > >> and > > >> >>>> > > > receiving > > >> >>>> > > > > > > > it via AMQP on Artemis 2.17.0 ends with receiving the > > >> following > > >> >>>> > > > > string > > >> >>>> > > > > > > > as message body: > > >> >>>> > > > > > > > > > >> >>>> > > > > > > > "Conversion to AMQP error: Error reading in > > >> simpleString, length=y > > >> >>>> > > > is > > >> >>>> > > > > > > > greater than readableBytes=x" > > >> >>>> > > > > > > > > > >> >>>> > > > > > > > Broker lists the message as Type: Default and Large: > > >> True. > > >> >>>> > > > > > > > > > >> >>>> > > > > > > > The receiver is basically Qpid's: > > >> >>>> > > > > > > > > > >> >>>> > > > > > > > > > >> >>>> > > > > > > >> >>>> > > > > > >> https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py > > >> >>>> > > > > > > > > > >> >>>> > > > > > > > The doc > > >> >>>> > > > > > > > > > >> >>>> > > > > > > >> >>>> > > > > > >> https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html > > >> >>>> > > > > > > > states both Core and AMQP support Large Message. Is > > >> this approach > > >> >>>> > > > > > > > valid? > > >> >>>> > > > > > > > > > >> >>>> > > > > > > > Thanks > > >> >>>> > > > > > > > > > >> >>>> > > > > > > > > > >> >>>> > > > > > > > >> >>>> > > > > > > > >> >>>> > > > > > > > >> >>>> > > > > > -- > > >> >>>> > > > > > Clebert Suconic > > >> >>>> > > > > > > >> >>>> > > > -- > > >> >>>> > > > Clebert Suconic > > >> >>>> > > > > > >> >>>> > > > >> >>>> > > > >> >>>> > > > >> >>>> > -- > > >> >>>> > Clebert Suconic > > >> >>>> > > >> >>>> > > >> >>>> > > >> >>>> -- > > >> >>>> Clebert Suconic > > >> > > > >> > -- > > >> > Clebert Suconic > > >> > > >> > > >> > > >> -- > > >> Clebert Suconic > > >> > > > -- > > > Clebert Suconic > > > > > -- > > Clebert Suconic > > >
