vbhanuchander-lang opened a new pull request, #7860: URL: https://github.com/apache/hop/pull/7860
Towards #2653 Hop can talk to Kafka and to nothing else. This adds JMS transforms so RabbitMQ, ActiveMQ "Classic" and ActiveMQ Artemis can be read and written directly, which is what the issue asks for — "prevent developers from having to create middleware to pull messages off the queue and send them to HOP". ## On the design @mattcasters proposed making the message type a plugin type behind a single transform, with an interface for connect/disconnect, next message, message information and acknowledge. I have built **that interface and the first non-Kafka implementation**, but kept the interface inside this plugin rather than adding a plugin type to core. The reasoning: an interface validated by exactly one implementation is a guess. `IMessageQueueProvider` here is shaped by your sketch and exercised by a real JMS provider, so the shape is now testable against reality. Promoting it to a Hop plugin type — and moving the Kafka transforms behind it — is the natural next step, and it is a much better conversation to have with a working second broker in the tree than without one. Nothing here blocks that; it is an additive plugin and core is untouched. If you would rather I go straight to the plugin type in this PR, say so and I will. @fpapon — you noted ActiveMQ and Artemis are separate projects now but "it's all JMS so not a big deal". That is the approach taken: one JMS implementation, both brokers reachable, Artemis bundled and Classic through JNDI. ## What's included **`IMessageQueueProvider`** — connect / receive / acknowledge / send / close, with `MessageQueueRecord` carrying key, body, destination, timestamp and properties. Broker concepts that do not generalise stay behind the interface. **`JmsProvider`** — JMS 3.0 implementation. The ActiveMQ Artemis client is bundled so DIRECT mode works with no setup; the factory is constructed reflectively so the plugin still loads if that jar is removed. Any other JMS 3.0 provider works through JNDI by adding its client jar. **`JmsConnection` metadata type** (category Connections) so broker URL and credentials are configured once and shared by both transforms. **JMS consumer** — queue or topic into rows. Message selectors (evaluated broker-side), durable subscriptions, optional transactions, and five optional output fields (body, correlation id, destination, message id, timestamp). **JMS producer** — rows to a queue or topic as text messages, passing rows through unchanged so it can sit mid-pipeline. ## Two behaviours worth reviewing **Acknowledgement.** The consumer uses `CLIENT_ACKNOWLEDGE` and acknowledges only *after* `putRow`. `AUTO_ACKNOWLEDGE` would confirm on receipt, so a failure later in the pipeline would lose the message silently — the same class of problem as #7742. The docs are explicit that this guarantees the row was handed downstream, not that the pipeline finished with it. **Stopping.** Consumption ends on a message limit, a receive timeout with nothing waiting, or the pipeline stopping. I chose a draining consumer over one that blocks forever, because a pipeline that never ends is awkward to operate; continuous polling is a scheduled re-run. Happy to change this if you would rather it behaved like the Kafka consumer's streaming model. ## Tests 9 unit tests, all passing: - `JmsProviderTest` — the JMS-to-Hop mapping, which is the part that can be wrong without a broker: full `TextMessage` mapping including properties and the acknowledge handle; non-text messages yielding identifiers with a null body rather than a guessed encoding; absent timestamp; null property values. - `JmsConsumerMetaTest` — only named fields reach the output row and in declared order, timestamp typed as Date, variables resolved in field names, empty configuration producing an empty row, and destination-type parsing. ``` [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0 [INFO] BUILD SUCCESS ``` `spotless:apply` clean; `apache-rat:check` reports 0 unapproved over 22 files. **No broker integration test yet.** The right home is `integration-tests` with an Artemis container, in the style of the Kafka project I added in #7747. I did not want to guess at the harness for a new broker in the same PR that introduces the transforms — happy to add it here or as a follow-up, whichever you prefer. I could also not verify the assembly build locally (`assemblies/plugins` needs many unrelated artifacts my offline repo lacks). The change there is a six-line dependency entry copied from the Kafka one, plus the module in `plugins/transforms/pom.xml` and versions in `lib/pom.xml` — CI will confirm it. ## New dependencies | Artifact | Version | Why | |---|---|---| | `jakarta.jms:jakarta.jms-api` | 3.1.0 | The API compiled against. EPL-2.0 / GPL-2.0-with-CP. | | `org.apache.activemq:artemis-jakarta-client` | 2.42.0 | Bundled default client, Apache-2.0. | Both are declared in `lib/pom.xml` like every other managed version. Let me know if you would rather ship the API only and have users supply every client jar. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
