SEPURI-SAI-KRISHNA opened a new pull request, #13166:
URL: https://github.com/apache/seatunnel/pull/13166

   Yes, glad to contribute to Apache SeaTunnel.
   
   ### Purpose of this pull request
   
   Part of #13078 (slice 1, the queue source) for the "JMS / AMQP (generic)" 
row of #10753. The sink follows in a separate PR, as agreed there.
   
   This adds the `connector-jms` module with the `Jms` source. It reads one 
queue of any broker that has a JMS 1.1 (`javax.jms`) client, through that 
broker's own client library. `connector-activemq` and `connector-rabbitmq` are 
not changed.
   
   The design is the one reviewed in #13078:
   
   - **One transacted session per checkpoint epoch** on one connection per 
reader. `snapshotState(N)` seals the open session with no JMS call, the next 
`pollNext` closes the sealed consumer and opens the next session, and 
`notifyCheckpointComplete(N)` commits every sealed session up to N. An aborted 
checkpoint commits nothing; the next completed one commits it too. Uncommitted 
sessions are rolled back on close, so delivery is at least once.
   - **Receive and emit happen under the checkpoint lock**, one message at a 
time with `receiveNoWait()`, so no message is received before a barrier and 
emitted after it. An empty queue never holds the lock.
   - **A failed commit fails the reader**, and a connection loss reported to 
the `ExceptionListener` fails it before any commit is attempted. Received but 
uncommitted messages are bounded by `max_in_flight_messages` (default 1000).
   - **Connection factory**: `connection_factory_class` with 
`connection_factory_properties` applied through setters (dotted keys such as 
`prefetchPolicy.queuePrefetch` walk getters), or a JNDI lookup. Errors name the 
key, never the value.
   - **`cancel()`** closes the connection without taking any lock, which JMS 
allows from another thread.
   - `TextMessage` and `BytesMessage` bodies go through the existing `json` and 
`text` formats. Other types and empty bodies fail the reader without a commit; 
`ObjectMessage` is never deserialized; errors carry the message id, not the 
body.
   
   As confirmed by @SEZ9 on #13078: plugin `Jms`, module `connector-jms`, the 
only new dependency is `geronimo-jms_1.1_spec` 1.1.1 (Apache License 2.0), and 
vendor clients go in `${SEATUNNEL_HOME}/plugins/connector-jms/`. The docs list 
the jars for each tested client.
   
   Differences from the issue text:
   
   - The source's queue option is `destination`, the key the issue gives the 
sink.
   - **Correction to item 9.** The spike saw ActiveMQ Classic return `null` 
from `receiveNoWait()` right after a broker side drop. In the connector tests 
every provider, Classic included, threw on the first poll after the drop (2 to 
26 ms later), so a reader without the listener would not "idle forever" on 
these clients. The listener stays: it fails the reader before a commit is 
attempted, and 
`JmsSourceReaderTest#connectionLossReportedOnlyToTheExceptionListenerFailsTheReader`
 covers a provider that reports the loss only to the listener.
   - The Java baseline is now 11 (#11545). The test clients are still the 
versions from the issue: ActiveMQ Classic 5.15.16, Artemis 2.19.1, Qpid JMS 
0.61.0.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes, a new source connector `Jms`, documented in EN and ZH 
(`docs/*/connectors/source/Jms.md`). No existing option or behavior changes.
   
   ### How was this patch tested?
   
   **Unit tests** (84) in `connector-jms`, against brokers embedded in the test 
JVM and reached over TCP: ActiveMQ Classic (default prefetch, and 
`queuePrefetch` 0 set through a dotted property), Artemis core, and Artemis 
over AMQP 1.0 through Qpid JMS. The readers are configured the way users 
configure them, through `connection_factory_class` and its properties.
   
   - `JmsSourceReaderTest` (55): nothing committed before a completed 
checkpoint; messages received after a snapshot are not committed by it; an 
aborted checkpoint is committed by the next one; the final checkpoint commits 
from another thread after polling stopped; repeated notifications are harmless; 
a failed commit and a broker side drop fail the reader and lose nothing; 15 
rotations over a 30 message backlog read each message once; the in-flight 
bound; parallel readers; selector and `BytesMessage`; unsupported and empty 
messages; conversion errors do not expose the body; JNDI; cancel while 
connecting and before the first poll.
   - `JmsConnectionFactoriesTest` (16), `JmsSourceFactoryTest` (9), 
`JmsSourceSplitEnumeratorTest` (4, with a `DefaultSerializer` round trip of 
split and state).
   - Passing on JDK 17 and with the test JVM on JDK 11.
   
   Each design rule was also broken on purpose and the tests failed: committing 
the open epoch on completion (the naive single session), no exception listener, 
no in-flight bound, accepting empty bodies, ignoring a cancel during connect, 
committing only the completed checkpoint's own epoch, and not closing sealed 
consumers (the last one stalled Classic with prefetch 0, Artemis and Qpid).
   
   **E2E** `JmsIT` against `apache/activemq-artemis:2.43.0`, with the 27 jars 
of `artemis-jms-client` 2.19.1 copied into `plugins/connector-jms/` as the docs 
describe:
   
   - Zeta: a message is committed once a checkpoint completes (checked with the 
broker's `messageCount`), and with no completed checkpoint it stays on the 
queue and comes back after cancel.
   - Flink 1.13.6, 1.15.3, 1.18.0 and 1.20.1: a message is committed after a 
Flink checkpoint.
   - Spark is not supported and is disabled in the test.
   
   The test turns off the broker's disk usage check, because Artemis blocks 
producers once its disk is 90% full.
   
   Also run: `spotless:check`, `seatunnel-ci-tools` tests, `verify` for both 
new modules, and `ruff check` and `pytest` in `seatunnel-cli` for the alias 
change (335 passed).
   
   The branch is based on 26ad719a48. Merged locally with `dev` at e25c90a016, 
there are no conflicts, the 84 unit tests, `ConnectorOptionCheckTest` and the 
`seatunnel-cli` tests pass, and `JmsIT` compiles. I did not run `JmsIT` on that 
merge; its runs above are on the branch.
   
   ### Check list
   
   * [X] If any new Jar binary package adding in your PR, please add License 
Notice according
     [New License 
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md):
 `geronimo-jms_1.1_spec` 1.1.1 is Apache License 2.0 and is shaded into the 
connector jar, as `connector-activemq` already does through `activemq-client`; 
connector dependencies are outside `tools/dependencies/checkLicense.sh`. No 
vendor client is bundled.
   * [X] If necessary, please update the documentation to describe the new 
feature. https://github.com/apache/seatunnel/tree/dev/docs: `Jms` source and 
changelog, EN and ZH.
   * [X] If necessary, please update `incompatible-changes.md` to describe the 
incompatibility caused by this PR: not needed, new connector.
   * [X] If you are contributing the connector code, please check that the 
following files are updated:
     1. Update 
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
 and add new connector information in it: `seatunnel.source.Jms`
     2. Update the pom file of 
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml):
 done
     3. Add ci label in 
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml):
 `jms`
     4. Add e2e testcase in 
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/):
 `connector-jms-e2e`
     5. Update connector 
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config):
 done
   
     Also added the `jms` keyword to the AI CLI's connector aliases in 
`seatunnel-cli/seatunnel_cli/connectors.py`, next to `activemq`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] [Feature][Connector-V2] A... SEPURI-SAI-KRISHNA (via GitHub) via commits

Reply via email to