mattcasters opened a new pull request, #7362: URL: https://github.com/apache/hop/pull/7362
# PR Description: Implement OData Input Transform Plugin ## Purpose This Pull Request implements a new **OData Input** pipeline transform plugin for Apache Hop ([Issue #7308](https://github.com/apache/hop/issues/7308)). This plugin allows users to stream entities from OData service collections (supporting both OData V2 and V4 protocols) directly into Hop pipelines. --- ## Key Features 1. **Lightweight Custom Engine**: - Engineered using Apache HttpClient 5.x and Jackson for parsing JSON payloads. - Built to avoid heavy external SDK dependencies, keeping the plugin package footprint minimal. 2. **Automatic Pagination**: - Out-of-the-box page fetching using `@odata.nextLink` (OData V4) and `__next` (OData V2) metadata attributes returned from service responses. 3. **Authentication Support**: - Supports `No Authentication`, `Basic Authentication` (Username/Password), and `Bearer Token` authentication types. - Implemented via a standard `ODataAuthType` enum that maps to translated descriptions inside the UI. 4. **Rich GUI Configuration Dialog (`ODataInputDialog`)**: - **Connection Tab**: Set endpoint URLs, authentication parameters, select entity sets, and retrieve available entity sets dynamically from the service root. - **Query Tab**: Configure standard OData operators like `$select`, `$filter`, `$orderby`, `$top`, and `$skip`. - **Fields Tab**: Defines Hop field mapping, data types, and formats. Features a **Get Fields** button that parses the OData service's `$metadata` XML endpoint (namespace-aware) to auto-populate schema mappings. 5. **AsciiDoc Documentation**: - Completed transform user guide integrating options descriptions, examples, and screenshot captures. --- ## Changes Made ### Core Plugin * `[NEW]` [ODataField.java](file:///home/matt/git/mattcasters/hop/plugins/transforms/odata/src/main/java/org/apache/hop/pipeline/transforms/odata/ODataField.java): Domain object for mapping JSON properties to Hop values. * `[NEW]` [ODataAuthType.java](file:///home/matt/git/mattcasters/hop/plugins/transforms/odata/src/main/java/org/apache/hop/pipeline/transforms/odata/ODataAuthType.java): Enum implementing `IEnumHasCodeAndDescription` for authentication profiles. * `[NEW]` [ODataInputMeta.java](file:///home/matt/git/mattcasters/hop/plugins/transforms/odata/src/main/java/org/apache/hop/pipeline/transforms/odata/ODataInputMeta.java): Metadata container, defining XML/JSON serialization. * `[NEW]` [ODataInputData.java](file:///home/matt/git/mattcasters/hop/plugins/transforms/odata/src/main/java/org/apache/hop/pipeline/transforms/odata/ODataInputData.java): Runtime buffers and HTTP caches. * `[NEW]` [ODataInput.java](file:///home/matt/git/mattcasters/hop/plugins/transforms/odata/src/main/java/org/apache/hop/pipeline/transforms/odata/ODataInput.java): Main execution logic, page resolver, and row producer. * `[NEW]` [ODataInputDialog.java](file:///home/matt/git/mattcasters/hop/plugins/transforms/odata/src/main/java/org/apache/hop/pipeline/transforms/odata/ODataInputDialog.java): SWT configuration UI. * `[NEW]` [messages_en_US.properties](file:///home/matt/git/mattcasters/hop/plugins/transforms/odata/src/main/resources/org/apache/hop/pipeline/transforms/odata/messages/messages_en_US.properties): Localization properties. * `[NEW]` [odata.svg](file:///home/matt/git/mattcasters/hop/plugins/transforms/odata/src/main/resources/odata.svg): Transform GUI icon. * `[NEW]` [pom.xml](file:///home/matt/git/mattcasters/hop/plugins/transforms/odata/pom.xml): Maven module descriptor. ### Build and Packaging * `[MODIFY]` [plugins/transforms/pom.xml](file:///home/matt/git/mattcasters/hop/plugins/transforms/pom.xml): Registered the `odata` transform submodule. * `[NEW]` [assembly.xml](file:///home/matt/git/mattcasters/hop/plugins/transforms/odata/src/assembly/assembly.xml): Transform assembly packager. * `[MODIFY]` [assemblies/plugins/pom.xml](file:///home/matt/git/mattcasters/hop/assemblies/plugins/pom.xml): Bundled OData plugin assembly within Hop Client ZIP packages. ### Integration Testing (Docker Compose) * `[NEW]` [integration-tests/odata/](file:///home/matt/git/mattcasters/hop/integration-tests/odata/): Integration test pipeline and workflow project (retrieving data from MockServer and asserting counts). * `[NEW]` [integration-tests-odata.yaml](file:///home/matt/git/mattcasters/hop/docker/integration-tests/integration-tests-odata.yaml): Docker Compose definition starting MockServer on `odata_mock` and configuring it with quiet `WARN` logging. * `[NEW]` [initializerJson.json](file:///home/matt/git/mattcasters/hop/integration-tests/odata/import/initializerJson.json): MockServer expectation definitions containing paginated customer datasets. Expectation with `$skip=2` query parameter set to `priority: 10` to ensure proper evaluation order. ### User Documentation * `[NEW]` [odata-input.adoc](file:///home/matt/git/mattcasters/hop/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/odata-input.adoc): Comprehensive user manual page for the transform. * `[MODIFY]` [nav.adoc](file:///home/matt/git/mattcasters/hop/docs/hop-user-manual/modules/ROOT/nav.adoc): Added to sidebar navigation. * `[MODIFY]` [transforms.adoc](file:///home/matt/git/mattcasters/hop/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms.adoc): Added to the transform plugin index. --- ## Verifications & Testing 1. **Compilation**: - The plugin compiles cleanly and builds successfully in parallel mode: ```bash ./mvnw clean install -T4 -Pfast-build ``` 2. **Formatting**: - Checked and formatted via Spotless/Checkstyle rules (0 warnings, 0 errors). 3. **Integration Tests**: - Executed the Docker-based testing runner: ```bash ./integration-tests/scripts/run-tests-docker.sh PROJECT_NAME=odata ``` - All tests exited with Code `0` (Success). Pipeline execution logs confirmed that 3 records were fetched across paginated requests and successfully matched count validation assertions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
