mattcasters opened a new pull request, #7379: URL: https://github.com/apache/hop/pull/7379
Here is a summary recap of the changes made for this pull request for issue #4120 : ### 1. Simple Mapping Fix (`SimpleMapping.java`) - **Problem**: When a parent pipeline runs in single-threaded mode (such as a Kafka message-processing sub-pipeline), Simple Mapping transforms failed to pass rows to their child pipeline, resulting in 0 rows outputted. - **Solution**: Updated `SimpleMapping` to detect if the parent pipeline has `PipelineType.SingleThreaded`. If so, it disables `rename_on_output` on input, wraps execution in a `SingleThreadedPipelineExecutor`, and triggers `data.executor.oneIteration()` inside the parent pipeline's `batchComplete()` step to sequentially pump rows through the mapping. ### 2. Exposing the Single-Threaded Engine Plugin - Created a new `@PipelineEnginePlugin` under `plugins/engines/single-threaded/` leveraging the existing core `SingleThreadedPipelineExecutor`. - Added the module to `plugins/engines/pom.xml`, registered it in `assemblies/client/pom.xml`, and included it in the packaged assemblies. - Defined `IMeasuringLocalPipelineRunConfiguration` to handle sampling across the standard local and single-threaded local engines. ### 3. Kafka Integration Tests - Re-enabled the Kafka integration tests suite. - Fixed a listener/host networking bug in `integration-tests-kafka.yaml` (advertised listener bound to `localhost` inside container, preventing test runner from reaching broker). - Created a new `main-0002-kafka-test-mapping` workflow containing a Kafka consumer executing a sub-pipeline with a mapping transform to verify and prevent regressions on this issue. ### 4. Documentation - Added user manual documentation at `docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/single-threaded-pipeline-engine.adoc` and updated the index. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
