Re: Scaling Pipelines & Workflows

2022-10-18 Thread Diego Mainou
Sorry Big thumb issue... Here : " Additionally, it copies a document to the very same directory." I've seen endless issues with that behaviour. Particularly in cloud based solutions. Try copying to a different folder [ https://www.bizcubed.com.au/ |

Re: Scaling Pipelines & Workflows

2022-10-18 Thread Diego Mainou
Here : " Additionally, it copies a document to the very same directory." I've seen endless issues with that behaviour. Particularly in cloud based solutions. Tr [ https://www.bizcubed.com.au/ | ] Diego Mainou Project Delivery Ma

Re: HTTP GET and POST actions

2022-10-18 Thread Hans Van Akelyen
Hi Christian, We are always open to new feature requests, feel free to create a ticket and explain the use case and expected results. I think in this case you want to evaluate a return value and not just see if you get a status 200? For the pipeline Transforms, it’s hard to tell what is going wr

HTTP GET and POST actions

2022-10-18 Thread Himpe, Christian
Dear Hop Community, I am an new user and I am attempting to build a small "Workflow" in which a "GET" and then a "POST" are send to an HTTP endpoint and the answer "result: ok" is checked. Currently, I am doing this with scripts, but I was wondering, if I am overlooking the appropriate "action"

Re: Scaling Pipelines & Workflows

2022-10-18 Thread Hans Van Akelyen
Hi Jochen, Hop Server runs in a JVM, you can edit the hop-server.sh script to add extra options to the JVM (allocate more memory to the JVM default is 2048MB). Parallelisation and scaling, each Hop transform creates its own thread and will consume records on the input side and place them to the

[ANNOUNCE] Apache Hop 2.1.0

2022-10-18 Thread Bart Maertens
The Apache Hop PMC and community are pleased to announce the general availability of Apache Hop 2.1.0. This 2.1.0 release is the result of a massive effort by the Apache Hop community and contains four and a half months of work on over 200 tickets. The Hop Orchestration Platform, or Apache Hop, ai

Scaling Pipelines & Workflows

2022-10-18 Thread Jochen Gatternig
Dear all Are there option/parameters in the Hop server that allow parallelization and scaling of the processing? We tested it with a pipeline configuration which read data from a source table, created XMLs and write them to a filesystem. Additionally, it copies a document to the very same direc

Re: Memory management | Pipeline executor on Beam

2022-10-18 Thread Hans Van Akelyen
Thanks for the follow up Fabian! We will see how/if we can get HOP-4193 sorted out to support your full use case. Cheers, Hans On 18 October 2022 at 09:54:15, Fabian Peters (p...@mercadu.de) wrote: Hi all, To anybody reading this in the future: As Hans said, don't use Beam Direct for heavy lif

Re: Memory management | Pipeline executor on Beam

2022-10-18 Thread Fabian Peters
Hi all, To anybody reading this in the future: As Hans said, don't use Beam Direct for heavy lifting. On Dataflow, the original pipeline works w/o any issues, no workarounds required. I still would like to have a scheduled Dataflow job to run my entire workflow, avoiding any additional VM/Cont