Re: Documentation

2021-03-10 Thread Hans Van Akelyen
dig even further? I know I wouldn't) and sorted them in what I hope is a logical order. Cheers, Hans On Mon, 22 Feb 2021 at 10:27, Hans Van Akelyen wrote: > I think we should indeed see the user manual as a user oriented and thus > Hop GUI manual, though it can still contain concep

Helm charts Available

2021-07-02 Thread Hans Van Akelyen
Hi All, In our efforts to make Hop available and easy to run on all sorts of platforms we have recently added Helm charts to our repository to help you deploy hop on K8s clusters. Currently we have added a single chart to deploy Hop Server. We will be adding charts for running Hop workloads and H

Re: How to use FTPS action

2021-07-19 Thread Hans Van Akelyen
Hi Florian, You are right, there used to be an action for FTPS, unfortunately we had to remove it because it was using non Apache License compatible libraries. You should however also be able to use the Copy Files or Move Files action and use a VFS url to retrieve the files. For more information

Re: How to use FTPS action

2021-07-20 Thread Hans Van Akelyen
> I also tried to add a path right after adn/or a port but nothing really > worked. > Once again, your help will be really much appreciated. > > Regards, > Florian Chevalier > > - Mail original - > De: "Hans Van Akelyen" > À: "users" >

Re: How to use FTPS action

2021-07-21 Thread Hans Van Akelyen
edException: > Connection closed without indication > > I guess this is due to the certificate which is not accepted when using > Hop. > Is there an action I can use to do this ? > > Regards, > Florian Chevalier > > - Mail original - > De: "Hans Va

Re: Get System Info component question / No type "command line Argument 1,2,3..."

2021-07-22 Thread Hans Van Akelyen
Hi Christophe, It seems to be an error in the docs, this functionality has been stripped out. I suggest moving to named parameters and using those, you could create a ticket if you have a feeling this functionality should be re-introduced. Cheers, Hans On Thu, 22 Jul 2021 at 09:32, Christophe Ch

Re: How to use FTPS action

2021-07-27 Thread Hans Van Akelyen
a.library.path=$LIBPATH > -Dtrust_all_cert=true -classpath "${CLASSPATH}" > org.apache.hop.ui.hopgui.HopGui $@ > But it don't seem to have any effect. > Should I add the parameter to all the scripts or is there a way to add a > Java parameter to Hop just at one place

Re: Best Practice for scheduling multiple workflow

2021-09-08 Thread Hans Van Akelyen
Hi Sothy, As with most answers, it depends. Hop workflows and pipelines can be scheduled in numerous ways and if you already have scheduling software present in your organization you can add the workflows to that one, eg. airflow/luigi/... That being said, using cron is also a commonly used solut

Re: General Questions about Hop 1.0

2021-11-01 Thread Hans Van Akelyen
Hi Peter, Thank you for trying out Hop, I will answer as much as I can inline. We really value the feedback. On a general note, you are totally right our architecture, infrastructure docs are lacking/missing at this point. We have mainly focussed on getting all features working and should now move

Re: MySQL Connection Issue

2021-11-06 Thread Hans Van Akelyen
Hi Anthony, As Diego already stated due to license restrictions we can not include this driver in the distribution. When going to the docs of the mysql database connection [1] you will see a link on where to download the driver. I do notice it gives you a link on where to download the driver, it d

Apache Hop is now a Top Level Project

2021-12-17 Thread Hans Van Akelyen
Hi Hopperians, We are pleased and proud to announce that Hop has graduated from incubator and is now a Top Level Project. This marks the end of a journey that started over a year ago, and the beginning of a new chapter for Apache Hop. The graduation will increase Hop's reach and audience. We will

Re: installing HOP WEB on a server

2021-12-18 Thread Hans Van Akelyen
Hi Tore, We have an instruction page on how to deploy Hop Web to a tomcat web server (https://hop.apache.org/dev-manual/latest/hopweb/index.html). We currently are not releasing the artefacts needed to do this so you will first have to do a local build of the source code to have the necessary part

Resources have been moved

2021-12-30 Thread Hans Van Akelyen
Hi All, This is a short message to inform you all that our resources have been moved by the INFRA team. These are the new locations for our artifacts and source code. Github: - source code: https://github.com/apache/hop - Website source: https://github.com/apache/hop-website Docker Hub:

Re: elasticsearch output

2022-01-18 Thread Hans Van Akelyen
Hi Hans-Peter, The Beam API has Elasticsearch support (version 7.9.2) so we could add it. I have created a jira case for you to implement these Read and Write Transforms [1]. Feel free to create tickets for any features you would like to have or need to make your project successful. Kind regards

Re: Adapted pipeline kafka-consumer-read-pokemon-data

2022-01-19 Thread Hans Van Akelyen
Hi, We also have a new version being released tomorrow which contains some code hardening on the kafka transforms. If the issue then still persists we can take a deep dive together with you to see what is happening. Cheers, Hans On Wed, 19 Jan 2022 at 15:30, Sergio Ramazzina (SERASOFT) < sergio.

Re: Adapted pipeline kafka-consumer-read-pokemon-data

2022-01-19 Thread Hans Van Akelyen
Hi Hans-Peter, A freezing UI is definitely a no-go so I am interested in a reproduction path for this so we can avoid this. The Kafka transforms have received some love and are being used in projects so I'm sure we will figure it out. As getting all the information via email might be a painful pro

Re: Adapted pipeline kafka-consumer-read-pokemon-data

2022-01-20 Thread Hans Van Akelyen
t; > sergio.ramazz...@serasoft.it>: > >> Hi Hans-Peter, >> >> I totally agree with Hans Van Akelyen. We have kafka consumer in >> production without any problem so a reproduction path is very welcome. >> >> Best >> >> Sergio >> Il

Re: Adapted pipeline kafka-consumer-read-pokemon-data

2022-01-20 Thread Hans Van Akelyen
Hi > > Exactly ☺ > I have attached it. > > Op do 20 jan. 2022 om 11:20 schreef Hans Van Akelyen < > hans.van.akel...@gmail.com>: > >> Do you mean you added a self reference to the current pipeline in there? >> In any case a reproduction path would be nice, we

Re: Trouble with Hop (V1.0.0 and 1.1.0) GUI on macbook

2022-01-25 Thread Hans Van Akelyen
Hi, On macos we follow the setting of the system so for hop to use light mode you would need to set macos to light theme. Cheers, Hans On Tue, 25 Jan 2022, 22:07 Nick Noordam, wrote: > Hi Bart, > > Thanks for the quick response! > Dark mode: Both are enabled to begin with. I disable them, clic

Re: AWS S3 Integration

2022-01-26 Thread Hans Van Akelyen
Hi David, One thing we do not have yet is a listing of the buckets so the lowest level you can go is s3://bucket-name and that should work. [image: Screenshot 2022-01-26 at 22.40.52.png] Kind regards, Hans On Wed, 26 Jan 2022 at 21:53, David Hughes wrote: > Hi Matt, > > Thank you for the up

Re: running workflows imported from PDI

2022-02-14 Thread Hans Van Akelyen
Hi Paul, In your workflow you will probably have other pipeline or workflow actions, In the import dialog there is an option to specify the default pipeline and workflow run configuration, if these were not set the actions to run a pipeline or workflow will not have a run configuration specified a

Applications for Travel Assistance to ApacheCon NA 2022 now open

2022-04-04 Thread Hans Van Akelyen
The ASF Travel Assistance Committee (TAC) is pleased to announce that travel assistance applications for ApacheCon NA 2022 are now open! We will be supporting ApacheCon North America in New Orleans, Louisiana, on October 3rd through 6th, 2022. TAC exists to help those that would like to attend Ap

Re: [Question] How to call the REST API with argument on Apache Hop Web Service?

2022-04-19 Thread Hans Van Akelyen
Hi Light, I hope you are well, and sorry for the delay. The steps you have followed seem correct but I was unable to reproduce your problem. The only "mistake" I can think of that you possibly made by following the example is in the usage of the "Get Variables" transform. In the Get variables Tra

Re: Apache Hop Kafka Consumer throws error when I run pipeline using the Flink Engine

2022-04-22 Thread Hans Van Akelyen
Hi Monajit, This is the auto scaling nature of Flink fighting against the requirement of having a single threaded pipeline for Kafka messages (as we need to know when messages are finished. When running on Flink the best solution would be to use the Beam Kafka Consumer. Another solution (but not

Re: Apache Hop Kafka Consumer throws error when I run pipeline using the Flink Engine

2022-04-23 Thread Hans Van Akelyen
eam Kafka Consumer but the >>> issue is it only supports AVRO. I need to consume json messages >>> >>> Thanks >>> Mono >>> >>> On Fri, Apr 22, 2022 at 12:21 AM Hans Van Akelyen < >>> hans.van.akel...@gmail.com> wrote: >>> &g

Re: Apache Hop Kafka Consumer throws error when I run pipeline using the Flink Engine

2022-04-25 Thread Hans Van Akelyen
If we can solve this kafka >>> consumer issue, then we are all set for prime time. >>> >>> Really appreciate your responses so far. >>> >>> Thanks >>> Mono >>> >>> On Fri, Apr 22, 2022, 15:49 Matt Casters wrote: >>>

Re: Apache Hop Kafka Consumer throws error when I run pipeline using the Flink Engine

2022-05-16 Thread Hans Van Akelyen
sitently when >> I run it on Flink or Beam Runner using the GUI. It did a few times and then >> just stopped working. The Beam producer works fine. Not sure what am I >> missing here >> >> Thanks >> Mono >> >> On Mon, Apr 25, 2022 at 10:18 AM Hans Van A

Re: Neo4j Cypher Action - Implicit Transactions

2022-07-01 Thread Hans Van Akelyen
Hi Axel, As the Cypher script action allows you to execute multiple scripts separated by a semicolon the code contains an explicit transaction and commits after all commands are executed successfully. I added a ticket [1] to investigate if we can add an option to avoid this and allow PERIODIC COM

Re: Rest transform on Beam: NoClassDefFoundError: javax/ws/rs/core/Link$Builder

2022-07-14 Thread Hans Van Akelyen
Hi Fabian, Thanks for testing things out, you have indeed found a situation where the base Beam libraries are conflicting with our transforms. When using the Hop engine we are able to keep the classpaths separate and each transform/plugin is isolated in its own classloader. When running in Beam w

Re: PARQUET FILE OUTPUT Column Datatypes

2022-08-05 Thread Hans Van Akelyen
Hi Jonathan, We translate all our types to the supported Parquet native types which are documented here [1]. We have not implemented the notion of the logical types in this output transform. The cast from Hop types to Parquet types can be found here [2]. All of our data types should be supported

Re: Access input ?

2022-08-07 Thread Hans Van Akelyen
Hi, Thank you for reaching out, we can take a look at adding a transform to read data from access so feel free to create a jira ticket. In the meantime there is another solution, there is a JDBC driver available to read data from access files you can find it here [1]. You can use a generic connec

Re: Difference between "Unit test" and "Development" in "Type of test" dropdown in "Create and add data sets" dialog

2022-08-09 Thread Hans Van Akelyen
Hi Gerhard, The "Type of test" is just a label. There is no functional difference, you can even fill in a custom value. When using the "Execute Unit Tests" transform you can however use this label to filter which types of tests should be run. Kr, Hans On Tue, 9 Aug 2022 at 10:41, MITTERLECHNER G

Re: Dataflow template creation

2022-08-10 Thread Hans Van Akelyen
Hi Fabian, You have indeed found something we have not yet documented, mainly because we have not yet tried it out ourselves. The main class that gets called when running Beam pipelines is "org.apache.hop.beam.run.MainBeam". I was hoping the "Import as pipeline" button on a job would give you eve

Re: Running multiple workflows from a batch file

2022-08-12 Thread Hans Van Akelyen
Hi Andreas, Are you trying to run workflows from different projects in this runall workflow? This will probably not work as everything will be running from the context of the workflow starting the other workflows. You will either have to rethink your project structure in this case or split the se

Re: Running multiple workflows from a batch file

2022-08-16 Thread Hans Van Akelyen
after the other. > > Is there a solution for this? > > Best regards, > Andreas > > > Am Fr., 12. Aug. 2022 um 11:49 Uhr schrieb Hans Van Akelyen < > hans.van.akel...@gmail.com>: > >> Hi Andreas, >> >> Are you trying to run workflows from diff

Re: [EXTERN]Re: Difference between "Unit test" and "Development" in "Type of test" dropdown in "Create and add data sets" dialog

2022-08-16 Thread Hans Van Akelyen
code, it seems that the two types “UNIT_TEST” and > “DEVELOPMENT” are hard coded ENUM values in file “TestType.java”: > > “public enum TestType { > > DEVELOPMENT, > > UNIT_TEST; > > } > > “ > > > > Best regards, > > Gerhard > > > > > &g

Re: Dataflow template creation

2022-08-16 Thread Hans Van Akelyen
POINT ["/opt/google/dataflow/java_template_launcher"] >> >> >> And "todays-directories.json": >> >> { >> "defaultEnvironment": {}, >> "image": " >> europe-west1-docker.pkg.dev/dashboard-foo/dataflow/hop:latest&qu

Re: Dataflow template creation

2022-08-18 Thread Hans Van Akelyen
o see whether I can get a bit further, but if you have time to > have a look at it, I'd much appreciate! > > cheers > > Fabian > > Am 16.08.2022 um 11:09 schrieb Hans Van Akelyen < > hans.van.akel...@gmail.com>: > > Hi Fabian, > > Did you g

Re: Dataflow template creation

2022-08-18 Thread Hans Van Akelyen
pects that a job file gets generated in a specific location and it will then pick up this file to execute the actual job. This is the part we would probably have to change our code a bit to save the job specification to that location and not start another job via the Beam API. Until we get t

Re: Dataflow template creation

2022-08-24 Thread Hans Van Akelyen
o Hans, > > Just catching up on the day's mails. I'm really grateful to you for > looking into this in depth and even coming up with a working setup! I've > been swamped with other work but would never have gotten this far anyway. > Nogmaals bedankt! ;) > >

Re: Common local and Beam file output

2022-08-25 Thread Hans Van Akelyen
Hi Fabian, Did you try running those transforms with the "SINGLE_BEAM" option in the number of copies? (for more info here in the Non-Beam output transforms section) This being said, we are working on

Re: Dimension lookup/update

2022-08-30 Thread Hans Van Akelyen
Hi Mike, The Dimension Lookup/update transform has been created to support Type 2 Slowly changing dimensions explanation about slowly changing dimensions can be found on wikipedia . It indeed means you can have multiple rows for the same ke

Re: Dimension lookup/update

2022-08-30 Thread Hans Van Akelyen
Another solution to avoid all the lookups is to use a table input and load the source table. Then you can use "Merge Rows (diff)" to do a comparison between the old set and the new set. It will mark the rows as new,identical,changed,deleted after that you can use "Synchronize after merge" and it wi

Re: Download options for Java 11

2022-08-30 Thread Hans Van Akelyen
Hi Andreas, We have a documentation page with the different Java builds that we know work. Feel free to pick one. https://hop.apache.org/manual/latest/supported-jvms.html Cheers, Hans On Tue, 30 Aug 2022 at 15:06, Andreas Krause wrote: > Hi Hop users, > > Hop official java version is 11. I've

Re: Dimension lookup/update

2022-08-31 Thread Hans Van Akelyen
; > *Sent:* Tuesday, August 30, 2022 at 1:53 PM > *From:* "Hans Van Akelyen" > *To:* users@hop.apache.org > *Subject:* Re: Dimension lookup/update > Another solution to avoid all the lookups is to use a table input and load > the source table. > Then you can use &quo

Re: Workflow on GCP with BigQuery Output

2022-08-31 Thread Hans Van Akelyen
Thank you Israel for the support! @Fabian, so the solution in your case was to change the Temp location in the Direct runner to a GCS path? In our documentation about the runner config we state that it should be a GCS location for the Dataflow runner, in the Direct runner we do not state this exp

Re: Join rows transform loops forever on Beam

2022-09-02 Thread Hans Van Akelyen
Hi Fabian, Merge join does not require your data to be sorted when executing on Beam https://hop.apache.org/manual/latest/pipeline/beam/getting-started-with-beam.html#_universal_transforms Chases, Hans On Fri, 2 Sep 2022 at 08:34, Fabian Peters wrote: > Good morning Matt, > > Thanks for your q

Re: 'Group by' fails remotely when is part of a workflow, works locally

2022-09-07 Thread Hans Van Akelyen
Hi Maykel, Thank you for the ticket, we will take a look at it as soon as possible. Cheers, Hans On Mon, 5 Sept 2022 at 15:44, Maykel Moya wrote: > Hello > > I've reported the issue with a reproducer included: > > https://issues.apache.org/jira/browse/HOP-4194 > > Regards > m > > On 2/9/22 17:

Re: Merge join issue on Beam [was: Join rows transform loops forever on Beam]

2022-09-30 Thread Hans Van Akelyen
) >> 2022/09/02 09:20:20 - General - >> at >> org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:228) >> 2022/09/02 09:20:20 - General - >> at >> org.apache.beam.repackaged.direct_java.runners.core.Sim

Re: Memory management | Pipeline executor on Beam

2022-10-10 Thread Hans Van Akelyen
Hi Fabian, Could you provide a bit more information? in the past couple of weeks, some major changes have been made to improve the performance. Are you using a Hop local engine configuration when executing the pipeline executor or trying the Beam-Direct? If it is the second I fear that's not reall

Re: Memory management | Pipeline executor on Beam

2022-10-18 Thread Hans Van Akelyen
or does not work on Beam. But I've found a workaround for now by executing the "embedded" pipeline via the local runner and writing the results to GCS, then picking them up in a later pipeline to get inserted in BigQuery. cheers Fabian Am 10.10.2022 um 18:35 schrieb Hans Van A

Re: Scaling Pipelines & Workflows

2022-10-18 Thread Hans Van Akelyen
Hi Jochen, Hop Server runs in a JVM, you can edit the hop-server.sh script to add extra options to the JVM (allocate more memory to the JVM default is 2048MB). Parallelisation and scaling, each Hop transform creates its own thread and will consume records on the input side and place them to the

Re: HTTP GET and POST actions

2022-10-18 Thread Hans Van Akelyen
Hi Christian, We are always open to new feature requests, feel free to create a ticket and explain the use case and expected results. I think in this case you want to evaluate a return value and not just see if you get a status 200? For the pipeline Transforms, it’s hard to tell what is going wr

Re: [ANNOUNCE] Apache Hop 2.1.0

2022-10-19 Thread Hans Van Akelyen
Hi Andreas, The best way to avoid problems when switching between versions is to use the HOP_CONFIG_FOLDER system variable. Another way to upgrade is to copy over the config folder from your old to your new installation. More instructions can be found in our installation and configuration guide (

Re: Where I can find variable for 'Input location' for Beam Input?

2022-11-04 Thread Hans Van Akelyen
Hi, Everything is stored under the config folder in your hop directory. You can also save all this information externally by setting the HOP_CONFIG_FOLDER environment variable. For more useful variables https://hop.apache.org/manual/latest/protips/index.html Cheers, Hans On 4 November 2022 a

Re: Beam File Input - unexpected result

2022-11-04 Thread Hans Van Akelyen
Hi, Seems like an odd thing you are encountering. Not sure how you are ending up with that result, if you think you are hitting a bug feel free to create a ticket with a reproduction path. As for debugging, you are right that the Beam file definitions currently have no way of doing a best guess

Re: Beam File Input - unexpected result

2022-11-07 Thread Hans Van Akelyen
will Beam engine give me some performance gain comparing to Hop processing engine? I mean in my scenario where I need to join/merge information from two or more files (large files; each almost millions of lines). Best Sent: Friday, November 04, 2022 at 10:10 PM From: "Hans Van Akelyen&q

Release 2.2 cutoff period

2022-11-22 Thread Hans Van Akelyen
Hi All, Unless there are objections I would like to mark the cutoff date for Hop version 2.2 on the 31st of November. We have 110 tickets done and then we can wrap up this release before everyone enters vacation mode. Cheers, Hans

Github Issues have been activated

2022-11-29 Thread Hans Van Akelyen
Dear Users and developers, As discussed in previous threads we will be moving to GitHub issues for better inclusion and follow up of our work. We will not be migrating the current Jira issues to GitHub. We will use this opportunity to cleanup the existing issues, many of them have become invalid.

Re: Beam File Input - unexpected result

2022-11-29 Thread Hans Van Akelyen
after pipeline execution. Regards Sent: Friday, November 04, 2022 at 10:10 PM From: "Hans Van Akelyen" To: users@hop.apache.org Subject: Re: Beam File Input - unexpected result Hi, Seems like an odd thing you are encountering. Not sure how you are ending up with that result, if you think

Re: Any trick to read data from Excel file where no of columns is not known?

2022-12-02 Thread Hans Van Akelyen
It’s not part of our distribution but there is a python transform [1] Cheers, Hans [1] https://github.com/m-a-hall/hop-cpython On 2 December 2022 at 12:59:04, pod...@gmx.com (pod...@gmx.com) wrote: Thank you Matt, that looks quite mosaic :-) Maybe I'd try to use Python in the first stages. B

2.4 cutoff date

2023-03-14 Thread Hans Van Akelyen
Hi Hoppers, We have been doing a release every 2 months. I think keeping up our bi-monthly release frequency is a good pacing, currently we have 72 closed tickets. Following this pacing will put our next cutoff date at Friday 24th March. Please finish up what you still want to include in the 2.4 r

Re:

2023-03-17 Thread Hans Van Akelyen
Hi Andreas, You can use the String Operations transform for this. https://hop.apache.org//manual/latest/pipeline/transforms/stringoperations.html Cheers, Hans On 17 March 2023 at 17:11:44, Andreas Krause (and78...@gmail.com) wrote: Hi all, i would like to see a "Replace CR / LF for all string

Re: DB2 JDBC4

2023-03-20 Thread Hans Van Akelyen
After adding the option to the connection and pressing the test button is the parameter correctly added to the url? I’m also not sure if “false” is a valid value for this setting according to the docs https://www.ibm.com/docs/en/db2/11.5?topic=SSEPGG_11.5.0/com.ibm.db2.luw.apdv.java.doc/src/tpc/im

Jira project has been disabled

2023-03-20 Thread Hans Van Akelyen
Dear Hop users, A while back we announced that our primary location for handling requests and issues will be moved to GitHub. We can now announce that this entire process has been completed. All remaining tickets in Jira have been reviewed and moved to GitHub if deemed actionable. A cross referen

Re: How to replace Parameter with ? question mark

2023-04-18 Thread Hans Van Akelyen
Hi, The Execute SQL script is a row processing transform, it requires input rows to work. The values used to replace the ? Need to come from a field in the stream. The simplest way to make your example work is to add a Data Grid transform [1] to the canvas, in the Meta tab create a field named “c

Re: Exporting Metrics tab into Excel file

2023-04-19 Thread Hans Van Akelyen
Hi There, You can export all logging information to any source you want. For this we have created a logging concept that executes a pipeline in the backend to store the information you want. You can read more about this in the docs [1] [2] [3] Cheers, Hans [1] logging basics: https://hop.apache.

Re: Scheduling Pipeline/Workflow

2023-05-14 Thread hans . van . akelyen
Hi All, For dataflow you can also take advantage of the scheduler in dataflow itself. There is a how-to on using the jobs tab in dataflow [1]. To use this you only need to place your pipeline and the needed metadata in a Google Storage bucket and you can fire off a pipeline on a schedule. We tr

Re: Hop and Casts for Postgres

2023-05-22 Thread hans . van . akelyen
Hi Andrew, You can use the select values transform and in there use the metadata tab to change the datatype of a field. This can be used to change dates to strings or int/strings to boolean and so on [1]. Hope this helps, Kr, Hans [1]  https://hop.apache.org//manual/latest/pipeline/transforms

2.5 Release

2023-05-23 Thread hans . van . akelyen
Hi friends, This is a notice that we are ending the development period for our 2.5 release the end of this week (28 May). Quite some work has been done for this release and I would like to thank everyone involved in making everything we do here possible! We will continue our 2 month release cyc

Re: 2.5 Release

2023-05-25 Thread hans . van . akelyen
Hi all, Let’s extend it one week to the 4th of June. The Apache Beam team is also working on a release (2.48.0 [1]) and then we can also include this dependency upgrade in our 2.5 release. Cheers, Hans [1] https://lists.apache.org/thread/hdcodx02fot3s9qt6n4457cj2prfcbl9 On 25 May 2023 at 14:

Re: Fw: ETL metadata injection - how to add header to text file

2023-05-31 Thread hans . van . akelyen
Hi, Providing a String with value Y/1/true should work to set that checkbox to true. If you always need a header it is also possible to turn it on in the template and not worry about the injection. Cheers, Hans On 31 May 2023 at 14:54 +0200, pod...@gmx.com, wrote: > Sorry for bringing this up, b

Re: CPU usage spikes to 100%

2023-06-29 Thread hans . van . akelyen
Hi Andreas, First step I would suggest is to try the latest version. To give a full answer to the rest of the question we would need some more information, what is your definition of a simple workflow? What are you doing? If your workflow consists of pipelines and those pipelines contain transf

Re: How to get date - x days in calculator?

2023-07-26 Thread hans . van . akelyen
Hi There, The calculator transform has the solution for you! Though a bit a brain scratcher... In the calculator you have the option to do “Date A + B Days”. Just throw in a negative number and it will go back in time ;) Cheers, Hans On 26 Jul 2023 at 19:16 +0200, pod...@gmx.com, wrote: > > Dear

Re: workflowLogChannelId

2023-08-23 Thread hans . van . akelyen
Hi Rob, Yes this is possible. We have the following internal variables available for usage: Workflow: ${Internal.Workflow.ID} Pipeline: ${Internal.Pipeline.ID} Cheers, Hans On 24 Aug 2023 at 00:25 +0200, Rob Burgess , wrote: > Hi > > Is there a system variable for the workflowLogChannelId and I

Re: Read files from HDFS

2023-10-19 Thread hans . van . akelyen
Hi Chenghao, How are you formatting your path? For files on hdfs you file path should be something like "hdfs://somehost:8080/downloads/some_dir" Cheers, Hans On 19 Oct 2023 at 11:00 +0200, Che Chenghao , wrote: > Hi Bart, > > Thanks for your reply! > > I tried to get filenames from hdfs and wr

Re: How to create an array of String in Apache Hop

2023-11-08 Thread hans . van . akelyen
Hi Michel, I think you can have the desired result by using the “JSON Fragment” option in the Enhanced JSON Output Transform. See following sample:  https://drive.google.com/file/d/1qlJOCUE29eqZ73JHRzSVTT4FIwWKodIR/view?usp=sharing Cheers, Hans On 8 Nov 2023 at 13:04 +0100, M , wrote: > > "myS

Re: Load files in a BLOB field of a database

2023-11-08 Thread hans . van . akelyen
Hi Chris, The only thing that function does is create a new byte[size]. You should be able to replace it by Var content = new byte[file.length()] Cheers, Hans On 8 Nov 2023 at 15:24 +0100, ch...@radical-airlines.com, wrote: > Hello ! > > Once upon a time in the old PDI archive we had a sample KT

[DISCUSS] Old dockerhub images

2023-12-12 Thread hans . van . akelyen
Hi all, Someone pointed out that we have 2 old repositories still living on docker hub from when we were incubating [1] [2]. These repositories contain hop versions 1.0 and older. What should we do with them? Leave them be as they could still be used by someone or do a request to the infra team

Re: [DISCUSS] Old dockerhub images

2023-12-20 Thread hans . van . akelyen
Hi All, As there are no objections to this I am moving forward in the removal of the old Docker Hub location. Cheers, Hans On 12 Dec 2023 at 16:45 +0100, hans.van.akel...@gmail.com, wrote: > Hi all, > > Someone pointed out that we have 2 old repositories still living on docker > hub from when w

Re: Open Sans (300) Light default on website

2024-01-10 Thread hans . van . akelyen
Hi Thad, Thank you for bringing this up we highly appreciate the feedback. Except for some general guidelines on content and links that must be included in the website there are no mandatory guidelines [1] [2]. It seems a page has been added that we have not included yet but that is another pro

Re: [DISCUSS] Old dockerhub images

2024-01-10 Thread hans . van . akelyen
Hi All, As discussed here we concluded that the old docker images were no longer needed. I have created an infra ticket on JIRA (INFRA-25365) and it has already been closed. The old dockerhub locations should no longer be visible for end users. Cheers, Hans On 27 Dec 2023 at 23:16 +0100, Sergio

Apache Hop translation service working again

2024-02-19 Thread hans . van . akelyen
Hi All, Just a small FYI that our translation service is up and running again. We did lose all log-in information so you will have to create a new account. The service can be found in its original location [1] Cheers, Hans [1] https://translate.project-hop.org/

[DISCUSS] 2.8 release

2024-02-19 Thread hans . van . akelyen
Hi Hoppers, The original release date of 2.8 was planned for the end of Januari. I would like to propose to put the new cut-off date at Friday (23 feb) then I will prepare the release in the weekend for a vote. Things we can still do before the end of the week: • Apache Beam upgrade (to version

Re: Apache Hop Fundamentals Course is now available in Spanish 🇪🇸

2024-02-27 Thread hans . van . akelyen
Thank you for another great contribution to this community Adalennis! Cheers, Hans On 27 Feb 2024 at 11:23 +0100, Adalennis Buchillón Soris , wrote: > Hello from Datavin3 > > Apache Hop Fundamentals is now available in Spanish. > > This course is: > 🆓 Free > 🎓 An introduction to Apache Hop > 🚀 Pa

Re: Shell execute

2024-03-13 Thread hans . van . akelyen
Hi David, Extra logging lines can be added by using "log", for the Debug level is log.logDebug(); Cheers, Hans On 13 Mar 2024 at 00:12 +0100, David McNeill , wrote: > So, probably more of a developer question... > > How do I add a log debug just prior to this > > execProcess(cmdArray.toArray(new

CVE-2024-24683: Apache Hop Engine: ID isn't escaped when generating HTML

2024-03-18 Thread Hans Van Akelyen
Severity: low Affected versions: - Apache Hop Engine before 2.8.0 Description: Improper Input Validation vulnerability in Apache Hop Engine.This issue affects Apache Hop Engine: before 2.8.0. Users are recommended to upgrade to version 2.8.0, which fixes the issue. When Hop Server writes lin

Re: "Get Mail" from Gmail.

2024-04-05 Thread hans . van . akelyen
Hi Rob, When using XOAUTH2 the transform does not expect a password in the password field but an OAuth 2 token [1]. Obtaining this token from Google is a bit tricky. We could create GUI elements to fetch that token and this might be a solution for the GUI, but it won't be a real deployment solu

Re: Does Hop have something like PENTAHO_JAVA_HOME?

2024-04-24 Thread Hans Van Akelyen
Yes we do, and its named HOP_JAVA_HOME Cheers, Hans On 2024/04/23 13:54:21 pod...@gmx.com wrote: > One more question about Hop JAVA location. > > Does Hop have something like PENTAHO_JAVA_HOME? > > > > I have software on my PC that use different versions of Java. > > > > Thanks for help >

Re: Connecting to an Oracle Database in a Docker container

2024-05-22 Thread Hans Van Akelyen
Hi Davide, That might be the root cause of this issue. The container creates a new project using the provided variables [1]. We currently do not have the options to create/specify a parent project. Kr, Hans [1] https://github.com/apache/hop/blob/main/docker/resources/load-and-execute.sh On 22

Upgrade to java 17

2024-05-26 Thread Hans Van Akelyen
Hi All, This is a message to inform everyone that the builds and codebase has been moved to build with Java 17. It seems some things are no longer working as expected, the main one is the "Scripts" transform. I have also removed Powermock from our codebase as it was no longer supported. I notic

[HELP WANTED] Start testing Hop 2.10 builds

2024-06-11 Thread Hans Van Akelyen
Hi All, As some of you already know, our next release will be a big one. Our goal will be to start releasing Apache Hop using Java 17 in stead of Java 11. This upgrade is not without risks of some things breaking. To avoid this I am calling on all of you to help us. Currently our builds are wor

Re: Weblate - this project is live?

2024-06-20 Thread Hans Van Akelyen
Seems to be working on my end. Does this link not work for you? https://translate.project-hop.org/accounts/register/ Cheers, Hans On Thu, 20 Jun 2024 at 21:04, wrote: > Hi, > > I cannot login to https://translate.project-hop.org/accounts/login/ > I cannot reset password: 'The password can not b

Re: Weblate - this project is live?

2024-06-22 Thread Hans Van Akelyen
;m > transfered to > https://hop.apache.org/accounts/complete/email/?verification_code=xxx > page where I get message: > > Not Found > > The requested URL was not found on this server. > > > *Sent:* Thursday, June 20, 2024 at 10:46 PM > *From:* "Hans Van Akelye

Re: Uploading files to S3 bucket

2024-07-23 Thread Hans Van Akelyen
Hi Rob, All our transforms (except for CSV input) can read an write directly to S3. It will use the default configured AWS account (there is no support for profiles). Multi-account support is on the roadmap. More information can be found under the Virtual File System section in the documentatio

Re: Sybase JDBC Connection ok, but query doesn't work

2024-08-08 Thread Hans Van Akelyen
Hi Danilo, It could be that the older JTDS drivers don't really work that well anymore with newer Sybase versions. I would try using a generic database connection with one of their newer drivers. You will have to figure out what the driver class is but normally this should be well documented.

Re: join slack

2024-11-05 Thread Hans Van Akelyen
Invite should be arriving soon On 5 Nov 2024 at 16:03 +0100, Enrico Maria Carmona , wrote: > Hi, > I'd like to join the #hop channel @ the-asf.slack.com with the > enrico.carm...@asst-brianza.it email. > > Thank you! > > Enrico >

Re: buen dia solicito apoyo para unirse al canal invitación para unirse al canal #hop @the-asf.slack.com el correo que deseo unir es mrbeta...@hotmail.com quedo atento Saludos

2024-11-05 Thread Hans Van Akelyen
Invite should be arriving soon On 4 Nov 2024 at 17:35 +0100, Merquiades Rufino Betanco , wrote: > > > > >

Re: slack invite

2024-11-28 Thread Hans Van Akelyen
Done, invite should arrive shortly Cheers, Hans On 28 Nov 2024 at 16:40 +0100, Nicolas Dupont , wrote: > Hi, > > Can you send me an invite for the slack channel please ? > > thx >

Re: Please invite me to slack!

2024-11-28 Thread Hans Van Akelyen
So no one else does it, this has been done. Cheers, Hans On 28 Nov 2024 at 10:51 +0100, dmai...@gmail.com , wrote: > Tx > > Diego

  1   2   >