Re: Java prefix external pipeline with go sdk

2022-08-02 Thread Jack McCluskey via user
Okay so I have gone back and confirmed that the translation failed error also originates from the direct runner ( https://github.com/apache/beam/blob/8da6363b6fef8da5e73976be2b1277a776c05239/sdks/go/pkg/beam/runners/direct/direct.go#L337) so you should try to execute on your runner of choice. I'm g

Re: Java prefix external pipeline with go sdk

2022-08-02 Thread Sergiusz Rokosz
Hello, Thank you very much for the reply. I will try running the second example with dataflow runner. Here is the full output from the first example: 2022/08/02 12:50:19 Executing pipeline with the direct runner. 2022/08/02 12:50:19 Pipeline: 2022/08/02 12:50:19 Nodes: {1: []uint8/bytes GLO} {2:

Re: Java prefix external pipeline with go sdk

2022-08-01 Thread Jack McCluskey via user
Hey Sergiusz, I'm looking into the cross-language problems you're running into. I can't say with 100% certainty for the first issue, but the problem with the second is that you're running on the Go direct runner, which does not support cross-language transforms. You'll need to start up a more full

Re: Java prefix external pipeline with go sdk

2022-08-01 Thread Sergiusz Rokosz
Dont know if this is connected but when executing a pipeline with cross language source I am getting an error too. For example package main import ( "context" "flag" "reflect" "github.com/apache/beam/sdks/v2/go/pkg/beam" "github.com/apache/beam/sdks/v2/go

Java prefix external pipeline with go sdk

2022-08-01 Thread Sergiusz Rokosz
Hello, I was trying to recreate java prefix pipeline from https://beam.apache.org/documentation/sdks/python-multi-language-pipelines/ but with go sdk. My pipeline code is: package main import ( "context"