Re: Cross-Language pipeline fails with PROCESS SDK Harness

Great. Thanks. On Mon, Jun 1, 2020 at 9:14 AM Alexey Romanenko wrote: > Yes, I tested it with the cross-language transform (Java pipeline with > Python external transform). > > On 1 Jun 2020, at 17:49, Chamikara Jayalath wrote: > > To clarify, is the error resolved with the cross-language trans

Re: Cross-Language pipeline fails with PROCESS SDK Harness

Yes, I tested it with the cross-language transform (Java pipeline with Python external transform). > On 1 Jun 2020, at 17:49, Chamikara Jayalath wrote: > > To clarify, is the error resolved with the cross-language transform as well ? > If not please file a Jira. > > On Mon, Jun 1, 2020 at 8:2

Re: Cross-Language pipeline fails with PROCESS SDK Harness

To clarify, is the error resolved with the cross-language transform as well ? If not please file a Jira. On Mon, Jun 1, 2020 at 8:24 AM Kyle Weaver wrote: > > It would be useful to print out such errors with Error level log, I > think. > > I agree, using environment_type=PROCESS is difficult eno

Re: Cross-Language pipeline fails with PROCESS SDK Harness

> It would be useful to print out such errors with Error level log, I think. I agree, using environment_type=PROCESS is difficult enough without hiding the logs by default. I re-opened the issue. On Mon, Jun 1, 2020 at 11:01 AM Alexey Romanenko wrote: > Thanks! It was an issue with a setting *v

Re: Cross-Language pipeline fails with PROCESS SDK Harness

Thanks! It was an issue with a setting virtualenv for a worker console where it should be running. It would be useful to print out such errors with Error level log, I think. > On 29 May 2020, at 18:55, Kyle Weaver wrote: > > That's probably a problem with your worker. You'll need to get additi

Re: Cross-Language pipeline fails with PROCESS SDK Harness

That's probably a problem with your worker. You'll need to get additional logs to debug (see https://jira.apache.org/jira/browse/BEAM-8278) On Fri, May 29, 2020 at 12:48 PM Alexey Romanenko wrote: > Many thanks! It helped to avoid the error. I saw this option in the xlang > tests before but I di

Re: Cross-Language pipeline fails with PROCESS SDK Harness

Many thanks! It helped to avoid the error. I saw this option in the xlang tests before but I didn’t add it since I was confused because of the name =) Also, I think we need to added “—sdk_location=container” for Expansion Service Finally, I've managed to only Java and xlang pipeline (with Python

Re: Cross-Language pipeline fails with PROCESS SDK Harness

Alexey, can you try adding --experiments=beam_fn_api to your pipeline options? We add the option automatically in Python [1] but we don't in Java. I filed BEAM-10151 [2] to document this workflow. Alexey, perhaps you can help with that. [1] https://github.com/apache/beam/blob/a5b2046b10bebc59c5bd

Re: Cross-Language pipeline fails with PROCESS SDK Harness

Yes, I did run only Java pipeline with Portable Runner and there is the same error. Also, I did the same (without cross-language component) against Beam 2.19 and 2.20. It works fine against Beam 2.19 (as expected, since I tested it already before) and fails with kind the same error against Bea

Re: Cross-Language pipeline fails with PROCESS SDK Harness

Can you try removing the cross-language component(s) from the pipeline and see if it still has the same error? On Thu, May 28, 2020 at 4:15 PM Alexey Romanenko wrote: > For testing purposes, it’s just “Create.of(“Name1”, “Name2”, ...)" > > On 28 May 2020, at 19:29, Kyle Weaver wrote: > > What s

Re: Cross-Language pipeline fails with PROCESS SDK Harness

For testing purposes, it’s just “Create.of(“Name1”, “Name2”, ...)" > On 28 May 2020, at 19:29, Kyle Weaver wrote: > > What source are you using? > > On Thu, May 28, 2020 at 1:24 PM Alexey Romanenko > wrote: > Hello, > > I’m trying to run a Cross-Language pipel

Re: Cross-Language pipeline fails with PROCESS SDK Harness

I haven't had any changes since #11670 in this space so nothing is missing from the release. Also, the GreedyPipelineFuser has not been updated to support XLang as it has some baked-in assumptions around flatten[1] and likely other issues. It has worked because the simple examples we have run have

Re: Cross-Language pipeline fails with PROCESS SDK Harness

This might have to do with https://github.com/apache/beam/pull/11670. +Lukasz Cwik was there a subsequent fix that was not included in the release ? On Thu, May 28, 2020 at 10:29 AM Kyle Weaver wrote: > What source are you using? > > On Thu, May 28, 2020 at 1:24 PM Alexey Romanenko > wrote: >

Re: Cross-Language pipeline fails with PROCESS SDK Harness

What source are you using? On Thu, May 28, 2020 at 1:24 PM Alexey Romanenko wrote: > Hello, > > I’m trying to run a Cross-Language pipeline (Beam 2.21, Java pipeline with > an external Python transform) with a PROCESS SDK Harness and Spark Portable > Runner but it fails. > To do that I have a ru

Cross-Language pipeline fails with PROCESS SDK Harness

Hello, I’m trying to run a Cross-Language pipeline (Beam 2.21, Java pipeline with an external Python transform) with a PROCESS SDK Harness and Spark Portable Runner but it fails. To do that I have a running Spark Runner Job Server (Spark local) and standalone Expansion Service (Python) which co