Query regarding support for ROLLUP

Hi All, I was trying to use "GROUP BY WITH ROLLUP" (2.29.0 version) which I saw here - https://beam.apache.org/documentation/dsls/sql/calcite/query-syntax/# "select warehouse, SUM(quantity) as quantity from PCOLLECTION group by ROLLUP(warehouse)")); Warehouse | quantity ---

Potential bug with Kafka (or other external) IO in Python Portable Runner

Hey, I'm trying to run a pipeline with a Kafka Source, using an EXTERNAL environment. However, when the pipeline is run, the error below is thrown, which implies that for some reason the external environment pipeline options didn't get in. When replacing the Kafka Source with an S3 source (for exam

Re: Potential bug with Kafka (or other external) IO in Python Portable Runner

Is it possible that you don't have the "docker" command available in your system ? On Tue, May 4, 2021 at 10:28 AM Nir Gazit wrote: > Hey, > I'm trying to run a pipeline with a Kafka Source, using an EXTERNAL > environment. However, when the pipeline is run, the error below is thrown, > which im

Re: Potential bug with Kafka (or other external) IO in Python Portable Runner

Yes that’s on purpose. I’m running in Kubernetes which makes it hard to install docker on the pods so I don’t want to use the docker environment. That’s why I specified EXTERNAL environment in PipelineOptions. However, it seems that it doesn’t get propagated. On Tue, 4 May 2021 at 20:59 Chamikara

Re: Potential bug with Kafka (or other external) IO in Python Portable Runner

Ah, I think you need the DOCKER environment to use cross-language transforms not the EXTERNAL environment (agree that the terminology is confusing). On Tue, May 4, 2021 at 11:04 AM Nir Gazit wrote: > Yes that’s on purpose. I’m running in Kubernetes which makes it hard to > install docker on the

Re: Potential bug with Kafka (or other external) IO in Python Portable Runner

But looking at the code of the exception it seems that it tries to use docker only because it thinks it's in a docker environment, no? S

Re: Potential bug with Kafka (or other external) IO in Python Portable Runner

When you use cross-language Java transforms from Python we use the default environment for Java transforms which always gets set to Docker. https://github.com/apache/beam/blob/7f0d11e65bbcd3e9c565a50aa6a56c0631c4358b/sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/Ex