Re: Identifying issue with simple Apache Beam setup

2023-09-27 Thread Logan G Engstrom
Thank you! On Sep 27, 2023, at 11:05 AM, Byron Ellis via user mailto:user@beam.apache.org>> wrote: Hi Logan, I suspect your issue is that you're using "Create" to hold all of your data. That particular transform's data is typically stored within the pipeline definition itself, which is probab

Re: Identifying issue with simple Apache Beam setup

2023-09-27 Thread Byron Ellis via user
Hi Logan, I suspect your issue is that you're using "Create" to hold all of your data. That particular transform's data is typically stored within the pipeline definition itself, which is probably why you're seeing the protobuf max errors. It's really meant for things more like command line parame

Identifying issue with simple Apache Beam setup

2023-09-26 Thread Logan G Engstrom
Hi, I am new to learning Apache Beam and have had trouble with performance; despite searching around I have not been able to identify a solution. My issue is that even though I am using parallelizable operations + converting to PCollections, Beam is incredibly slow. Furthermore, when running o