Have a look at the default impl, it's fairly simple:
https://github.com/apache/beam/blob/master/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/DefaultS3FileSystemSchemeRegistrar.java
Your registrar has to provide one or two custom configurations using a new
uri scheme
Hi Logan,
I suspect your issue is that you're using "Create" to hold all of your
data. That particular transform's data is typically stored within the
pipeline definition itself, which is probably why you're seeing the
protobuf max errors. It's really meant for things more like command line
parame
Thank you!
On Sep 27, 2023, at 11:05 AM, Byron Ellis via user
mailto:user@beam.apache.org>> wrote:
Hi Logan,
I suspect your issue is that you're using "Create" to hold all of your data.
That particular transform's data is typically stored within the pipeline
definition itself, which is probab
