Re: [Question] Read Parquet Schema from S3 Directory

2023-10-12 Thread Robert Bradshaw via dev
You'll probably need to resolve "s3a:///*.parquet" out into a concrete non-glob filepattern to inspect it this way. Presumably any individual shard will do. match and open from https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileSystems.html may be useful. On Wed, Oct 11, 2

[Question] Read Parquet Schema from S3 Directory

2023-10-11 Thread Ramya Prasad via dev
Hello, I am a developer trying to use Apache Beam in my Java application, and I'm running into an issue with reading multiple Parquet files from a directory in S3. I'm able to successfully run this line of code, where tempPath = "s3:///*.parquet": PCollection records = pipeline.apply("Read parque