[
https://issues.apache.org/jira/browse/BEAM-12714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476382#comment-17476382
]
Kenneth Knowles commented on BEAM-12714:
----------------------------------------
You are certainly correct about not passing the full metadata:
https://github.com/apache/beam/blob/c345fe14e581e7675c56f9beeeb8917d1c4e7376/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystem.java#L138
Note that there are some issues with setting the fields, which you can read
about at
https://cloud.google.com/storage/docs/transcoding#content-type_vs_content-encoding
GCS does its best to decompress for you, which cannot be disabled and causes
problems. There's a host of issues filed about it: BEAM-1016, BEAM-1874,
BEAM-7411, BEAM-8168,
The TL;DR is that GCS sends the wrong file size for transcoded files. The
headers give the compressed size and then too many bytes come across the wire.
The Java SDK has trial-and-error code to survive this, while the Python SDK
does not. I believe the intention here is to avoid writing files in a way that
will break downstream consumers. That said, I'm open to change here if there is
a good solution. Philosophically, I do believe it setting the metadata in the
best way possible, and I do believe the current status is a compromise, not
ideal.
> FileSystem.create does not pass full Metadata to GCS
> ----------------------------------------------------
>
> Key: BEAM-12714
> URL: https://issues.apache.org/jira/browse/BEAM-12714
> Project: Beam
> Issue Type: Bug
> Components: io-py-gcp
> Affects Versions: 2.31.0
> Reporter: Andreas Bergmeier
> Priority: P3
>
> When I call *FileSystem.create(foo, mime_type="text/plain",
> compression_type=CompressionTypes.BZIP2)* I would expect the following
> metadata set on GCS Blob:
> bq. content_type: text/plain
> bq. content_encoding: application/bzip2
> What we currently get is:
> bq. content_type: application/x-bz2
> Looking at the code it seems to be a problem with GcsIO interface, which does
> not seem to expose *content_type* and *content_encoding* properly.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
