Michael reminded me that although we don't support direct manipulation
over Parquet metadata, you can still save/query metadata to/from Parquet
via DataFrame per-column metadata. For example:
import sqlContext.implicits._
import org.apache.spark.sql.types.MetadataBuilder
val path = "file:///tmp/parquet/meta"
// Saving metadata
val meta = new MetadataBuilder().putString("appVersion", "1.0.2").build()
sqlContext.range(10).select($"id".as("id",
meta)).coalesce(1).write.mode("overwrite").parquet(path)
// Querying metadata
sqlContext.read.parquet(path).schema("id").metadata.getString("appVersion")
The metadata is saved together with Spark SQL schema as a JSON string.
For example, the above code generates the following Parquet metadata
(inspected with parquet-meta):
file:
file:/private/tmp/parquet/meta/part-r-00000-77cb2237-e6a8-4cb6-a452-ae205ba7b660.gz.parquet
creator: parquet-mr version 1.6.0
extra: org.apache.spark.sql.parquet.row.metadata =
{"type":"struct","fields":[{"name":"id","type":"long","nullable":true,*"metadata":{"appVersion":"1.0.2"}*}]}
Cheng
On 9/22/15 9:37 AM, Cheng Lian wrote:
I see, this makes sense. We should probably add this in Spark SQL.
However, there's one corner case to note about user-defined Parquet
metadata. When committing a write job, ParquetOutputCommitter writes
Parquet summary files (_metadata and _common_metadata), and
user-defined key-value metadata written in all Parquet part-files get
merged here. The problem is that, if a single key is associated with
multiple values, Parquet doesn't know how to reconcile this situation,
and simply gives up writing summary files. This can be particular
annoying for appending. In general, users should avoid storing
"unstable" values like timestamps as Parquet metadata.
Cheng
On 9/22/15 1:58 AM, Borisa Zivkovic wrote:
thanks for answer.
I need this in order to be able to track schema metadata.
basically when I create parquet files from Spark I want to be able to
"tag" them in some way (giving the schema appropriate name or
attaching some key/values) and then it is fairly easy to get basic
metadata about parquet files when processing and discovering those
later on.
On Mon, 21 Sep 2015 at 18:17 Cheng Lian <lian.cs....@gmail.com> wrote:
Currently Spark SQL doesn't support customizing schema name and
metadata. May I know why these two matters in your use case? Some
Parquet data models, like parquet-avro, do support it, while some
others
don't (e.g. parquet-hive).
Cheng
On 9/21/15 7:13 AM, Borisa Zivkovic wrote:
> Hi,
>
> I am trying to figure out how to write parquet metadata when
> persisting DataFrames to parquet using Spark (1.4.1)
>
> I could not find a way to change schema name (which seems to be
> hardcoded to root) and also how to add data to key/value
metadata in
> parquet footer.
>
> org.apache.parquet.hadoop.metadata.FileMetaData#getKeyValueMetaData
>
> org.apache.parquet.schema.Type#getName
>
> thanks
>
>
