Monica Parpi created AIRFLOW-3541:
-------------------------------------
Summary: Include --useAvroLogicalTypes setting for the Airflow
operator GoogleCloudStorageToBigQueryOperator
Key: AIRFLOW-3541
URL: https://issues.apache.org/jira/browse/AIRFLOW-3541
Project: Apache Airflow
Issue Type: New Feature
Components: operators
Environment: Composer
Reporter: Monica Parpi
Getting an error [1] when loading data from GCS into BigQuery using the
GoogleCloudStorageToBigQueryOperator for a Avro schema containing this column:
{
"name": "date_requested",
"type": "int",
"logicalType": "date",
}
This column contains the number of days since the epoch as indicated here [2].
It is possible to load this data into a DATE column by using the
"--useAvroLogicalTypes" parameter when initiating a load using the CLI [3]
It would be very useful if this setting could be enabled via the Airflow
operator that performs the equivalent data load [4].
Thank you!
[ [1]
https://issuetracker.google.com/35905894|https://issuetracker.google.com/35905894]
[[2]
http://avro.apache.org/docs/current/spec.html#Date|http://avro.apache.org/docs/current/spec.html#Date]
[3]
[https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.useAvroLogicalTypes]
[4]
[https://airflow.apache.org/_modules/airflow/contrib/operators/gcs_to_bq.html#GoogleCloudStorageToBigQueryOperator]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
