GitHub user jamescao opened a pull request:
https://github.com/apache/flink/pull/1064
[FLINK1919] add HCatOutputFormat
[FLINK1919]
Add `HCatOutputFormat` for Tuple data types for java and scala api also fix
a bug for the scala api's `HCatInputFormat` for hive complex types.
Java api includes check if the schema of the HCatalog table and the Flink
tuples match if the user provides a `TypeInformation` in the constructor. For
data types other than tuples, the OutputFormat requires a preceding Map
function that converts to `HCatRecords`
scala api includes check if the schema of the HCatalog table and the Scala
tuples match. For data types other than scala `Tuple`, the OutputFormat
requires a preceding Map function that converts to `HCatRecords` scala api
require suser to import `org.apache.flink.api.scala._` to allow the type be
captured by the scala macro.
The Hcatalog jar in maven central is compiled using hadoop1, which is not
compatible with hive jars for testing, so a cloudera hcatalog jar is pulled
into the pom for testing purpose. It can be removed if not required.
java `List` and `Map` can not be cast to scala `List` and `Map`,`
JavaConverters` is used to fix a bug in HcatInputFormat scala api
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jamescao/flink hcatbranch
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1064.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1064
----
commit aa149f4d2c74db9c7c1d8f82dbe923e3930713b1
Author: James Cao <james...@outlook.com>
Date: 2015-08-06T01:52:45Z
add HCatOutputFormat
java api and scala api
fix scala HCatInputFormat bug for complex type
pull in cloudera Hcatalog jar for end to end test
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
