[
https://issues.apache.org/jira/browse/FLINK-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16171711#comment-16171711
]
ASF GitHub Bot commented on FLINK-5944:
---------------------------------------
GitHub user mlipkovich opened a pull request:
https://github.com/apache/flink/pull/4683
[FLINK-5944] Support reading of Snappy files
## What is the purpose of the change
Support reading of Snappy compressed text files (both Xerial and Hadoop
snappy)
## Brief change log
- *Added InputStreamFactories for Xerial and Hadoop snappy*
- *Added config parameter to control whether Xerial or Hadoop snappy
should be used*
## Verifying this change
- *Manually verified the change by running word count for text files
compressed using different Snappy versions*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? JavaDocs
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mlipkovich/flink FLINK-5944
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4683.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4683
----
commit c4d4016f1e6b44833d24994c97532b4c5243e4d2
Author: Mikhail Lipkovich <mikhail_lipkov...@epam.com>
Date: 2017-09-19T13:34:10Z
[FLINK-5944] Support reading of Snappy files
----
> Flink should support reading Snappy Files
> -----------------------------------------
>
> Key: FLINK-5944
> URL: https://issues.apache.org/jira/browse/FLINK-5944
> Project: Flink
> Issue Type: New Feature
> Components: Batch Connectors and Input/Output Formats
> Reporter: Ilya Ganelin
> Assignee: Mikhail Lipkovich
> Labels: features
>
> Snappy is an extremely performant compression format that's widely used
> offering fast decompression/compression.
> This can be easily implemented by creating a SnappyInflaterInputStreamFactory
> and updating the initDefaultInflateInputStreamFactories in FileInputFormat.
> Flink already includes the Snappy dependency in the project.
> There is a minor gotcha in this. If we wish to use this with Hadoop, then we
> must provide two separate implementations since Hadoop uses a different
> version of the snappy format than Snappy Java (which is the xerial/snappy
> included in Flink).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
