[
https://issues.apache.org/jira/browse/NIFI-10309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard resolved NIFI-10309.
-----------------------------------
Resolution: Feedback Received
Apache NiFi 1.x is no longer maintained and no new release is planned on the
1.x release line. Marking as resolved as part of a cleanup operation. Please
open a new one with an updated description if this is still relevant for NiFi
2.x.
> LZ4 and SNAPPY codecs doesn't work for "getter" processors for HDFS
> -------------------------------------------------------------------
>
> Key: NIFI-10309
> URL: https://issues.apache.org/jira/browse/NIFI-10309
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 1.16.3
> Environment: CentOS Linux release 8.3.2011
> Hadoop 3.3.3
> Nifi 1.16.3
> Reporter: Katerina Navara
> Priority: Major
>
> # create in HDFS some file with compression (lz4 or snappy)
> # Create GetHDFS Or FetchHDFS processor
> # set prop “{*}Compression codec{*}” to LZ4 or SNAPPY.
> # start processor
> Actual result:
> snappy case:
> {code:java}
> "ERROR [Timer-Driven Process Thread-1]
> o.apache.nifi.processors.hadoop.GetHDFS
> GetHDFS[id=0182100b-a13c-1f5a-f526-933b6e2767a1] Error retrieving file
> hdfs://localhost:9000/test/snappy/file.snappy from HDFS due to
> org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import
> data from org.apache.hadoop.io.compress.BlockDecompressorStream@5a5668ea for
> StandardFlowFileRecord[uuid=fe485e3e-6594-49b5-b46d-13fabbe734dd,claim=,offset=0,name=fe485e3e-6594-49b5-b46d-13fabbe734dd,size=0]
> due to java.lang.OutOfMemoryError: Java heap space
> org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import
> data from org.apache.hadoop.io.compress.BlockDecompressorStream@5a5668ea for
> StandardFlowFileRecord[uuid=fe485e3e-6594-49b5-b46d-13fabbe734dd,claim=,offset=0,name=fe485e3e-6594-49b5-b46d-13fabbe734dd,size=0]
> due to java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:3562)
> at
> org.apache.nifi.processors.hadoop.GetHDFS.processBatchOfFiles(GetHDFS.java:382)"
> {code}
> lz4 case:
> {code:java}
> o.apache.nifi.processors.hadoop.GetHDFS
> GetHDFS[id=01821007-a13c-1f5a-e2aa-fbea28643aac] Error retrieving file
> hdfs://localhost:9000/test/lz4/file.lz4 from HDFS due to
> java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Factory
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
