Josh Meyer created NIFI-3197:
--------------------------------
Summary: Unable to use Snappy Compression Codec on PutHDFS
Key: NIFI-3197
URL: https://issues.apache.org/jira/browse/NIFI-3197
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.1.0, 1.0.0
Environment: centos 7.2
NiFi 1.1.0 or NiFi 1.0.0
HDP 2.4.3.0
Reporter: Josh Meyer
When setting the 'Compression code' to SNAPPY on PutHDFS NiFi is unable to
compress and push data to HDFS. Attached is the sample workflow that is having
trouble. Below are the errors for NiFi 1.0.0 and the errors for NiFi 1.1.0.
Same configuration, but the error is a bit different.
I have tried setting both LD_LIBRARY_PATH as an environment variable, and then
I tried setting java.library.path in the bootstrap.conf.
NiFi 1.1.0 nifi-app.log error message:
{code}
2016-12-14 15:13:56,563 ERROR [Timer-Driven Process Thread-6]
o.apache.nifi.processors.hadoop.PutHDFS
PutHDFS[id=fdd52005-0158-1000-ac0f-2a87ed12a1e7] Failed to write to HDFS due to
java.lang.RuntimeException: native snappy library not available: this version
of libhadoop was built without snappy support.: java.lang.RuntimeException:
native snappy library not available: this version of libhadoop was built
without snappy support.
2016-12-14 15:13:56,644 ERROR [Timer-Driven Process Thread-6]
o.apache.nifi.processors.hadoop.PutHDFS
java.lang.RuntimeException: native snappy library not available: this version
of libhadoop was built without snappy support.
at
org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65)
~[hadoop-common-2.7.3.jar:na]
at
org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134)
~[hadoop-common-2.7.3.jar:na]
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
~[hadoop-common-2.7.3.jar:na]
at
org.apache.hadoop.io.compress.CompressionCodec$Util.createOutputStreamWithCodecPool(CompressionCodec.java:131)
~[hadoop-common-2.7.3.jar:na]
at
org.apache.hadoop.io.compress.SnappyCodec.createOutputStream(SnappyCodec.java:100)
~[hadoop-common-2.7.3.jar:na]
at
org.apache.nifi.processors.hadoop.PutHDFS$1$1.process(PutHDFS.java:313)
~[nifi-hdfs-processors-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2082)
~[na:na]
at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2053)
~[na:na]
at org.apache.nifi.processors.hadoop.PutHDFS$1.run(PutHDFS.java:300)
~[nifi-hdfs-processors-1.1.0.jar:1.1.0]
at java.security.AccessController.doPrivileged(Native Method)
[na:1.8.0_111-debug]
at javax.security.auth.Subject.doAs(Subject.java:360)
[na:1.8.0_111-debug]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1678)
[hadoop-common-2.7.3.jar:na]
at
org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:230)
[nifi-hdfs-processors-1.1.0.jar:1.1.0]
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
[nifi-api-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099)
[nifi-framework-core-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
[nifi-framework-core-1.1.0.jar:1.1.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_111-debug]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_111-debug]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_111-debug]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_111-debug]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_111-debug]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_111-debug]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111-debug]
{code}
NiFi 1.0.0 nifi-bootstrap.log
{code}
2016-12-14 15:32:50,373 INFO [NiFi Bootstrap Command Listener]
org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and listening for
Bootstrap requests on port 40280
2016-12-14 15:43:47,752 INFO [NiFi logging handler] org.apache.nifi.StdOut
FATAL ERROR in native method: JNI received a class argument that is not a class
2016-12-14 15:43:47,758 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect(Native
Method)
2016-12-14 15:43:47,759 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:232)
2016-12-14 15:43:47,759 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:149)
2016-12-14 15:43:47,760 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:142)
2016-12-14 15:43:47,761 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
org.apache.hadoop.io.compress.CompressorStream.close(CompressorStream.java:106)
2016-12-14 15:43:47,761 INFO [NiFi logging handler] org.apache.nifi.StdOut
at org.apache.nifi.processors.hadoop.PutHDFS$1.process(PutHDFS.java:295)
2016-12-14 15:43:47,761 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1880)
2016-12-14 15:43:47,762 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1851)
2016-12-14 15:43:47,762 INFO [NiFi logging handler] org.apache.nifi.StdOut
at org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:276)
2016-12-14 15:43:47,762 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
2016-12-14 15:43:47,762 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064)
2016-12-14 15:43:47,762 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
2016-12-14 15:43:47,762 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
2016-12-14 15:43:47,763 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
2016-12-14 15:43:47,763 INFO [NiFi logging handler] org.apache.nifi.StdOut
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
2016-12-14 15:43:47,763 INFO [NiFi logging handler] org.apache.nifi.StdOut
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
2016-12-14 15:43:47,763 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
2016-12-14 15:43:47,764 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
2016-12-14 15:43:47,764 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2016-12-14 15:43:47,764 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2016-12-14 15:43:47,764 INFO [NiFi logging handler] org.apache.nifi.StdOut
at java.lang.Thread.run(Thread.java:745)
2016-12-14 15:43:47,764 INFO [NiFi logging handler] org.apache.nifi.StdOut
Current thread is 139782124459776
2016-12-14 15:43:47,764 INFO [NiFi logging handler] org.apache.nifi.StdOut
Dumping core ...
2016-12-14 15:43:48,525 WARN [main] org.apache.nifi.bootstrap.RunNiFi Apache
NiFi appears to have died. Restarting...
2016-12-14 15:43:48,545 INFO [main] org.apache.nifi.bootstrap.RunNiFi
Successfully started Apache NiFi with PID 27342
2016-12-14 15:43:52,874 INFO [NiFi Bootstrap Command Listener]
org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and listening for
Bootstrap requests on port 43961
2016-12-14 15:44:09,020 INFO [main] o.a.n.b.NotificationServiceManager
Successfully loaded the following 0 services: []
2016-12-14 15:44:09,053 INFO [main] org.apache.nifi.bootstrap.RunNiFi
Registered no Notification Services for Notification Type NIFI_STARTED
2016-12-14 15:44:09,054 INFO [main] org.apache.nifi.bootstrap.RunNiFi
Registered no Notification Services for Notification Type NIFI_STOPPED
2016-12-14 15:44:09,054 INFO [main] org.apache.nifi.bootstrap.RunNiFi
Registered no Notification Services for Notification Type NIFI_DIED
2016-12-14 15:44:09,231 INFO [main] org.apache.nifi.bootstrap.Command Apache
NiFi has accepted the Shutdown Command and is shutting down now
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
