NIFI RabbitMQ Processor

I have started a NIFI RabbitMQ module: https://github.com/MDL/nifi-rabbitmq-bundle What works: - Direct exchanges. Light workloads have been tested. A few million items have been pushed through the system without any problems. I have also run it for a few hours at a time without problem. W

Re: need help with Async code

thanks for clarifying on getControllerServiceIdentifiers API. I have an other question: If I have a processor that is designed to have on side effect on FlowFile, what is the best/clean way to read content of the flowfile? e.g., my processor’s only job is to log content of FlowFile, is there a me

Re: need help with Async code

Sumo, I did some digging around on your Github Repo and see that you've migrated your ControllerService lookup to your @OnScheduled method, making use of the ProcessContext. This approach is certainly more preferred in terms of allowing configuration of the Processor than the prior method you out

Re: PutHDFS Configuration Issue

Hey Domenic - Since it looks like you're going to be using Kerberos, you should probably be aware of NIFI-997 , which I recently posted a patch to fix. Please let us know if any other issues come up. Ricky On Wed, Sep 30, 2015 at 2:01 PM, Bryan Ben

Re: PutHDFS Configuration Issue

Glad that first issue was resolved! I am by no means a kerberos expert, but having set this up once before, the setup should be something like the following: nifi.kerberos.krb5.file=/etc/krb5.conf (or wherever your conf file is) This is the file that would have your realms defined. Then on PutH

Re: [jira] [Commented] (NIFI-994) Processor to tail files

Joe can you please reply as a comment in the ticket instead of an email thread? It makes it a lot easier to keep track of the discussion. Thanks, Joe- - - - - - Joseph Percivall linkedin.com/in/Percivall e: joeperciv...@yahoo.com On Wednesday, September 30, 2015 1:27 PM, Joe Skora wrote: I

Re: PutHDFS Configuration Issue

Thank you so much for the quick reply! Restarting NiFi resolved the issue! I had not thought of that. However, a new issue was raised that I am currently working through: "Can't find Kerberos realm". Do you have any idea where this should be set and what value it is looking for? I really apprecia

Re: PutHDFS Configuration Issue

Sorry, hit send too soon, was going to say... Also, there is a second check in the validation to make sure the keytab file is readable, but I don't think that is the issue here because it would produce a more specific error message. -Bryan On Wed, Sep 30, 2015 at 1:24 PM, Bryan Bende wrote: >

Re: [jira] [Commented] (NIFI-994) Processor to tail files

I think we are on the same page, but I left out some details. The key is that the processor always starts at the beginning when it finds a file but discards content it thinks was previously committed downstream. One approach could be storing a checksum of processed content with the other state wh

Re: PutHDFS Configuration Issue

Hi Domenic, It sounds like you are on the right path... just to confirm, did you restart NiFi after setting nifi.kerberos.krb5.file in nifi.properties? It will only pick up changes to nifi.properties on a restart. Also, On Wed, Sep 30, 2015 at 12:38 PM, DomenicPuzio wrote: > Hello, > > I am t

PutHDFS Configuration Issue

Hello, I am trying to set up the PutHDFS processor on NiFi, and I am running into an issue. I have the Hadoop Configuration Resources set and pointing to my core-site.xml and hdfs-site.xml, and I have a Kerberos Principal and Keytab file. However, I am getting the error below. 'Kerberos Principle

Re: [jira] [Commented] (NIFI-994) Processor to tail files

Joe, The problem with "tail -F" is that if NiFi is restarted and then we do essentially "tail -F" we may have missed a lot of data that was written to the log file while NiFi was down. The idea behind this Processor is to be able to recover that data, even if it was written to a log file (or an

Re: [jira] [Commented] (NIFI-994) Processor to tail files

For a NiFi processor, I think the "tail -F" makes more sense. As opposed to the normal behavior that follows existing file descriptors, "tail -F" follows on filename (or pattern) so it tracks the current instance of a file, letting it handle new files during the run, log rotations, etc.. I defini

Re: nifi Cluster setup issue

Chakri, Make sure that the property nifi.web.http.host is set. It is trying to connect to 3002 because you have told it to connect on that port using this property nifi.cluster.node.protocol.port. Its strange that the debug statement has localhost for the api address. Thats why I say to make sure

Re: [jira] [Commented] (NIFI-994) Processor to tail files

Maybe this processor could copy the behavior of the GNU Coreutils "tail" command, aka Linux *tail*? (source is here ) If it can, it appears to use the Linux iNotify service

Re: Development environment for running nifi-web-ui

Dave, The nifi-web-ui artifact is bundled in the nifi-framework-nar. You could rebuild those two modules and then drop the nifi-framework-nar artifact into an existing Apache NiFi installation. Apache NiFi contains an embedded web server so there's no need to try to deploy in another. Matt On We

Re: What sort of major release support plan do we want to provide?

Assuming major upgrades will sometimes break compatibility, I think we need to plan on providing general bug fixes, not just security or critical fixes, to at least the last major release. I like the idea of those patch releases becoming less frequent as the age of the release increases. I don't th

Development environment for running nifi-web-ui

How do you guys set up your environment to develop nifi-web-ui without rebuilding the entire project each time? If I run only the nifi-web-ui in Tomcat I (obviously) get dependency issues. Many Thanks, Dave

[GitHub] nifi pull request: NIFI-988: PutDistributedMapCache processor

Github user markap14 commented on the pull request: https://github.com/apache/nifi/pull/92#issuecomment-144378199 @joemeszaros the concern that I have with the notion of the ExtendedDistributedMapCacheClient is that once that is released, it will have the same caveats as the Distribut

[GitHub] nifi pull request: NIFI-988: PutDistributedMapCache processor

Github user joemeszaros commented on the pull request: https://github.com/apache/nifi/pull/92#issuecomment-144332289 @markap14 Did you get a proper answer for your cache related questions? If you do not have any concern with this new processor, it would be reasonable to implement the