Re: Nifi 1.0.0 breaking changes?

2016-09-01 Thread Joe Witt
Devin, You might find this helpful. We try to maintain it for each release. https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance Please let us know if you run into something you find wasn't here. We want to make it as accurate as possible but across more than 500 JIRAs I am a

Re: PostHTTP Penalize file on HTTP 5xx response

2016-09-01 Thread McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote)
Hmmm, given this I wonder penalizing the flow file is going to help. I’d like maintain delivery order as best I can. If the web service is having intermittent problems some files might be penalized but others, added to the flow later, don’t get penalized and are sent out of order. It might be

Re: Back Pressure Indicators?

2016-09-01 Thread Bryan Bende
Peter, This is something that has come up before, but hasn't been addressed yet. https://issues.apache.org/jira/browse/NIFI-766 -Bryan On Thu, Sep 1, 2016 at 10:25 AM, Peter Wicks (pwicks) wrote: > One of the things I've been noticing that doesn't seem to have any status > indicator for is ba

Re: Nifi 1.0.0 breaking changes?

2016-09-01 Thread Bryan Bende
Devin, I know there are JIRAs in 1.0.0 that are marked with a "Migration" label, those are ones that might impact upgrading from 0.x. I'm not sure there is something that explains every change to the API. Is there a particular area you are interested in? are you talking about custom processors de

Re: Apache NiFi - 3 tier Architecture

2016-09-01 Thread Andrew Grande
Nishad, You can split NiFi clusters at will and connect them securely. Take a look at Remote Process Group and an underlying site-to-site protocol. In fact, linking data centers and layers is one of the mainstream NiFi use cases. Andrew On Thu, Sep 1, 2016, 1:05 PM Nishad Hameed wrote: > Hi A

Re: Apache NiFi - 3 tier Architecture

2016-09-01 Thread Bryan Bende
Hello, I'm not sure if this answers your question, but Apache NiFi is a single application that can only be run the way it is packaged in the distribution. It happens to run an embedded web server which is Jetty, but from the user perspective you just launch a Java application. There aren't diffe

Re: Nifi Cross Account Download With A Profile Flag

2016-09-01 Thread James Wing
Keren, I'm certain cross-account access can work in 0.6.0, I've done it. The timeout error calling sts:AssumeRole is not the same error you started with, right? Earlier, the error was 403 "Access Denied", which would have been farther in the auth process. Any idea what changed? Are you using t

Re: Nifi Cross Account Download With A Profile Flag

2016-09-01 Thread Tseytlin, Keren
Hey James, No problem. I’ve tried running a bunch of different ways to do it manually to try and avoid the CLI profile flag. But it seems like that is the only it works locally (without Nifi). My cross account role doesn’t require an External ID, and that feature doesn’t exist in 0.6.0 either. An

Apache NiFi - 3 tier Architecture

2016-09-01 Thread Nishad Hameed
Hi All, We are planning to use the "Apache NiFi" for our data flow and scheduling service. But as we have 3 tier architecture, we need to see if we can split NiFi. Like 1. Web/Presentation layer 2. Application layer 3. Data Layer Is there any project already implemented th

Minifi question

2016-09-01 Thread Gunjan Dave
Hello everyone, minifi as i understand has specific use case of small footprint data collection agent. Hiwever, today i was trying it out and saw an interesting possibility, would need NiFi teams opinion on it. Can minifi be also modified to work as test harness agent? Like can currrently processe

Re: Nifi Cross Account Download With A Profile Flag

2016-09-01 Thread James Wing
Keren, I'm sorry if my advice is a bit confusing, there have been some changes to AWS credentials over the last few versions. NiFi 0.6.0 does not have the option to use a CLI profile in AWSCredentialsProviderControllerService, I think that was introduced in 0.7.0. Would it be possible to share s

Re: Nifi Cross Account Download With A Profile Flag

2016-09-01 Thread Tseytlin, Keren
Thanks for your responses! @James - we are on version 0.6.0. Using Hortonworks Data Flow 1.2.0.0. I¹ve set up debugging, and it shows me that it¹s trying to connect, but it times out on connecting. It would be awesome if it would also return the account ID of the credentials it is trying to use.

Back Pressure Indicators?

2016-09-01 Thread Peter Wicks (pwicks)
One of the things I've been noticing that doesn't seem to have any status indicator for is back pressure. When back pressure kicks in I usually notice because I start thinking that NiFi is being buggy or is broken, "Why aren't these Flow Files moving through the queue?". Then I'll notice the G