Re: CsvToAttributes processor

And according to the IETF RFC 2822 (Email), the reply-to field can hold multiple mailboxes, so we will investigate if we can get the dev@ and users@ lists to reply to the list *and* the sender by default. This might really clog people’s inboxes though, so it needs to be evaluated. Andy LoPresto

Re: CsvToAttributes processor

Hi François, I hope this is what you were looking for. If you do not get the entire thread via this email, you can see the thread in a web view here [1]. [1] https://lists.apache.org/thread.html/ffa390534d35056d3ad8ab5116f25665b73687855214afe95fcf6cab@%3Cdev.nifi.apache.org%3E

Re: Logback stops writing to nifi-app.log after restart

https://issues.apache.org/jira/browse/NIFI-2962 On Thu, Oct 27, 2016 at 5:41 PM, Joe Witt wrote: > Good point. Will file a jira amd address. > > Thanks > Joe > > On Oct 27, 2016 3:59 PM, "Joe Gresock" wrote: > >> > >> Ahh, that would explain it! Maybe we can change the comment in >> logback.xm

Re: nifi is running out of memory

moving dev to bcc Yes I believe the issue here is that FetchS3 doesn't do chunked transfers and so is loading all into memory. I've not verified this in the code yet but it seems quite likely. Krish if you can verify that going with a larger heap gets you in the game can you please file a JIRA.

Re: nifi is running out of memory

Hello, Are you running with all of the default settings? If so you would probably want to try increasing the memory settings in conf/bootstrap.conf. They default to 512mb, you may want to try bumping it up to 1024mb. -Bryan On Thu, Oct 27, 2016 at 5:46 PM, Gop Krr wrote: > Hi All, > > I have

Please reply !!Clarification on using regular expression and pass arguments to the executestream command

Hi Dev community I have the below use-case which I am trying to achieve it through NIFI processor. I have a regular expression format for a file which will be picking up through "LIST SFTP processor " with File filter regex : "^(ACRS)[.](COF)[.](FRAUD)[.](PROD)[.](.*)[.](CCOD.+)" Based on th

Re: Logback stops writing to nifi-app.log after restart

Good point. Will file a jira amd address. Thanks Joe On Oct 27, 2016 3:59 PM, "Joe Gresock" wrote: > > Ahh, that would explain it! Maybe we can change the comment in logback.xml > < https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resource

Re: Logback stops writing to nifi-app.log after restart

Ahh, that would explain it! Maybe we can change the comment in logback.xml above this element so it's more clear: 30 On Thu, Oct 27, 2016 at 2:14

Re: is it possible to perform DML statements on data from different database products that has been loaded into NiFi

Raghav, This is not currently possible inside NiFi. However, you could use Presto [1] to set up connections to each database, then you could create a DBCPConnectionPool in NiFi to connect to Presto, and issue the JOIN query with ExecuteSQL. There is an example of joins across databases here [2].

Re: Logback stops writing to nifi-app.log after restart

Joe, I believe that maxHistory is not 'how many files' to keep but 'how long to keep them for' where the length of time you specify is a function of the rollover interval. So, if you have hourly rollover as you showed previously and if you have a value of '10' it will be trying to keep around up

Re: 0.7.1 download items

Thanks Joe - looks good to me. On Thu, Oct 27, 2016 at 9:34 AM, Joe Skora wrote: > Done. > > I double checked the people, mailing list, and download updates. > > Let me know if anything doesn't look right. > > On Thu, Oct 27, 2016 at 8:54 AM, Joe Skora wrote: > >> No problem. I will do that rig

is it possible to perform DML statements on data from different database products that has been loaded into NiFi

Hi, So I have this use case where I want to get tables from different data sources into NiFi and perform join operation between these tables and get the results using the select query. For example, select query on one table from teradata, select query on one from DB2, etc. and then be able to join

Re: Replace Text for updating the flow file content

Arsalan, What version of NiFi are you using? I ran with what will become NiFi 1.1.0 using your JSONTest.txt and SplitJson with $.Records and I got 10 flow files on the "split" relationship, none to failure. Are you sure what is coming in from GetHTTP is JSON with the schema you expect? For the P

Re: Replace Text for updating the flow file content

Thanks for the help. Unfortunately there is a problem. I have used the SplitJson processor with $.Records to split the record array. I get the following warning WARN [Timer-Driven Process Thread-3] o.a.nifi.processors.standard.SplitJson SplitJson[id=0157101a-f11b-1e48-29c6-bf7c07ddd553] JsonPath $[

Re: [DISCUSS] NiFi 1.1.0 release

Team, Just an update on things with working toward an Apache NiFi 1.1.0 release. There are still about 33 JIRAs there now and some are awaiting review and are some are under active progress. Yet there is good traction and progress. I think we should just stay vigilant with what makes it in and ke

Re: Nifi ExecuteScript slow performance

Bala, Jython in ExecuteScript is noticably slower than other languages like Javascript and Groovy, but it shouldn't be that slow. Can you share your script code? Also, is this just to investigate before writing a more complex script in ExecuteScript, or do you just want to add an attribute to a fl

Re: [ANNOUNCE] New Apache NiFi Committer Bryan Rosander

A belated welcome! The TLS toolkit is good stuff!!! On Thu, Oct 27, 2016 at 9:17 AM, Bryan Rosander wrote: > Thanks everyone! > > I'm very grateful for all the help I've received in the community thus far > and am looking forward to working with all of you more! > > Thanks, > Bryan > > On Wed,

Re: Logback stops writing to nifi-app.log after restart

It was not during the file handle exhaustion issu. However, t I imagine it has to do with logback not being happy with that filename format, since I just restarted with %i and it now writes to the log upon restart. To demonstrate the rollover behavior, I changed to 10MB and started generating mi

Re: 0.7.1 download items

Done. I double checked the people, mailing list, and download updates. Let me know if anything doesn't look right. On Thu, Oct 27, 2016 at 8:54 AM, Joe Skora wrote: > No problem. I will do that right away. > > On Thu, Oct 27, 2016 at 8:37 AM, Joe Witt wrote: > >> Joe >> >> Anytime you make c

Re: CsvToAttributes processor

Hello again nifi folks, I did not get a direct reply to my email below. However, I've since noticed in the mailing list archive that some of you have kindly replied, although the emails did not make it to my inbox ! I wasn't part of the mailing list at the time, I am now, I guess that's why

Re: [ANNOUNCE] New Apache NiFi Committer Bryan Rosander

Thanks everyone! I'm very grateful for all the help I've received in the community thus far and am looking forward to working with all of you more! Thanks, Bryan On Wed, Oct 26, 2016 at 4:41 PM, Andre wrote: > Welcome aboard Byan! > > Your contributions are much appreciated by many NiFi users

Re: Logback stops writing to nifi-app.log after restart

if this was during the file handling exhaustion issue you'll see a lot of undesirable behaviors. If this is happening when file handles are not a problem then it will require deeper digging. That said, the base configuration does indeed write to nifi-app.log and roll over in the expected manner (

Re: Logback stops writing to nifi-app.log after restart

I'll try that, though I believe we removed the .%i because it continued to make logs more than the configured 10 when rolling over, which ran us out of disk space. On Thu, Oct 27, 2016 at 1:00 PM, Joe Witt wrote: > Joe > > I suspect you want to get closer to the original entry with the %i > part

Re: Logback stops writing to nifi-app.log after restart

Joe I suspect you want to get closer to the original entry with the %i part. So perhaps /opt/nifi/data/logs/nifi-app_%d{-MM-dd_HH}.%i.log Thanks Joe On Thu, Oct 27, 2016 at 8:57 AM, Joe Gresock wrote: > 23c23 > < > ${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log > --- >>

Re: Logback stops writing to nifi-app.log after restart

23c23 < ${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log --- > /opt/nifi/data/logs/nifi-app.log 31c31 < ${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{-MM-dd_HH}.%i.log --- > /opt/nifi/data/logs/nifi-app_%d{-MM-dd_HH}.log 36c36,55 < 30 --- >

Re: 0.7.1 download items

No problem. I will do that right away. On Thu, Oct 27, 2016 at 8:37 AM, Joe Witt wrote: > Joe > > Anytime you make changes to the codebase of nifi-site you'll want to > commit them (likely on master). Then you'll want to deploy the site > and the process for that can be found in the readme > h

Re: Logback stops writing to nifi-app.log after restart

Joe Can you highlight any changes (diff) from the standard configuration provided? Thanks Joe On Thu, Oct 27, 2016 at 8:34 AM, Joe Gresock wrote: > Before I create a ticket, I wanted to see if anyone had run into this. > > Whenever we restart nifi, the nifi-app.log no longer gets written to. >

Re: 0.7.1 download items

Joe Anytime you make changes to the codebase of nifi-site you'll want to commit them (likely on master). Then you'll want to deploy the site and the process for that can be found in the readme https://github.com/apache/nifi-site/blob/master/README.md. What happened yesterday is that Bryan update

Logback stops writing to nifi-app.log after restart

Before I create a ticket, I wanted to see if anyone had run into this. Whenever we restart nifi, the nifi-app.log no longer gets written to. If I delete all the nifi-app* logs and then restart, it writes logs just fine, but if I leave the logs it doesn't write to the log upon restarting nifi. Su

Re: Nifi ExecuteScript slow performance

Bala, Are you saying that step 2 (executing the script) is taking three seconds? Is that per message? Can you show the log or screenshot of how you're tracking that? Thanks Joe On Thu, Oct 27, 2016 at 6:02 AM, balacode63 wrote: > Dear All, > > I've added a ExecuteScript in python. This script

Nifi ExecuteScript slow performance

Dear All, I've added a ExecuteScript in python. This script is a simple script which will just add one attribute to the flowfile. The flow is definded as below, 1. Listen from mqtt (ConsumeMQTT) 2. Add attribute to nifi flow (Execute script) 3. Write to a file/post/ or some custom logic