Hi Nifi Devs, We are having an issue with PutMongoRecord Processor(same for PutMongo as well) which is routing the flowfiles to the failure relationship when the Mongo database(or the server hosting the db) is down. We have a set-up which looks like below on a high-level overview.
[cid:[email protected]] As you can clearly see, all the failure relationships of different processors in the dataflow are connected to an Error Handling Process Group. Now, when the Mongodb is down, the PutMongoRecord processor is routing all the flowfiles to the failure relationship. Now, I know that a processor could route a flowfile to the failure relationship for multitude of reasons and I believe PutMongoRecord routing the flowfiles to failure when the database is down is not ideal for our scenario. Our dataflow is real-time streaming and when mongo database is down and flowfiles get routed to failure relationship, we get hundreds of emails because of the PutEmail processor in the Error Handling Process Group. We configured the Error Handling process group to check if we get any erroneous data which is against our schema or whatever maybe the reason. Is there any logic you guys help us in this scenario? I also observed that when we load the data to a RDBMS, lets say SQL Server, using the PutSql processor, it doesn’t route the flowfiles to the failure relationship when the sql server is down. I remember the flowfiles were held up in the connector above it until the db is up and running. I also noticed that PutSql has retry and PutMongo(and Record) doesn’t have any retry. Please help us what could be best thing for us to as this is critical to us. Thanks in Advance. Please reach out to me if you have any questions. Thanks and Regards [image005] Deepak Reddy | Data Engineer IT Centers of Excellence 13736 Riverport Dr., Maryland Heights, MO 63043 E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
