The more i think about this, the more i believe that the PV had to be provisioned as some kind of network attached storage, and the exception had to be happening because of intermittent network issue for this NAS.
Sent with ProtonMail Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Monday, February 10, 2020 8:41 PM, Endre Kovacs <[email protected]> wrote: > Hi, > > What cloud are you running this on? > Kubernetes version? > > Can you share the storageClass you use for creating the repo PVCs? > > what is the topology of the cluster? Muli zone? > > Best regards, > Endre > > Sent with ProtonMail Secure Email. > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > On Monday, February 10, 2020 7:41 PM, Shawn Weeks [email protected] > wrote: > > > The more I look at the error the more it looks like the container couldn't > > access the persistence volume for some period of time. Can you share your > > yaml configuration minus sensitive stuff and what environment your > > deploying K8s on, aka Azure, ECS, Rancher, Openshift, etc. > > Thanks > > Shawn > > On 2/10/20, 12:27 PM, "Shawn Weeks" [email protected] wrote: > > I've been testing out NiFi on K8s quite a bit lately and might be able to > > help. What are you using for your persistent volume, this kinda sounds like > > NiFi can't always access data in the content_repository folder? > > Thanks > > Shawn > > On 2/10/20, 11:53 AM, "Joe Witt" [email protected] wrote: > > ...sent a little too quick. Also seems like if it is related to your k8s > > process that the issue would occur on older nifi releases too. So that > > might be an interesting test for you. > > On Mon, Feb 10, 2020 at 9:52 AM Joe Witt [email protected] wrote: > > > > > Joe > > > > > > > > > > If you can replicate this in a non k8s environment that would > > likely make > > > > > resolution easier. Far too many variables/questions at this > > stage. > > > > > > > > > > Thanks > > > > > > > > > > On Mon, Feb 10, 2020 at 8:15 AM Joe Gresock <[email protected]> > > wrote: > > > > > > > > > >> I don't know if there is something wrong with my Kubernetes-ized > > > > >> deployment > > > > >> of NiFi, but I'm seeing the following error crop up a lot in my > > NiFi > > > > >> 1.11.1 > > > > >> deployment. I can see the flow file attributes in "List Queue" > > for flow > > > > >> files affected by this problem, but the content claim is missing. > > > > >> > > > > >> It's not easily reproducible, but I've been running data > > constantly > > > > >> through > > > > >> the cluster for a few days, and every 2-4 hours or so, this > > problem > > > > >> appears > > > > >> in one of my processors (it's not always the same processors). > > The error > > > > >> only goes away when I drain the relevant queue, but obviously > > that's not > > > > >> an > > > > >> option in production. > > > > >> > > > > >> Any thoughts on what might be causing intermittent truncations > > in the > > > > >> content_repository? > > > > >> > > > > >> 2020-02-10 10:38:57,205 ERROR [Timer-Driven Process Thread-6] > > > > >> o.a.nifi.processors.standard.ReplaceText > > > > >> ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2] > > > > >> ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2] fa > > > > >> iled to process session due to > > > > >> org.apache.nifi.processor.exception.FlowFileAccessException: > > Could not > > > > >> read > > > > >> from > > > > >> > > > > >> > > StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim > > > > >> [resourceClaim=StandardResourceClaim[id=1581318452649-212433, > > > > >> container=default, section=465], offset=0, > > > > >> > > > > >> > > length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]; > > > > >> Processor Administratively Yielded for 1 sec: > > > > >> org.apache.nifi.processor.exception.FlowFileAccessException: > > Could not > > > > >> read > > > > >> from > > > > >> > > > > >> > > StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim > > > > >> [resourceClaim=StandardResourceClaim[id=1581318452649-212433, > > > > >> container=default, section=465], offset=0, > > > > >> > > > > >> > > length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289] > > > > >> org.apache.nifi.processor.exception.FlowFileAccessException: > > Could not > > > > >> read > > > > >> from > > > > >> > > > > >> > > StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim > > > > >> [resourceClaim=StandardResourceClaim[id=1581318452649-212433, > > > > >> container=default, section=465], offset=0, > > > > >> > > > > >> > > length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289] > > > > >> at > > > > >> org.apache.nifi.controller.repository.io > > > > >> > > .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:93) > > > > >> at > > > > >> org.apache.nifi.controller.repository.io > > > > >> > > .TaskTerminationInputStream.read(TaskTerminationInputStream.java:68) > > > > >> at > > > > >> > > org.apache.nifi.stream.io.StreamUtils.fillBuffer(StreamUtils.java:89) > > > > >> at > > > > >> > > > > >> > > org.apache.nifi.processors.standard.ReplaceText$RegexReplace$2.process(ReplaceText.java:532) > > > > >> at > > > > >> > > > > >> > > org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2324) > > > > >> at > > > > >> > > > > >> > > org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2292) > > > > >> at > > > > >> > > > > >> > > org.apache.nifi.controller.repository.BatchingSessionFactory$HighThroughputSession.read(BatchingSessionFactory.java:192) > > > > >> at > > > > >> > > > > >> > > org.apache.nifi.processors.standard.ReplaceText$RegexReplace.replace(ReplaceText.java:529) > > > > >> at > > > > >> > > > > >> > > org.apache.nifi.processors.standard.ReplaceText.onTrigger(ReplaceText.java:316) > > > > >> at > > > > >> > > > > >> > > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > > > > >> at > > > > >> > > > > >> > > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176) > > > > >> at > > > > >> > > > > >> > > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213) > > > > >> at > > > > >> > > > > >> > > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) > > > > >> at > > org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) > > > > >> at > > > > >> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > > > >> at > > > > >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > > > > >> at > > > > >> > > > > >> > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > > > > >> at > > > > >> > > > > >> > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > > > > >> at > > > > >> > > > > >> > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > > > >> at > > > > >> > > > > >> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > >> at java.lang.Thread.run(Thread.java:748) > > > > >> Caused by: java.io.EOFException: null > > > > >> at org.apache.nifi.stream.io > > > > >> .StreamUtils.skip(StreamUtils.java:270) > > > > >> at > > > > >> org.apache.nifi.controller.repository.io > > > > >> > > .ContentClaimInputStream.formDelegate(ContentClaimInputStream.java:155) > > > > >> at > > > > >> org.apache.nifi.controller.repository.io > > > > >> > > .ContentClaimInputStream.getDelegate(ContentClaimInputStream.java:51) > > > > >> at > > > > >> org.apache.nifi.controller.repository.io > > > > >> .ContentClaimInputStream.read(ContentClaimInputStream.java:89) > > > > >> at > > > > >> org.apache.nifi.controller.repository.io > > > > >> > > .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49) > > > > >> at > > > > >> org.apache.nifi.controller.repository.io > > > > >> .LimitedInputStream.read(LimitedInputStream.java:86) > > > > >> at > > > > >> org.apache.nifi.controller.repository.io > > > > >> > > .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49) > > > > >> at > > > > >> org.apache.nifi.stream.io > > > > >> .ByteCountingInputStream.read(ByteCountingInputStream.java:52) > > > > >> at > > java.io.FilterInputStream.read(FilterInputStream.java:133) > > > > >> at > > > > >> org.apache.nifi.controller.repository.io > > > > >> > > .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:82) > > > > >> ... 20 common frames omitted > > > > >> > > > > >> > > > > >> Joe Gresock > > > > >> > > > > > > >
