Another data point.. I noticed that the only nodes that seem to have this error are ones with recent pod restarts. Perhaps this is just a risk if the nifi process suddenly restarts?
In the log, I only see normal log statements, and then suddenly: 2020-02-13 18:20:06,810 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonito2020-02-13 18:23:50,779 INFO [main] org.apache.nifi.NiFi Launching NiFi... The disk is definitely not filling up, as I have all of my disks monitored. Perhaps an OOM could be the culprit, but I don't see anything in the logs. In normal operation, should NiFi be at risk of truncating content claims in case of the JVM suddenly stopping? On Thu, Feb 13, 2020 at 9:18 AM Joe Gresock <[email protected]> wrote: > Sure, sorry for the delay. > > It's an AWS EBS persistent volume. Here are the PV and storage class > objects. It's single-zone (us-east-1d): > > apiVersion: v1 > kind: PersistentVolume > metadata: > annotations: > kubernetes.io/createdby: aws-ebs-dynamic-provisioner > pv.kubernetes.io/bound-by-controller: "yes" > pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs > labels: > failure-domain.beta.kubernetes.io/region: us-east-1 > failure-domain.beta.kubernetes.io/zone: us-east-1d > name: pvc-abb1e679-523d-4ef7-a42e-cb42d8b8da0d > spec: > accessModes: > - ReadWriteOnce > awsElasticBlockStore: > fsType: ext4 > volumeID: aws://us-east-1d/vol-<instanceId> # Redacting some details > capacity: > storage: 150Gi > claimRef: > apiVersion: v1 > kind: PersistentVolumeClaim > name: content-repository-nifi-4 > namespace: default > resourceVersion: "7190127" > uid: abb1e679-523d-4ef7-a42e-cb42d8b8da0d > mountOptions: > - debug > nodeAffinity: > required: > nodeSelectorTerms: > - matchExpressions: > - key: failure-domain.beta.kubernetes.io/region > operator: In > values: > - us-east-1 > - key: failure-domain.beta.kubernetes.io/zone > operator: In > values: > - us-east-1d > persistentVolumeReclaimPolicy: Retain > storageClassName: gp2 > volumeMode: Filesystem > --- > apiVersion: storage.k8s.io/v1 > kind: StorageClass > metadata: > annotations: > kubectl.kubernetes.io/last-applied-configuration: | > {"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1 > ","kind":"StorageClass","metadata":{"annotations":{" > storageclass.kubernetes.io/is-default-class > ":"true"},"labels":{"addonmanager.kubernetes.io.mode":"EnsureExists"},"name":"gp2"},"mountOptions":["debug"],"parameters":{"type":"gp2"},"provisioner":" > kubernetes.io/aws-ebs > ","reclaimPolicy":"Retain","volumeBindingMode":"WaitForFirstConsumer"} > storageclass.kubernetes.io/is-default-class: "true" > creationTimestamp: "2020-01-03T12:30:49Z" > labels: > addonmanager.kubernetes.io.mode: EnsureExists > name: gp2 > resourceVersion: "126035" > selfLink: /apis/storage.k8s.io/v1/storageclasses/gp2 > uid: b91b9233-cd1b-4850-b0e1-bb984c4a31c5 > allowVolumeExpansion: true > mountOptions: > - debug > parameters: > type: gp2 > provisioner: kubernetes.io/aws-ebs > reclaimPolicy: Retain > volumeBindingMode: WaitForFirstConsumer > > On Tue, Feb 11, 2020 at 9:20 AM Endre Kovacs > <[email protected]> wrote: > >> The more i think about this, the more i believe that the PV had to be >> provisioned as some kind of network attached storage, and the exception had >> to be happening because of intermittent network issue for this NAS. >> >> >> >> >> Sent with ProtonMail Secure Email. >> >> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ >> On Monday, February 10, 2020 8:41 PM, Endre Kovacs < >> [email protected]> wrote: >> >> > Hi, >> > >> > What cloud are you running this on? >> > Kubernetes version? >> > >> > Can you share the storageClass you use for creating the repo PVCs? >> > >> > what is the topology of the cluster? Muli zone? >> > >> > Best regards, >> > Endre >> > >> > Sent with ProtonMail Secure Email. >> > >> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ >> > On Monday, February 10, 2020 7:41 PM, Shawn Weeks >> [email protected] wrote: >> > >> > > The more I look at the error the more it looks like the container >> couldn't access the persistence volume for some period of time. Can you >> share your yaml configuration minus sensitive stuff and what environment >> your deploying K8s on, aka Azure, ECS, Rancher, Openshift, etc. >> > > Thanks >> > > Shawn >> > > On 2/10/20, 12:27 PM, "Shawn Weeks" [email protected] wrote: >> > > I've been testing out NiFi on K8s quite a bit lately and might be >> able to help. What are you using for your persistent volume, this kinda >> sounds like NiFi can't always access data in the content_repository folder? >> > > Thanks >> > > Shawn >> > > On 2/10/20, 11:53 AM, "Joe Witt" [email protected] wrote: >> > > ...sent a little too quick. Also seems like if it is related to your >> k8s >> > > process that the issue would occur on older nifi releases too. So that >> > > might be an interesting test for you. >> > > On Mon, Feb 10, 2020 at 9:52 AM Joe Witt [email protected] wrote: >> > > >> > > > Joe >> > > >> > > > >> > > >> > > > If you can replicate this in a non k8s environment that >> would likely make >> > > >> > > > resolution easier. Far too many variables/questions at >> this stage. >> > > >> > > > >> > > >> > > > Thanks >> > > >> > > > >> > > >> > > > On Mon, Feb 10, 2020 at 8:15 AM Joe Gresock < >> [email protected]> wrote: >> > > >> > > > >> > > >> > > >> I don't know if there is something wrong with my >> Kubernetes-ized >> > > >> > > >> deployment >> > > >> > > >> of NiFi, but I'm seeing the following error crop up a lot >> in my NiFi >> > > >> > > >> 1.11.1 >> > > >> > > >> deployment. I can see the flow file attributes in "List >> Queue" for flow >> > > >> > > >> files affected by this problem, but the content claim is >> missing. >> > > >> > > >> >> > > >> > > >> It's not easily reproducible, but I've been running data >> constantly >> > > >> > > >> through >> > > >> > > >> the cluster for a few days, and every 2-4 hours or so, >> this problem >> > > >> > > >> appears >> > > >> > > >> in one of my processors (it's not always the same >> processors). The error >> > > >> > > >> only goes away when I drain the relevant queue, but >> obviously that's not >> > > >> > > >> an >> > > >> > > >> option in production. >> > > >> > > >> >> > > >> > > >> Any thoughts on what might be causing intermittent >> truncations in the >> > > >> > > >> content_repository? >> > > >> > > >> >> > > >> > > >> 2020-02-10 10:38:57,205 ERROR [Timer-Driven Process >> Thread-6] >> > > >> > > >> o.a.nifi.processors.standard.ReplaceText >> > > >> > > >> ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2] >> > > >> > > >> ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2] fa >> > > >> > > >> iled to process session due to >> > > >> > > >> >> org.apache.nifi.processor.exception.FlowFileAccessException: Could not >> > > >> > > >> read >> > > >> > > >> from >> > > >> > > >> >> > > >> > > >> >> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim >> > > >> > > >> >> [resourceClaim=StandardResourceClaim[id=1581318452649-212433, >> > > >> > > >> container=default, section=465], offset=0, >> > > >> > > >> >> > > >> > > >> >> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]; >> > > >> > > >> Processor Administratively Yielded for 1 sec: >> > > >> > > >> >> org.apache.nifi.processor.exception.FlowFileAccessException: Could not >> > > >> > > >> read >> > > >> > > >> from >> > > >> > > >> >> > > >> > > >> >> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim >> > > >> > > >> >> [resourceClaim=StandardResourceClaim[id=1581318452649-212433, >> > > >> > > >> container=default, section=465], offset=0, >> > > >> > > >> >> > > >> > > >> >> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289] >> > > >> > > >> >> org.apache.nifi.processor.exception.FlowFileAccessException: Could not >> > > >> > > >> read >> > > >> > > >> from >> > > >> > > >> >> > > >> > > >> >> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim >> > > >> > > >> >> [resourceClaim=StandardResourceClaim[id=1581318452649-212433, >> > > >> > > >> container=default, section=465], offset=0, >> > > >> > > >> >> > > >> > > >> >> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289] >> > > >> > > >> at >> > > >> > > >> org.apache.nifi.controller.repository.io >> > > >> > > >> >> .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:93) >> > > >> > > >> at >> > > >> > > >> org.apache.nifi.controller.repository.io >> > > >> > > >> >> .TaskTerminationInputStream.read(TaskTerminationInputStream.java:68) >> > > >> > > >> at >> > > >> > > >> org.apache.nifi.stream.io >> .StreamUtils.fillBuffer(StreamUtils.java:89) >> > > >> > > >> at >> > > >> > > >> >> > > >> > > >> >> org.apache.nifi.processors.standard.ReplaceText$RegexReplace$2.process(ReplaceText.java:532) >> > > >> > > >> at >> > > >> > > >> >> > > >> > > >> >> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2324) >> > > >> > > >> at >> > > >> > > >> >> > > >> > > >> >> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2292) >> > > >> > > >> at >> > > >> > > >> >> > > >> > > >> >> org.apache.nifi.controller.repository.BatchingSessionFactory$HighThroughputSession.read(BatchingSessionFactory.java:192) >> > > >> > > >> at >> > > >> > > >> >> > > >> > > >> >> org.apache.nifi.processors.standard.ReplaceText$RegexReplace.replace(ReplaceText.java:529) >> > > >> > > >> at >> > > >> > > >> >> > > >> > > >> >> org.apache.nifi.processors.standard.ReplaceText.onTrigger(ReplaceText.java:316) >> > > >> > > >> at >> > > >> > > >> >> > > >> > > >> >> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) >> > > >> > > >> at >> > > >> > > >> >> > > >> > > >> >> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176) >> > > >> > > >> at >> > > >> > > >> >> > > >> > > >> >> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213) >> > > >> > > >> at >> > > >> > > >> >> > > >> > > >> >> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) >> > > >> > > >> at >> org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) >> > > >> > > >> at >> > > >> > > >> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> > > >> > > >> at >> > > >> > > >> >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) >> > > >> > > >> at >> > > >> > > >> >> > > >> > > >> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) >> > > >> > > >> at >> > > >> > > >> >> > > >> > > >> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) >> > > >> > > >> at >> > > >> > > >> >> > > >> > > >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> > > >> > > >> at >> > > >> > > >> >> > > >> > > >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> > > >> > > >> at java.lang.Thread.run(Thread.java:748) >> > > >> > > >> Caused by: java.io.EOFException: null >> > > >> > > >> at org.apache.nifi.stream.io >> > > >> > > >> .StreamUtils.skip(StreamUtils.java:270) >> > > >> > > >> at >> > > >> > > >> org.apache.nifi.controller.repository.io >> > > >> > > >> >> .ContentClaimInputStream.formDelegate(ContentClaimInputStream.java:155) >> > > >> > > >> at >> > > >> > > >> org.apache.nifi.controller.repository.io >> > > >> > > >> >> .ContentClaimInputStream.getDelegate(ContentClaimInputStream.java:51) >> > > >> > > >> at >> > > >> > > >> org.apache.nifi.controller.repository.io >> > > >> > > >> >> .ContentClaimInputStream.read(ContentClaimInputStream.java:89) >> > > >> > > >> at >> > > >> > > >> org.apache.nifi.controller.repository.io >> > > >> > > >> >> .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49) >> > > >> > > >> at >> > > >> > > >> org.apache.nifi.controller.repository.io >> > > >> > > >> .LimitedInputStream.read(LimitedInputStream.java:86) >> > > >> > > >> at >> > > >> > > >> org.apache.nifi.controller.repository.io >> > > >> > > >> >> .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49) >> > > >> > > >> at >> > > >> > > >> org.apache.nifi.stream.io >> > > >> > > >> >> .ByteCountingInputStream.read(ByteCountingInputStream.java:52) >> > > >> > > >> at >> java.io.FilterInputStream.read(FilterInputStream.java:133) >> > > >> > > >> at >> > > >> > > >> org.apache.nifi.controller.repository.io >> > > >> > > >> >> .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:82) >> > > >> > > >> ... 20 common frames omitted >> > > >> > > >> >> > > >> > > >> >> > > >> > > >> Joe Gresock >> > > >> > > >> >> > > >> > > > >> > > >> >> >> > > -- > Be on your guard; stand firm in the faith; be courageous; be strong. Do > everything in love. -*1 Corinthians 16:13-14* > -- Be on your guard; stand firm in the faith; be courageous; be strong. Do everything in love. -*1 Corinthians 16:13-14*
