Re: Questions related to check pointing

2021-01-12 Thread Raymond Wilson
Hi Zhenya, Thanks for the pointers - I will look into them. I have been doing some additional reading into this and discovered we are using a 4.0 NFS client, which seems to be the first 'no-no'; we will look at updating to use the 41 NFS client. We have modified our default timer cadence for che

Re: Questions related to check pointing

2021-01-10 Thread Zhenya Stanilovsky
fsync=37104ms too long for such pages amount : pages=33421, plz check how can you improve fsync on your storage.   > > >--- Forwarded message --- >From: "Raymond Wilson" < raymond_wil...@trimble.com > >To: user < user@ignite.apache.org >, "Zhenya Stanilovsky" < arzamas...@mail.ru >> >C

Re: Questions related to check pointing

2020-12-29 Thread Raymond Wilson
I noticed an entry in the Ignite 2.9.1 changelog: - Improved checkpoint concurrent behaviour I am having trouble finding the relevant Jira ticket for this in the 2.9.1 Jira area at https://issues.apache.org/jira/browse/IGNITE-13876?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%202.9.1%20

Re: Questions related to check pointing

2020-12-28 Thread Raymond Wilson
Hi Zhenya, 1. We currently use AWS EFS for primary storage, with provisioned IOPS to provide sufficient IO. Our Ignite cluster currently tops out at ~10% usage (with at least 5 nodes writing to it, including WAL and WAL archive), so we are not saturating the EFS interface. We use the default page

Re: Questions related to check pointing

2020-12-28 Thread Zhenya Stanilovsky
* Additionally to Ilya reply you can check vendors page for additional info, all in this page are applicable for ignite too [1]. Increasing threads number leads to concurrent io usage, thus if your have something like nvme — it`s up to you but in case of sas possibly better would be to reduce

Re: Questions related to check pointing

2020-12-28 Thread Raymond Wilson
As another detail, we have the WriteThrottlingEnabled property left at its default value of 'false', so I would not ordinarily expect throttling, correct? On Tue, Dec 29, 2020 at 10:04 AM Raymond Wilson wrote: > Hi Ilya, > > Regarding the throttling question, I have not yet looked at thread dump

Re: Questions related to check pointing

2020-12-28 Thread Raymond Wilson
Hi Ilya, Regarding the throttling question, I have not yet looked at thread dumps - the observed behaviour has been seen in production metrics and logging. What would you expect a thread dump to show in this case? Given my description of the sizes of the data regions and the numbers of pages bein

Re: Questions related to check pointing

2020-12-28 Thread Ilya Kasnacheev
Hello! 1. If we knew the specific circumstances in which a specific setting value will yield the most benefit, we would've already set it to that value. A setting means that you may tune it and get better results, or not. But in general we can't promise you anything. I did see improvements from in