Oh, I just now read the 16 threads correctly, I was thinking you wrote 16 cores! 8 cores is far below specification if your running M-size blueprint ingest figures. I've seen 16 core intel servers (2016 spec xeon CPU's) go up to 70% utilization so that kind of load would never work on 8 cores, but again, I don't know how much managed data you have and what your ingest figures are.
On Wed, Jul 26, 2017 at 2:02 PM, Zoltan Forray <zfor...@vcu.edu> wrote: > I kinda feel the same way since my networking folks say it isn't the 10G > links (Xymon shows peaks of 2Gb), eventhough at it's peak processing load > it would be handling 5-TSM servers sending replications across the same 10G > links also used for the NFS. > > If the current processes ever finish (delete of 9M objects is now into > 48-hours, I will let the server sit for a day-or-two to see if it > improves. I have noticed that even with the server idle (no processes or > sessions), the CPU load-average was still higher than the 16-threads > available. I am seriously thinking about going back to the original 96GB > of RAM since it seems a lot of this slowdown started after bumping to > 192GB. > > On Wed, Jul 26, 2017 at 3:16 AM, Stefan Folkerts < > stefan.folke...@gmail.com> > wrote: > > > Interesting, why would NFS be the problem if the deletion of objects > > doesn't really touch the storagepools? > > > > I would wager that a straight up dd on the system to create a large file > > via 10Gb/s on NFS would be blazing fast but the database backup is slow > > because it's almost never idle, it's always behind it's intern processes > > such as reorgs. > > > > place your bets! :-) > > > > http://www.strawpoll.me/13536369 > > > > > > On Mon, Jul 24, 2017 at 3:55 PM, Sasa Drnjevic <sasa.drnje...@srce.hr> > > wrote: > > > > > Not sure of course...But, I would blame NFS > > > > > > Did you check the negotiated speed of your NFS eth 10G ifaces? > > > And that network? > > > > > > Regards, > > > > > > -- > > > Sasa Drnjevic > > > www.srce.unizg.hr > > > > > > > > > On 24.7.2017. 15:49, Zoltan Forray wrote: > > > > 8-cores/16-threads. It wasn't bad when it was replicating from > > 4-SP/TSM > > > > servers. We had to stop all replication due to running out of space > > and > > > > until I finish this cleanup, I have been holding off replication. > So, > > > the > > > > deletion has been running standalone. > > > > > > > > I forgot to mention that DB backups are also running very long. > 1.5TB > > DB > > > > backup runs 8+hours to NFS storage. These are connected via 10G. > > > > > > > > On Mon, Jul 24, 2017 at 9:41 AM, Sasa Drnjevic < > sasa.drnje...@srce.hr> > > > > wrote: > > > > > > > >> On 24.7.2017. 15:25, Zoltan Forray wrote: > > > >>> Due to lack of resources, we have had to stop replication on one of > > our > > > >> SP > > > >>> servers. The replication target server is 7.1.6.3 RHEL 7, Dell T710 > > > with > > > >>> 192GB RAM. NFS/ISILON storage. > > > >>> > > > >>> After removing replication from the nodes on source server, I have > > been > > > >>> cleaning up the replication server by deleting the filespaces for > the > > > >> nodes > > > >>> we are no longer replicating. > > > >>> > > > >>> My issue is the delete filespaces on the replication server is > taking > > > >>> forever. It took over a week to delete one filespace with > 31-million > > > >>> objects? > > > >> > > > >> > > > >> That is definitely tooooo loooong :-( > > > >> > > > >> It would take 6-8 hrs max, in my environment even under "standard" > > > load... > > > >> > > > >> How many CPU cores does it have? > > > >> > > > >> And how is/was it performing the role of a target repl. server > > > >> performance wise? > > > >> > > > >> Regards, > > > >> > > > >> -- > > > >> Sasa Drnjevic > > > >> www.srce.unizg.hr > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >>> > > > >>> To me it is highly unusual to take this long. Your thoughts on > this? > > > >>> > > > >>> -- > > > >>> *Zoltan Forray* > > > >>> Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator > > > >>> Xymon Monitor Administrator > > > >>> VMware Administrator > > > >>> Virginia Commonwealth University > > > >>> UCC/Office of Technology Services > > > >>> www.ucc.vcu.edu > > > >>> zfor...@vcu.edu - 804-828-4807 > > > >>> Don't be a phishing victim - VCU and other reputable organizations > > will > > > >>> never use email to request that you reply with your password, > social > > > >>> security number or confidential personal information. For more > > details > > > >>> visit http://infosecurity.vcu.edu/phishing.html > > > >>> > > > >> > > > > > > > > > > > > > > > > -- > > > > *Zoltan Forray* > > > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator > > > > Xymon Monitor Administrator > > > > VMware Administrator > > > > Virginia Commonwealth University > > > > UCC/Office of Technology Services > > > > www.ucc.vcu.edu > > > > zfor...@vcu.edu - 804-828-4807 > > > > Don't be a phishing victim - VCU and other reputable organizations > will > > > > never use email to request that you reply with your password, social > > > > security number or confidential personal information. For more > details > > > > visit http://infosecurity.vcu.edu/phishing.html > > > > > > > > > > > > > -- > *Zoltan Forray* > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator > Xymon Monitor Administrator > VMware Administrator > Virginia Commonwealth University > UCC/Office of Technology Services > www.ucc.vcu.edu > zfor...@vcu.edu - 804-828-4807 > Don't be a phishing victim - VCU and other reputable organizations will > never use email to request that you reply with your password, social > security number or confidential personal information. For more details > visit http://infosecurity.vcu.edu/phishing.html >
