> Op 7 juli 2017 om 2:20 schreef Reed Dier <reed.d...@focusvq.com>: > > > I could easily see that being the case, especially with Micron as a common > thread, but it appears that I am on the latest FW for both the SATA and the > NVMe: > > > $ sudo ./msecli -L | egrep 'Device|FW' > > Device Name : /dev/sda > > FW-Rev : D0MU027 > > Device Name : /dev/sdb > > FW-Rev : D0MU027 > > Device Name : /dev/sdc > > FW-Rev : D0MU027 > > Device Name : /dev/sdd > > FW-Rev : D0MU027 > > Device Name : /dev/sde > > FW-Rev : D0MU027 > > Device Name : /dev/sdf > > FW-Rev : D0MU027 > > Device Name : /dev/sdg > > FW-Rev : D0MU027 > > Device Name : /dev/sdh > > FW-Rev : D0MU027 > > Device Name : /dev/sdi > > FW-Rev : D0MU027 > > Device Name : /dev/sdj > > FW-Rev : D0MU027 > > Device Name : /dev/nvme0 > > FW-Rev : 0091634 > > D0MU027 and 1634 are the latest reported FW from Micron, current as of > 04/12/2017 and 12/07/2016, respectively. > > Could be current FW doesn’t play nice, so thats on the table. But for now, > its a thread that can’t be pulled any further. >
Also keep in mind that only with SATA 3.1 there is Queued TRIM. Many controllers out there are SATA 3.0 and they block when trimming. So it could either be a firmware or the SATA controller. I also have PM863a SSDs which block during TRIM because the controller (Intel X99) doesn't support it. Wido > Appreciate the feedback, > > Reed > > > On Jul 6, 2017, at 1:18 PM, Peter Maloney > > <peter.malo...@brockmann-consult.de> wrote: > > > > Hey, > > > > I have some SAS Micron S630DC-400 which came with firmware M013 which did > > the same or worse (takes very long... 100% blocked for about 5min for 16GB > > trimmed), and works just fine with firmware M017 (4s for 32GB trimmed). So > > maybe you just need an update. > > > > Peter > > > > > > > > On 07/06/17 18:39, Reed Dier wrote: > >> Hi Wido, > >> > >> I came across this ancient ML entry with no responses and wanted to follow > >> up with you to see if you recalled any solution to this. > >> Copying the ceph-users list to preserve any replies that may result for > >> archival. > >> > >> I have a couple of boxes with 10x Micron 5100 SATA SSD’s, journaled on > >> Micron 9100 NVMe SSD’s; ceph 10.2.7; Ubuntu 16.04 4.8 kernel. > >> > >> I have noticed now twice that I’ve had SSD’s flapping due to the fstrim > >> eating up the io 100%. > >> It eventually righted itself after a little less than 8 hours. > >> Noout flag was set, so it didn’t create any unnecessary rebalance or > >> whatnot. > >> > >> Timeline showing that only 1 OSD ever went down at a time, but they seemed > >> to go down in a rolling fashion during the fstrim session. > >> You can actually see in the OSD graph all 10 OSD’s on this node go down 1 > >> by 1 over time. > >> > >> > >> And the OSD’s were going down because of: > >> > >>> 2017-07-02 13:47:32.618752 7ff612721700 1 heartbeat_map is_healthy > >>> 'OSD::osd_op_tp thread 0x7ff5ecd0c700' had timed out after 15 > >>> 2017-07-02 13:47:32.618757 7ff612721700 1 heartbeat_map is_healthy > >>> 'FileStore::op_tp thread 0x7ff608d9e700' had timed out after 60 > >>> 2017-07-02 13:47:32.618760 7ff612721700 1 heartbeat_map is_healthy > >>> 'FileStore::op_tp thread 0x7ff608d9e700' had suicide timed out after 180 > >>> 2017-07-02 13:47:32.624567 7ff612721700 -1 common/HeartbeatMap.cc > >>> <http://heartbeatmap.cc/>: In function 'bool > >>> ceph::HeartbeatMap::_check(const ceph::heartbeat_handle_d*, const char*, > >>> time_t)' thread 7ff612721700 time 2017-07-02 13:47:32.618784 > >>> common/HeartbeatMap.cc <http://heartbeatmap.cc/>: 86: FAILED assert(0 == > >>> "hit suicide timeout") > >> > >> > >> I am curious if you were able to nice it or something similar to mitigate > >> this issue? > >> Oddly, I have similar machines with Samsung SM863a’s with Intel P3700 > >> journals that do not appear to be affected by the fstrim load issue > >> despite identical weekly cron jobs enabled. Only the Micron drives (newer) > >> have had these issues. > >> > >> Appreciate any pointers, > >> > >> Reed > >> > >>> Wido den Hollander wido at 42on.com > >>> <mailto:ceph-users%40lists.ceph.com?Subject=Re%3A%20%5Bceph-users%5D%20Watch%20for%20fstrim%20running%20on%20your%20Ubuntu%20systems&In-Reply-To=%3C5486BF08.3010505%4042on.com%3E> > >>> Tue Dec 9 01:21:16 PST 2014 > >>> Hi, > >>> > >>> Last sunday I got a call early in the morning that a Ceph cluster was > >>> having some issues. Slow requests and OSDs marking each other down. > >>> > >>> Since this is a 100% SSD cluster I was a bit confused and started > >>> investigating. > >>> > >>> It took me about 15 minutes to see that fstrim was running and was > >>> utilizing the SSDs 100%. > >>> > >>> On Ubuntu 14.04 there is a weekly CRON which executes fstrim-all. It > >>> detects all mountpoints which can be trimmed and starts to trim those. > >>> > >>> On the Intel SSDs used here it caused them to become 100% busy for a > >>> couple of minutes. That was enough for them to no longer respond on > >>> heartbeats, thus timing out and being marked down. > >>> > >>> Luckily we had the "out interval" set to 1800 seconds on that cluster, > >>> so no OSD was marked as "out". > >>> > >>> fstrim-all does not execute fstrim with a ionice priority. From what I > >>> understand, but haven't tested yet, is that running fstrim with ionice > >>> -c Idle should solve this. > >>> > >>> It's weird that this issue didn't come up earlier on that cluster, but > >>> after killing fstrim all problems we resolved and the cluster ran > >>> happily again. > >>> > >>> So watch out for fstrim on early Sunday mornings on Ubuntu! > >>> > >>> -- > >>> Wido den Hollander > >>> 42on B.V. > >>> Ceph trainer and consultant > >> > >> > >> _______________________________________________ > >> ceph-users mailing list > >> ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> > > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
