On Wed, 2013-03-20 at 16:14 +0000, Grant Edwards wrote: > On 2013-03-20, Carlos Hendson <skyc...@gmx.net> wrote: > > > That's by no means conclusive, however, I've also run a complete pass of > > memcheck for over an hour without any issues reported. > > FWIW. I've had flakey memory that ran memcheck fine for several hours > and multiple passes -- but if I let it run long enough, it would fail. > I wouldn't be confident unless memtest ran for at least 12 hours (24 > would be even better). > > I'd also keep an eye on CPU core temperature. > > A failing hard-drive can also cause some pretty strange behavior. If > you're drives are smart (AFAICT, all recent ones are), ask them how > they're feeling with 'smartclt' or something like that. >
I'll run a 24 hour memtest this weekend. I started a long test on the hard drive: smartctl -t long /dev/sda smartctl -a /dev/sda also appears to indicate various errors (the output is attached). I'll trying to track down some documentation as to what they're actually reporting. Looking at the difference between the output of smartctl for before and during the test, there has been an increase in errors detected for ID #195 Before test: 195 Hardware_ECC_Recovered 0x001a 001 001 000 Old_age Always - 241822 During test: 195 Hardware_ECC_Recovered 0x001a 001 001 000 Old_age Always - 243582 Could this be the cause of the stalls during compiles? If it is the cause, is it possible for the kernel to detect such failures and report them? Thanks to you and everyone else for your ideas and suggestions. Regards, Carlos
smartctl 5.42 2011-10-20 r3458 [x86_64-linux-3.7.10-gentoo] (local build) Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net === START OF INFORMATION SECTION === Model Family: SAMSUNG SpinPoint M5 Device Model: SAMSUNG HM250JI Serial Number: S0TVJQSQ501163 LU WWN Device Id: 5 0f0000 031501163 Firmware Version: HS100-10 User Capacity: 250,059,350,016 bytes [250 GB] Sector Size: 512 bytes logical/physical Device is: In smartctl database [for details use: -P show] ATA Version is: 7 ATA Standard is: ATA/ATAPI-7 T13 1532D revision 0 Local Time is: Wed Mar 20 20:55:00 2013 CET SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED See vendor-specific Attribute list for marginal Attributes. General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 32) The self-test routine was interrupted by the host with a hard or soft reset. Total time to complete Offline data collection: ( 103) seconds. Offline data collection capabilities: (0x51) SMART execute Offline immediate. No Auto Offline data collection support. Suspend Offline collection upon new command. No Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 103) minutes. SCT capabilities: (0x003f) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 100 100 051 Pre-fail Always - 40 3 Spin_Up_Time 0x0007 252 252 025 Pre-fail Always - 2000 4 Start_Stop_Count 0x0032 099 099 000 Old_age Always - 2012 5 Reallocated_Sector_Ct 0x0033 054 054 010 Pre-fail Always - 428 7 Seek_Error_Rate 0x000f 252 252 051 Pre-fail Always - 0 8 Seek_Time_Performance 0x0025 252 252 015 Pre-fail Offline - 0 9 Power_On_Hours 0x0032 087 087 000 Old_age Always - 7095 10 Spin_Retry_Count 0x0033 252 252 051 Pre-fail Always - 0 11 Calibration_Retry_Count 0x0032 099 099 000 Old_age Always - 1151 12 Power_Cycle_Count 0x0032 099 099 000 Old_age Always - 1112 184 End-to-End_Error 0x0033 252 252 070 Pre-fail Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 34 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 57 190 Airflow_Temperature_Cel 0x0022 069 040 040 Old_age Always In_the_past 31 (Min/Max 8/60) 191 G-Sense_Error_Rate 0x0032 090 090 000 Old_age Always - 103348 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 332 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 9937 194 Temperature_Celsius 0x0022 064 040 000 Old_age Always - 31 (Min/Max 8/60) 195 Hardware_ECC_Recovered 0x001a 001 001 000 Old_age Always - 241822 196 Reallocated_Event_Count 0x0032 252 252 000 Old_age Always - 0 197 Current_Pending_Sector 0x0012 252 252 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 252 252 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0036 200 200 000 Old_age Always - 27 200 Multi_Zone_Error_Rate 0x000a 100 100 000 Old_age Always - 2 201 Soft_Read_Error_Rate 0x0032 252 252 000 Old_age Always - 0 SMART Error Log Version: 1 ATA Error Count: 26 (device log contains only the most recent five errors) CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 26 occurred at disk power-on lifetime: 6959 hours (289 days + 23 hours) When the command that caused the error occurred, the device was in an unknown state. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 08 08 a8 4a e0 Error: UNC 8 sectors at LBA = 0x004aa808 = 4892680 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 08 08 a8 4a e0 00 00:02:31.687 READ DMA c8 00 10 68 40 6b e0 00 00:02:24.250 READ DMA ef 05 fe 00 00 00 40 00 00:02:24.187 SET FEATURES [Enable APM] c8 00 10 e0 a7 4a e0 00 00:02:24.187 READ DMA c8 00 00 48 99 c7 e0 00 00:02:24.187 READ DMA Error 25 occurred at disk power-on lifetime: 5683 hours (236 days + 19 hours) When the command that caused the error occurred, the device was in an unknown state. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 20 40 9e 04 e1 Error: UNC 32 sectors at LBA = 0x01049e40 = 17079872 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 20 40 9e 04 e1 00 00:15:33.812 READ DMA c8 00 10 28 c3 94 e0 00 00:15:26.375 READ DMA ef 05 fe 00 00 00 40 00 00:15:26.312 SET FEATURES [Enable APM] c8 00 20 20 9e 04 e1 00 00:15:26.312 READ DMA c8 00 08 e0 a9 73 e0 00 00:15:26.312 READ DMA Error 24 occurred at disk power-on lifetime: 5534 hours (230 days + 14 hours) When the command that caused the error occurred, the device was in an unknown state. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 08 a0 c6 54 e0 Error: UNC 8 sectors at LBA = 0x0054c6a0 = 5555872 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 08 a0 c6 54 e0 00 00:10:28.500 READ DMA c8 00 68 d8 2b cc e0 00 00:10:21.062 READ DMA ef c2 00 00 00 00 40 00 00:10:21.062 SET FEATURES [Disable AAM] c8 00 10 58 c2 54 e0 00 00:10:21.062 READ DMA c8 00 20 b8 2b cc e0 00 00:10:21.062 READ DMA Error 23 occurred at disk power-on lifetime: 5463 hours (227 days + 15 hours) When the command that caused the error occurred, the device was in an unknown state. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 00 78 68 03 e0 Error: UNC at LBA = 0x00036878 = 223352 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 00 78 68 03 e0 00 00:50:45.250 READ DMA c8 00 00 68 c4 83 e0 00 00:50:37.812 READ DMA ef 05 fe 00 00 00 40 00 00:50:37.750 SET FEATURES [Enable APM] c8 00 00 78 67 03 e0 00 00:50:37.750 READ DMA c8 00 08 70 17 ab e0 00 00:50:37.750 READ DMA Error 22 occurred at disk power-on lifetime: 5383 hours (224 days + 7 hours) When the command that caused the error occurred, the device was in an unknown state. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 00 e8 4f 46 e0 Error: UNC at LBA = 0x00464fe8 = 4607976 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 00 e8 4f 46 e0 00 00:02:46.250 READ DMA c8 00 10 98 0f f5 e0 00 00:02:38.812 READ DMA ef 05 fe 00 00 00 40 00 00:02:38.750 SET FEATURES [Enable APM] c8 00 00 e8 4e 46 e0 00 00:02:38.750 READ DMA c8 00 88 e0 1f 3d e1 00 00:02:38.750 READ DMA SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 95 - Note: selective self-test log revision number (0) not 1 implies that no selective self-test has ever been run SMART Selective self-test log data structure revision number 0 Note: revision number not 1 implies that no selective self-test has ever been run SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Interrupted [00% left] (0-65535) 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
