Errors met rebuilden

[MisterE]

Laatst bij het starten van de PC was mijn soft-raid incompleet. het is een raid5 met 3 schijven.
Ik deed dus dit:
mdadm --assemble --force /dev/md99 /dev/sdd1 /dev/sde1 /dev/hdc1
aagezien hdc1 inmiddels als "spare" gezien word is ie nu aan het rebuilden.

niet lang daarna verschenen dit soort meldingen in "messages". en de rebuild tijd in enorm toegenomen.

Jul 10 12:47:21 x2 kernel: [ 116.410085] md: md99 stopped.
Jul 10 12:47:21 x2 kernel: [ 116.412521] md: bind<sde1>
Jul 10 12:47:21 x2 kernel: [ 116.412880] md: bind<hdc1>
Jul 10 12:47:21 x2 kernel: [ 116.413075] md: bind<sdd1>
Jul 10 12:47:21 x2 kernel: [ 116.445198] async_tx: api initialized (async)
Jul 10 12:47:21 x2 kernel: [ 116.462916] xor: automatically using best checksumming function: generic_sse
Jul 10 12:47:21 x2 kernel: [ 116.480011] generic_sse: 6286.000 MB/sec
Jul 10 12:47:21 x2 kernel: [ 116.480013] xor: using function: generic_sse (6286.000 MB/sec)
Jul 10 12:47:21 x2 kernel: [ 116.561016] raid6: int64x1 1876 MB/s
Jul 10 12:47:21 x2 kernel: [ 116.629010] raid6: int64x2 2413 MB/s
Jul 10 12:47:21 x2 kernel: [ 116.697033] raid6: int64x4 1858 MB/s
Jul 10 12:47:21 x2 kernel: [ 116.765012] raid6: int64x8 1663 MB/s
Jul 10 12:47:21 x2 kernel: [ 116.833063] raid6: sse2x1 662 MB/s
Jul 10 12:47:21 x2 kernel: [ 116.901040] raid6: sse2x2 1240 MB/s
Jul 10 12:47:21 x2 kernel: [ 116.969010] raid6: sse2x4 1907 MB/s
Jul 10 12:47:21 x2 kernel: [ 116.969012] raid6: using algorithm sse2x4 (1907 MB/s)
Jul 10 12:47:21 x2 kernel: [ 116.981531] md: raid6 personality registered for level 6
Jul 10 12:47:21 x2 kernel: [ 116.981535] md: raid5 personality registered for level 5
Jul 10 12:47:21 x2 kernel: [ 116.981537] md: raid4 personality registered for level 4
Jul 10 12:47:21 x2 kernel: [ 116.981747] raid5: device sdd1 operational as raid disk 0
Jul 10 12:47:21 x2 kernel: [ 116.981750] raid5: device sde1 operational as raid disk 1
Jul 10 12:47:21 x2 kernel: [ 116.982237] raid5: allocated 3230kB for md99
Jul 10 12:47:21 x2 kernel: [ 116.982286] 0: w=1 pa=0 pr=3 m=1 a=2 r=3 op1=0 op2=0
Jul 10 12:47:21 x2 kernel: [ 116.982290] 1: w=2 pa=0 pr=3 m=1 a=2 r=3 op1=0 op2=0
Jul 10 12:47:21 x2 kernel: [ 116.982297] RAID5 conf printout:
Jul 10 12:47:21 x2 kernel: [ 116.982299] --- rd:3 wd:2
Jul 10 12:47:21 x2 kernel: [ 116.982301] disk 0, o:1, dev:sdd1
Jul 10 12:47:21 x2 kernel: [ 116.982303] disk 1, o:1, dev:sde1
Jul 10 12:47:21 x2 kernel: [ 116.982340] md99: detected capacity change from 0 to 500113211392
Jul 10 12:47:21 x2 kernel: [ 116.982946] RAID5 conf printout:
Jul 10 12:47:21 x2 kernel: [ 116.982948] --- rd:3 wd:2
Jul 10 12:47:21 x2 kernel: [ 116.982951] disk 0, o:1, dev:sdd1
Jul 10 12:47:21 x2 kernel: [ 116.982953] disk 1, o:1, dev:sde1
Jul 10 12:47:21 x2 kernel: [ 116.982955] disk 2, o:1, dev:hdc1
Jul 10 12:47:21 x2 kernel: [ 116.983009] md: recovery of RAID array md99
Jul 10 12:47:21 x2 kernel: [ 116.983011] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
Jul 10 12:47:21 x2 kernel: [ 116.983013] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
Jul 10 12:47:21 x2 kernel: [ 116.983020] md: using 128k window, over a total of 244195904 blocks.
Jul 10 12:47:21 x2 kernel: [ 116.988162] md99: unknown partition table
Jul 10 12:49:46 x2 kernel: [ 262.048019] hdc: ide_dma_sff_timer_expiry: DMA status (0x61)
Jul 10 12:49:56 x2 kernel: [ 272.048020] hdc: DMA timeout error
Jul 10 12:49:56 x2 kernel: [ 272.048037] hdc: DMA disabled
Jul 10 12:49:57 x2 kernel: [ 272.192021] ide1: reset: success
Jul 10 12:55:25 x2 kernel: [ 600.592080] md99_resync D ffff8800581e5df0 0 3072 2 0x00000000
Jul 10 12:55:25 x2 kernel: [ 600.592086] ffff88005f9669f0 0000000000000046 ffff88005d152800 ffff88007c412c10
Jul 10 12:55:25 x2 kernel: [ 600.592091] ffff88005d152800 ffffffffa005f294 000000000000f8a0 ffff8800581e5fd8
Jul 10 12:55:25 x2 kernel: [ 600.592095] 00000000000155c0 00000000000155c0 ffff88005f9662e0 ffff88005f9665d8
Jul 10 12:55:25 x2 kernel: [ 600.592099] Call Trace:
Jul 10 12:55:25 x2 kernel: [ 600.592134] [<ffffffffa005f294>] ? scsi_request_fn+0x429/0x506 [scsi_mod]
Jul 10 12:55:25 x2 kernel: [ 600.592141] [<ffffffff8103fb0a>] ? __wake_up+0x30/0x44
Jul 10 12:55:25 x2 kernel: [ 600.592146] [<ffffffffa02bc74a>] ? unplug_slaves+0x7f/0xb4 [raid456]
Jul 10 12:55:25 x2 kernel: [ 600.592155] [<ffffffffa00d7a9f>] ? md_do_sync+0x5bc/0xaf6 [md_mod]
Jul 10 12:55:25 x2 kernel: [ 600.592161] [<ffffffff81064aae>] ? autoremove_wake_function+0x0/0x2e
Jul 10 12:55:25 x2 kernel: [ 600.592167] [<ffffffffa00d8828>] ? md_thread+0xf1/0x10f [md_mod]
Jul 10 12:55:25 x2 kernel: [ 600.592172] [<ffffffff8103ab60>] ? __wake_up_common+0x44/0x72
Jul 10 12:55:25 x2 kernel: [ 600.592177] [<ffffffffa00d8737>] ? md_thread+0x0/0x10f [md_mod]
Jul 10 12:55:25 x2 kernel: [ 600.592180] [<ffffffff810647e1>] ? kthread+0x79/0x81
Jul 10 12:55:25 x2 kernel: [ 600.592185] [<ffffffff81011b6a>] ? child_rip+0xa/0x20
Jul 10 12:55:25 x2 kernel: [ 600.592188] [<ffffffff81064768>] ? kthread+0x0/0x81

als ik echter met SMART kijk naar sdd

x2:~# smartctl -a /dev/sde
smartctl 5.40 2010-03-16 r3077 [x86_64-unknown-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family: SAMSUNG SpinPoint P120 series
Device Model: SAMSUNG SP2504C
Serial Number: S09QJ1GLA05367
Firmware Version: VT100-50
User Capacity: 250,059,350,016 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 7
ATA Standard is: ATA/ATAPI-7 T13 1532D revision 4a
Local Time is: Sat Jul 10 13:26:59 2010 CEST

==> WARNING: May need -F samsung3 enabled; see manual for details.

SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x02) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (4866) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 81) minutes.
SCT capabilities: (0x003f) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0007 253 253 025 Pre-fail Always - 5888
4 Start_Stop_Count 0x0032 099 099 000 Old_age Always - 1778
5 Reallocated_Sector_Ct 0x0033 253 253 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 253 253 051 Pre-fail Always - 0
8 Seek_Time_Performance 0x0025 253 253 015 Pre-fail Offline - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 8131
10 Spin_Retry_Count 0x0033 253 253 051 Pre-fail Always - 0
11 Calibration_Retry_Count 0x0012 253 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 099 099 000 Old_age Always - 1254
187 Reported_Uncorrect 0x0032 253 253 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 124 094 000 Old_age Always - 38
194 Temperature_Celsius 0x0022 124 094 000 Old_age Always - 38
195 Hardware_ECC_Recovered 0x001a 100 100 000 Old_age Always - 10521294
196 Reallocated_Event_Count 0x0032 253 253 000 Old_age Always - 0
197 Current_Pending_Sector 0x0012 253 253 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 253 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x000a 253 100 000 Old_age Always - 0
201 Soft_Read_Error_Rate 0x000a 100 100 000 Old_age Always - 0
202 Data_Address_Mark_Errs 0x0032 253 253 000 Old_age Always - 0

SMART Error Log Version: 1
ATA Error Count: 61 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 61 occurred at disk power-on lifetime: 4945 hours (206 days + 1 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
84 51 08 00 02 00 e0 Error: ICRC, ABRT 8 sectors at LBA = 0x00000200 = 512

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 08 00 02 00 e0 00 00:00:32.250 READ DMA
c8 00 40 20 00 00 e0 00 00:00:32.250 READ DMA
c8 00 18 08 00 00 e0 00 00:00:32.188 READ DMA
c8 00 08 00 00 00 e0 00 00:00:32.188 READ DMA
c8 00 08 f0 03 00 e0 00 00:00:31.313 READ DMA

Error 60 occurred at disk power-on lifetime: 299 hours (12 days + 11 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
84 51 80 37 e5 b8 e1 Error: ICRC, ABRT 128 sectors at LBA = 0x01b8e537 = 28894519

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 80 37 e5 b8 e1 00 00:08:10.688 READ DMA
c8 00 80 b7 e4 b8 e1 00 00:08:10.688 READ DMA
c8 00 68 4f e4 b8 e1 00 00:08:10.625 READ DMA
c8 00 70 df e3 b8 e1 00 00:08:10.625 READ DMA
c8 00 27 b8 e3 b8 e1 00 00:08:10.625 READ DMA

Error 59 occurred at disk power-on lifetime: 299 hours (12 days + 11 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
84 51 15 08 b4 b7 e1 Error: ICRC, ABRT 21 sectors at LBA = 0x01b7b408 = 28816392

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 15 08 b4 b7 e1 00 00:08:02.875 READ DMA
c8 00 01 07 b4 b7 e1 00 00:08:02.875 READ DMA
c8 00 76 91 b3 b7 e1 00 00:08:02.875 READ DMA
c8 00 64 2d b3 b7 e1 00 00:08:02.875 READ DMA
c8 00 25 08 b3 b7 e1 00 00:08:02.875 READ DMA

Error 58 occurred at disk power-on lifetime: 299 hours (12 days + 11 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
84 51 0f c8 1a 94 e2 Error: ICRC, ABRT 15 sectors at LBA = 0x02941ac8 = 43260616

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 0f c8 1a 94 e2 00 00:07:49.688 READ DMA
c8 00 01 c7 1a 94 e2 00 00:07:49.688 READ DMA
c8 00 03 c4 1a 94 e2 00 00:07:49.688 READ DMA
c8 00 80 44 1a 94 e2 00 00:07:49.688 READ DMA
c8 00 5d e7 19 94 e2 00 00:07:49.688 READ DMA

Error 57 occurred at disk power-on lifetime: 299 hours (12 days + 11 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
84 51 80 a7 60 92 e2 Error: ICRC, ABRT 128 sectors at LBA = 0x029260a7 = 43147431

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 80 a7 60 92 e2 00 00:07:32.438 READ DMA
c8 00 78 2f 60 92 e2 00 00:07:32.438 READ DMA
c8 00 62 cd 5f 92 e2 00 00:07:32.438 READ DMA
c8 00 25 a8 5f 92 e2 00 00:07:32.438 READ DMA
c8 00 01 a7 5f 92 e2 00 00:07:32.438 READ DMA

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]


Note: selective self-test log revision number (0) not 1 implies that no selective self-test has ever been run
SMART Selective self-test log data structure revision number 0
Note: revision number not 1 implies that no selective self-test has ever been run
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

dan lijkt mij sdd eigenlijk de schuldige.


maargoed, net vroeg ik dit op:

x2:~# mdadm --detail /dev/md99
/dev/md99:
Version : 0.90
Creation Time : Sun Sep 27 21:11:42 2009
Raid Level : raid5
Array Size : 488391808 (465.77 GiB 500.11 GB)
Used Dev Size : 244195904 (232.88 GiB 250.06 GB)
Raid Devices : 3
Total Devices : 3
Preferred Minor : 99
Persistence : Superblock is persistent

daar blijft ie nu "hangen". ook procstat staat stil.

ook SMART opvragen gaat nu mis:

x2:~# smartctl -a /dev/hdc
smartctl 5.40 2010-03-16 r3077 [x86_64-unknown-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

^C

iemand nog tips/test (dan natuurlijk snel backuppen)

[lamko]

Volgens mij ben je gewoon zwaar de sjaak, maar dat wist je zelf ook wel! Persoonlijk denk ik dat je wel op kunt houden en alvast een image kan trekken van de schijven en raid recovery software er overheen kunt draaien.

[dupondje]

Je krijgt de volgende error:
Jul 10 12:49:46 x2 kernel: [ 262.048019] hdc: ide_dma_sff_timer_expiry: DMA status (0x61)
Jul 10 12:49:56 x2 kernel: [ 272.048020] hdc: DMA timeout error
Jul 10 12:49:56 x2 kernel: [ 272.048037] hdc: DMA disabled

Misschien eens smart status opvragen van hdc ? en niet van sde ?

[Kippenijzer]

vervang je hdc (PATA toevallig?) door een SATA disk of draai heel snel backups. Je IDE controller gaat zo te zien over de zeik van het harde werk dat hij voor het rebuilden moet leveren (ofwel de disk is stuk)

[gertvdijk]

Euhm volgens mij is hier helemaal geen disk stuk, maar gaat hier mdadm over de zeik met een segfault. Kijk even of je de trace kan herleiden tot een bugreport.

Wat voor distro/versie en kernel gebruik je? Dan weet ik dat ik die niet moet gebruiken. mdadm mag echt nooit segfaulten imo.

[deadinspace]

MisterE schreef op zaterdag 10 juli 2010 @ 13:29:
Ik deed dus dit:
mdadm --assemble --force /dev/md99 /dev/sdd1 /dev/sde1 /dev/hdc1

Waarom dat commando? Waarom --force? Wat was precies de staat van je array voordat je dit deed?

Welke distributie gebruik je, en welke kernel draai je precies?

niet lang daarna verschenen dit soort meldingen in "messages". en de rebuild tijd in enorm toegenomen.

De rebuild hing (of hangt) te lang vast; het is me niet 100% duidelijk of dat iets te maken heeft met gaarheid van je disks of niet.

Het is voor ons trouwens een stuk leesbaarder als je [code] tags gebruikt ipv [quote].

als ik echter met SMART kijk naar sdd

De foutmelding heeft het over hdc, jij over hdd, en je commando over hde. Wat is nu precies wat?

gertvdijk schreef op maandag 12 juli 2010 @ 08:41:
Euhm volgens mij is hier helemaal geen disk stuk, maar gaat hier mdadm over de zeik met een segfault.

Waar zie jij een SEGV?