First, find out which drive has failed.
lvs2# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid5]
md2 : active raid1 sda4[2](F) sdb4[1]
31872896 blocks [2/1] [_U]
md1 : active raid1 sda3[2](F) sdb3[1]
2104448 blocks [2/1] [_U]
md0 : active raid1 sdb1[1] sda1[2](F)
1052160 blocks [2/1] [_U]
unused devices: <none>
The drive that has '(F)' right next to it has failed.
In this case, sda is the drive we need to replace.
Remove the bad drive from the mirror.
lvs2# mdadm -r /dev/md0 /dev/sda1
mdadm: hot removed /dev/sda1
lvs2# mdadm -r /dev/md1 /dev/sda3
mdadm: hot removed /dev/sda3
lvs2# mdadm -r /dev/md2 /dev/sda4
mdadm: hot removed /dev/sda4
If you get an error saying the device is busy and you don't see an F
right next to the device in mdstat. you can manually set the device as
faulty by running the following command:
mdadm --manage --set-faulty /dev/mdX /dev/sdbX
Now we need to turn swap off, but we need to find out which partition is setup for swap first.
Take a look at the swap configuration on the box.
lvs2# cat /proc/swaps
Filename Type Size Used Priority
/dev/sda2 partition 530136 4 -1
/dev/sdb2 partition 522104 0 -2
This shows partition /dev/sda2 (remember the failed drive was /dev/sda) is on the drive we want to replace.
turn off swap on the bad hard drive.
lvs2# swapoff /dev/sda2
Release the hard drive from the scsi controller.
lvs2# scsiadd -r 0 0 0 0
Attached devices:
Host: scsi0 Channel: 00 Id: 01 Lun: 00
Vendor: FUJITSU Model: MAN3367MC Rev: 5508
Type: Direct-Access ANSI SCSI revision: 03
Host: scsi0 Channel: 00 Id: 06 Lun: 00
Vendor: PE/PV Model: 1x2 SCSI BP Rev: 1.0
Type: Processor ANSI SCSI revision: 02
we are specifying 0 for ID in here because we are replacing /dev/sda.
Please look at the man page for scsiadd to find the currect arguments
for other drives.
Now, go ahead and physically replace the hard drive.
Add the new drive to scsi controller
lvs2# scsiadd -a 0 0 0 0
Attached devices:
Host: scsi0 Channel: 00 Id: 01 Lun: 00
Vendor: FUJITSU Model: MAN3367MC Rev: 5508
Type: Direct-Access ANSI SCSI revision: 03
Host: scsi0 Channel: 00 Id: 06 Lun: 00
Vendor: PE/PV Model: 1x2 SCSI BP Rev: 1.0
Type: Processor ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: SEAGATE Model: ST336754LC Rev: 0003
Type: Direct-Access ANSI SCSI revision: 03
Find out what the partition table looks like on the good drive.
lvs2# fdisk -l /dev/sdb
Disk /dev/sdb: 36.4 GB, 36420075008 bytes
255 heads, 63 sectors/track, 4427 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 * 1 132 1060258+ fd Linux raid autodetect
/dev/sdb2 133 197 522112+ 82 Linux swap / Solaris
/dev/sdb3 198 459 2104515 83 Linux
/dev/sdb4 460 4427 31872960 83 Linux
Now create the exact same partition table on the new hard drive using fdisk (Please refer to fdisk's man page for more info).
Create swap.
lvs2# mkswap -c /dev/sda2
Setting up swapspace version 1, size = 534638 kB
no label, UUID=63c4eb60-373c-44fc-a405-2983db85b5b5
Turn swap on.
lvs2# swapon /dev/sda2
Add the newly created partitions to the mirror.
lvs2# mdadm -a /dev/md0 /dev/sda1
mdadm: hot added /dev/sda1
lvs2# mdadm -a /dev/md1 /dev/sda3
mdadm: hot added /dev/sda3
lvs2# mdadm -a /dev/md2 /dev/sda4
mdadm: hot added /dev/sda4
install MBR.
lvs2# install-mbr -t 2 /dev/sda
You are done!
You can watch the mirror building process by looking at mdstat.
lvs2# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid5]
md2 : active raid1 sda4[2] sdb4[1]
31872896 blocks [2/1] [_U]
[==>..................] recovery = 11.8% (3776768/31872896) finish=11.6min speed=40081K/sec
md1 : active raid1 sda3[0] sdb3[1]
2104448 blocks [2/2] [UU]
md0 : active raid1 sda1[0] sdb1[1]
1052160 blocks [2/2] [UU]
unused devices: <none>
