{"id":469,"date":"2012-10-19T15:49:01","date_gmt":"2012-10-19T19:49:01","guid":{"rendered":"https:\/\/infotechguy.net\/?p=469"},"modified":"2025-02-22T11:29:37","modified_gmt":"2025-02-22T16:29:37","slug":"mdadm-recovering-from-drive-failure","status":"publish","type":"post","link":"https:\/\/infotechguy.net\/?p=469","title":{"rendered":"Linux — Recovering from Drive Failure with mdadm"},"content":{"rendered":"

So it happened. I had a drive fail on me. Degrading my RAID 6 media server. Luckily I was notified by mdadm and was able to order a new one from newegg.com and rebuild it.<\/p>\n

I want to walk through the steps I took getting my RAID file system backup and running, starting with the notification I received to my gmail account (which i received on my phone).<\/p>\n

<\/p>\n

    \n
  1. \n
      \n
    1. Screen shot of email regarding the drive failure.
      \n\"\"<\/a><\/li>\n
    2. At home, I could see on the chassis the error light indicator on the drive bay with the failed Hard Disk. Luckly I had RAID 6 which allowed the RAID 1 more disk of fault tolerance.<\/li>\n
    3. On the server command line, I needed to inform mdadm that I was removing the drive from the array. So first make sure the logical disk representing the array is unmounted. In my case \/dev\/md0<\/strong>.<\/li>\n
      umount \/dev\/md0<\/pre>\n
    4. Check which drive has failed.\n
      mdadm --detail \/dev\/md0<\/pre>\n
      \"\"<\/a>
      \nAs you can see from the screenshot above, mdadm is aware of the drive failure and which drive has failed. Also notice the State: information. Clean, Degraded. Which is a good thing, cause if not it would mean the RAID is not recoverable.<\/li>\n
    5. Next we have to remove the drive from mdadm’s knowledge.\n
      mdadm --manage \/dev\/md0 --remove \/dev\/sdj1<\/pre>\n
      If you rerun the command at step 4, you should no longer see \/dev\/sdj1 in mdadm scope.<\/li>\n
    6. Put the new drive in, and prepare it.
      \nDelete any partitions that may be on the new drive and create 1 primary partition that takes up all the blocks on the disk. Lastly set it to linux raid filesystem type.<\/p>\n
      fdisk \/dev\/sdj\n......................................\nCommand (m for help):n<\/strong>\nCommand action\n\te\textended\n\tp\tprimary partition (1-4)\nPartition p<\/strong>\nPartition number (1-4): 1<\/strong>\nFirst cylinder (1-182401, default 1): [blank]<\/strong>\nLast cylinder or +size or +sizeM or +sizeK (1-182401, default 182401): [blank]<\/strong>\n\nCommand (m for help): t<\/strong>\nHex code (type L to list codes): fd<\/strong>\nChanged system type of partition 1 to fd (Linux raid auto)\n\nCommand (m for help): w<\/strong>\nThe partition table has been altered!\n\nCalling ioctl() to re-read partition table.\nSyncing disks.\n\n<\/pre>\n<\/li>\n
    7. Now before we add the drive back and begin the recovery process, let’s increase our raid drive I\/O speed, as according to zackreed<\/a>.\n
      echo 50000 > \/proc\/sys\/dev\/raid\/speed_limit_min \necho 200000 > \/proc\/sys\/dev\/raid\/speed_limit_max<\/pre>\n
      Effectively what we are doing is we are increasing the throttling mdadm<\/strong> can use to transfer blocks in the RAID. This is limited to the transfer speed of your Hard Disk device regardless of what you set the minimum to. Therefore, in my case the Hard Disks are SATA-II and get average 45MB\/s (guessing, haven’t done a speed test in awhile). Our first statement sets 50000 bytes to the speed_limit_min<\/strong> throttle, this is well above the drive’s speed limitation, so don’t be alarmed in the next step when the transfer speed doesn’t show exactly 50MB\/s when rebuilding the RAID array.<\/li>\n
    8. \/dev\/sdj1 should now be available and prepared to use with the RAID array. Add it to the array.\n
      mdadm --manage \/dev\/md0 --add \/dev\/sdj1<\/pre>\n
      Now you can watch the progress.<\/p>\n
      watch cat \/proc\/mdstat<\/pre>\n
      \"\"<\/a><\/p>\n
      Also, if you rerun the mdadm –detail \/dev\/md0<\/strong>command you see that it is reporting rebuilding the drive.
      \n      \"\"<\/a><\/li>\n
    9. After a few hours, the watch cat \/proc\/mdstat stated that the recovery was no longer going on. Check mdadm details.\n
      mdadm --detail \/dev\/md0<\/code><\/pre>\n
      \"\"<\/a><\/li>\n
    10. Lastly remount te drive and do a fsck check on it, for mine th array is in ext4 so,\n
      fsck.ext4 -f \/dev\/md0<\/pre>\n
      This will do an consistency check on the file-system and should come back clean.<\/li>\n<\/ol>\n
      That’s it! Recovered from a drive failure and all my data on the RAID array is intact.<\/p>\n
      Sources:<\/b><\/p>\n