Replacing a Disk

This procedure describes how to replace a failed disk in a StoredSafe appliance. When a disk malfunctions or fails, a red banner appears in the Web GUI with a link to the status page. On the physical appliance, the affected disk is also marked by a red indicator light. These indicators may not always appear at the same moment, but both refer to the same underlying disk failure.

Note

StoredSafe 5.x.x and later ships with StorCLI (/opt/MegaRAID/storcli/storcli64). Earlier versions use MegaCLI (/opt/storedsafe/bin/megacli). Use the section that matches your appliance version: Using StorCLI (StoredSafe 5.x.x and later) or Using MegaCLI (versions prior to 5.x.x).

To replace the disk, follow these steps:

  1. Logon on the StoredSafe server console as storedsafe and spawn a root shell.

  2. Use the RAID management tool to identify the failed disk and mark it as offline, missing and ready for removal.

  3. Physically replace the failed disk with a new one of the same or larger capacity.

  4. Use the RAID management tool to verify that the new disk is recognized by the system and that the rebuild has started.

  5. Wait for the RAID array to rebuild. This may take several hours depending on the size of the disks and the load on the system. The RAID array remains usable during the rebuild, but expect reduced performance until the process is complete.

  6. Verify that the RAID array is healthy after the rebuild process is complete.

  7. Exit the root shell and log out from the StoredSafe server console.

  8. Verify in the Web GUI that the disk replacement was successful and that there are no warnings.

Spawning a root shell from the console menu looks like this:

┌────────────────────────────────────────────────────────────────────────────┐
│              System Settings on node1 (Version X.X.X build XXXX)           │
└────────────────────────────────────────────────────────────────────────────┘

┌─┬──────────────────────────────────────────────────────────────────────────┐
│1│Network Settings                                                          │
│2│GUI Settings                                                              │
│3│Backup Management                                                         │
│4│Storage Management                                                        │
│5│Service Management                                                        │
│6│Firmware Management                                                       │
│7│Database Maintenance                                                      │
│8│Appliance Management                                                      │
└─┴──────────────────────────────────────────────────────────────────────────┘

Move the cursor or enter its corresponding number (Q to Quit)

Main> Unsupported, proceed at own risk. Warranty may be voided.
Shell timeout set to 3600 seconds.
root@safe:~#

Using StorCLI (StoredSafe 5.x.x and later)

StorCLI addresses objects by path: /c0 is controller 0, /c0/v0 is virtual drive 0 and /c0/e252/s3 is the physical disk in enclosure 252, slot 3.

  1. Check the state of the virtual drive. Dgrd (Degraded) means one or more disks have failed or are missing:

    root@safe:~# /opt/MegaRAID/storcli/storcli64 /c0/vall show
...
DG/VD TYPE  State Access Consist Cache Cac sCC     Size Name
0/0   RAID5 Dgrd  RW     Yes     RWBD  -   ON  2.727 TB
...

  2. List the physical disks to identify the failed one:

    root@safe:~# /opt/MegaRAID/storcli/storcli64 /c0/eall/sall show
...
EID:Slt DID State DG       Size Intf Med SED PI SeSz Model  Sp Type
252:0     4 Onln   0 931.0 GB SATA HDD N   N  512B ...    U  -
252:1     5 Onln   0 931.0 GB SATA HDD N   N  512B ...    U  -
252:2     6 Onln   0 931.0 GB SATA HDD N   N  512B ...    U  -
252:3     7 Offln  0 931.0 GB SATA HDD N   N  512B ...    U  -
...

    In this example the failed disk is 252:3, referenced as /c0/e252/s3 below. Common states are Onln (online), Offln (offline), UBad (unconfigured bad), UGood (unconfigured good) and Rbld (rebuilding).

  3. Set the disk offline (if the controller hasn’t already done so due to the error):

    root@safe:~# /opt/MegaRAID/storcli/storcli64 /c0/e252/s3 set offline

  4. Mark the disk as missing:

    root@safe:~# /opt/MegaRAID/storcli/storcli64 /c0/e252/s3 set missing

  5. Spin down the disk to prepare it for removal:

    root@safe:~# /opt/MegaRAID/storcli/storcli64 /c0/e252/s3 spindown

    Note

    If set missing or spindown fails because the disk is in a state that doesn’t support the command, check the disk’s state with /c0/eall/sall show. If it shows Rbld, the controller has already started rebuilding onto that disk (this can happen if the disk was still responding on the bus and got re-detected as available). Stop the rebuild before retrying:

    root@safe:~# /opt/MegaRAID/storcli/storcli64 /c0/e252/s3 stop rebuild

    Then verify the state has reverted (e.g. to Offln) before retrying. If the disk keeps returning to Rbld, it may not be truly failed and the cabling/backplane connection should be checked before removal.

  6. Optionally, make the disk’s indicator light blink to locate it physically:

    root@safe:~# /opt/MegaRAID/storcli/storcli64 /c0/e252/s3 start locate

    Warning

    Never remove a disk while its state is Rbld. Wait for the rebuild to complete or stop the rebuild before replacing the drive.

  7. Physically replace the failed disk with a new one of the same or larger capacity, then stop the locate indicator:

    root@safe:~# /opt/MegaRAID/storcli/storcli64 /c0/e252/s3 stop locate

  8. Verify that the new disk is recognized. In most cases the controller starts the rebuild automatically and the disk shows as Rbld:

    root@safe:~# /opt/MegaRAID/storcli/storcli64 /c0/eall/sall show

    If the disk shows as UBad, mark it as good first:

    root@safe:~# /opt/MegaRAID/storcli/storcli64 /c0/e252/s3 set good

    If the disk carries a foreign configuration (F in the DG column), inspect it and, provided it belongs to no array you want to keep, clear it:

    root@safe:~# /opt/MegaRAID/storcli/storcli64 /c0/fall show
root@safe:~# /opt/MegaRAID/storcli/storcli64 /c0/fall del

    If the rebuild still hasn’t started, insert the disk into the missing position in the array and start the rebuild manually (dg, array and row are shown by /c0/dall show; row is the disk’s position in the array, not necessarily its slot number):

    root@safe:~# /opt/MegaRAID/storcli/storcli64 /c0/e252/s3 insert dg=0 array=0 row=3
root@safe:~# /opt/MegaRAID/storcli/storcli64 /c0/e252/s3 start rebuild

  9. Monitor the rebuild. Depending on the size of the array, this may take several hours. The RAID array remains usable during the rebuild, but expect reduced performance until the process is complete:

    root@safe:~# /opt/MegaRAID/storcli/storcli64 /c0/e252/s3 show rebuild
...
Drive-ID     Progress% Status      Estimated Time Left
/c0/e252/s3         45 In progress 1 Hours 12 Minutes
...

  10. Once the rebuild is complete, the virtual drive should report Optl (Optimal), indicating that the disk replacement was successful. Then leave the root shell:

    root@safe:~# /opt/MegaRAID/storcli/storcli64 /c0/vall show
...
0/0   RAID5 Optl  RW     Yes     RWBD  -   ON  2.727 TB
...

root@safe:~# exit
exit

Using MegaCLI (versions prior to 5.x.x)

  1. Check the state of the virtual drive:

    root@safe:~# /opt/storedsafe/bin/megacli -ldinfo -lALL -aALL

Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name                :
RAID Level          : Primary-5, Secondary-0, RAID Level Qualifier-3
Size                : 2.727 TB
Sector Size         : 512
Is VD emulated      : No
Parity Size         : 931.0 GB
State               : Degraded
Strip Size          : 256 KB
Number Of Drives    : 4
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAhead, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAhead, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Disk's Default
Encryption Type     : None
Bad Blocks Exist: No
PI type: No PI
Is VD Cached: No

    In the example above the array is reported as ‘State : Degraded’, indicating one or more physical disks have failed or are missing.

  2. List the physical disks to identify the failed one. The command produces a lengthy output; in this example it lists four physical disks and their key details:

    root@safe:~# /opt/storedsafe/bin/megacli -pdlist -aALL

Enclosure Device ID: 252
Slot Number: 0
...
Firmware state: Online, Spun Up

Enclosure Device ID: 252
Slot Number: 1
...
Firmware state: Online, Spun Up

Enclosure Device ID: 252
Slot Number: 2
...
Firmware state: Online, Spun Up

Enclosure Device ID: 252
Slot Number: 3
...
Firmware state: Offline

    In this example, the failed disk appears as “Enclosure Device ID: 252” and “Slot Number: 3”. For the MegaCLI syntax, this drive is therefore referenced as [252:3] in the commands below.

  3. Set the disk offline (if the controller hasn’t already done so due to the error):

    root@safe:~# /opt/storedsafe/bin/megacli -PDOffline -PhysDrv [252:3] -a0
Adapter: 0: EnclId-252 SlotId-3 state changed to OffLine.
Exit Code: 0x00

  4. Mark the disk as missing:

    root@safe:~# /opt/storedsafe/bin/megacli -PDMarkMissing -PhysDrv [252:3] -a0
EnclId-252 SlotId-3 is marked Missing.
Exit Code: 0x00

  5. Mark the disk as prepared for removal:

    root@safe:~# /opt/storedsafe/bin/megacli -PDPrpRmv -PhysDrv [252:3] -a0
EnclId-252 SlotId-3 is marked for removal.
Exit Code: 0x00

    Note

    If this command fails with “FW error description: The specified device is in a state that doesn’t support the requested command”, check the disk’s firmware state with megacli -PDList -aALL. If it shows Rebuild, the controller has already started rebuilding onto that disk (this can happen if the disk was still responding on the bus and got re-detected as available). Stop the rebuild before retrying:

    root@safe:~# /opt/storedsafe/bin/megacli -PDRbld -Stop -PhysDrv [252:3] -a0

    Then verify the firmware state has reverted (e.g. to Offline or Missing) before running -PDPrpRmv again. If the disk keeps returning to a Rebuild state, it may not be truly failed and the cabling/backplane connection should be checked before removal.

  6. Optionally, make the disk’s indicator light blink to locate it physically:

    root@safe:~# /opt/storedsafe/bin/megacli -PDLocate -start -PhysDrv [252:3] -a0
Adapter: 0: Device at EnclId-252 SlotId-3 — PD Locate Start Command was successfully sent to Firmware
Exit Code: 0x00

    Warning

    Never remove a disk while its firmware state is Rebuild. Wait for the rebuild to complete or stop the rebuild before replacing the drive.

  7. Physically replace the failed disk with a new one of the same or larger capacity, then stop the locate indicator:

    root@safe:~# /opt/storedsafe/bin/megacli -PDLocate -stop -PhysDrv [252:3] -a0

  8. Verify that the new disk is recognized. In most cases the controller starts the rebuild automatically and the disk shows Firmware state: Rebuild:

    root@safe:~# /opt/storedsafe/bin/megacli -PDList -aALL
...
Firmware state: Rebuild
...

    If the disk shows as Unconfigured(bad), mark it as good first:

    root@safe:~# /opt/storedsafe/bin/megacli -PDMakeGood -PhysDrv [252:3] -a0

    If the disk carries a foreign configuration (Foreign State: Foreign), inspect it and, provided it belongs to no array you want to keep, clear it:

    root@safe:~# /opt/storedsafe/bin/megacli -CfgForeign -Scan -a0
root@safe:~# /opt/storedsafe/bin/megacli -CfgForeign -Clear -a0

    If the rebuild still hasn’t started, replace the missing disk in the array and start the rebuild manually (array and row are shown by -PdGetMissing; the row is the disk’s position in the array, not necessarily its slot number):

    root@safe:~# /opt/storedsafe/bin/megacli -PdGetMissing -a0
root@safe:~# /opt/storedsafe/bin/megacli -PdReplaceMissing -PhysDrv [252:3] -Array0 -row3 -a0
root@safe:~# /opt/storedsafe/bin/megacli -PDRbld -Start -PhysDrv [252:3] -a0
Rebuild for EnclId-252 SlotId-3 started.
Exit Code: 0x00

    Warning

    Do not use -PDOnline on a new disk. It forces the disk online without rebuilding it, which leaves the array inconsistent.

  9. Monitor the rebuild. Depending on the size of the array, this may take several hours. The RAID array remains usable during the rebuild, but expect reduced performance until the process is complete:

    root@safe:~# /opt/storedsafe/bin/megacli -PDRbld -ShowProg -PhysDrv [252:3] -a0
...
Rebuild Progress: 45%
...

  10. Once the rebuild is complete, the RAID array should report as ‘State : Optimal’, indicating that the disk replacement was successful. Then leave the root shell:

    root@safe:~# /opt/storedsafe/bin/megacli -ldinfo -lALL -aALL
...
State               : Optimal
...

root@safe:~# exit
exit

Command reference

Prefix each command with the full path to the tool (/opt/MegaRAID/storcli/storcli64 or /opt/storedsafe/bin/megacli). E is the enclosure ID and S the slot number.

Action

StorCLI (5.x.x and later)

MegaCLI (prior to 5.x.x)

Show virtual drive state

/c0/vall show

-LDInfo -LAll -aALL

List physical disks

/c0/eall/sall show

-PDList -aALL

Set disk offline

/c0/eE/sS set offline

-PDOffline -PhysDrv [E:S] -a0

Mark disk missing

/c0/eE/sS set missing

-PDMarkMissing -PhysDrv [E:S] -a0

Prepare for removal

/c0/eE/sS spindown

-PDPrpRmv -PhysDrv [E:S] -a0

Locate disk

/c0/eE/sS start locate

-PDLocate -start -PhysDrv [E:S] -a0

Mark unconfigured bad disk as good

/c0/eE/sS set good

-PDMakeGood -PhysDrv [E:S] -a0

Show / clear foreign configuration

/c0/fall show / /c0/fall del

-CfgForeign -Scan -a0 / -CfgForeign -Clear -a0

Replace missing disk

/c0/eE/sS insert dg=N array=N row=N

-PdReplaceMissing -PhysDrv [E:S] -ArrayN -rowN -a0

Start rebuild

/c0/eE/sS start rebuild

-PDRbld -Start -PhysDrv [E:S] -a0

Stop rebuild

/c0/eE/sS stop rebuild

-PDRbld -Stop -PhysDrv [E:S] -a0

Show rebuild progress

/c0/eE/sS show rebuild

-PDRbld -ShowProg -PhysDrv [E:S] -a0