---
title: Replace an NVMe Node Drive
slug: l7f6-tnzLW_LVjM6doSpR
description: Learn the procedure for replacing NVMe drives in an Ocient node, ensuring data redundancy and fault tolerance.
docTags: 
createdAt: Thu Jul 21 2022 17:05:13 GMT+0000 (Coordinated Universal Time)
updatedAt: Mon Mar 30 2026 18:52:38 GMT+0000 (Coordinated Universal Time)
---

Ocient® software uses NVMe drives for storage because of their higher input or output (I/O) throughput than spinning media drives. These drives are referred to as data disks or drives.

In Ocient, these drives are accessed through a User Mode I/O (UIO) driver, so they cannot be accessed using the common set of Linux® commands (e.g., df/du/ls/file/parted/etc).

:::hint{type="warning"}
Working on these drives requires the `rolehostd` process to be stopped on the node, because the process has open file handles on the drives and making changes to the drive can make the process unstable.
:::

Execution of this process requires `sudo` privilege on the Foundation Node on which the drive is being replaced.

## Detection and Alerting

There are two approaches for detecting issues with NVMe drives on a Foundation Node. The device status of all drives on a specific Foundation Node can be checked using a SQL query of the system catalog to ensure all are active. Secondly, the presence of NVMe drives on the PCI bus can be inspected. You can run each command on a Foundation Node to assess the health of the drives on that node. Depending on the failure mode of a device, either of these methods could reveal an issue with an NVMe drive.

:::hint{type="info"}
In the case of an operating system drive failure, contact Ocient Support.
:::

### Check Drive Status Using System Catalog Tables

SQL queries on the `sys.storage_device_status` system catalog table provide a status of all payload drives. Connect to any SQL Node on the system and execute this query, replacing `<NODE_NAME>` with the name of your node.

```sql
SELECT
  n.name AS node_name,
  s.node_id,
  s.id AS serial_number,
  s.pci_address,
  s.device_status,
  s.device_model
FROM
  sys.nodes n
JOIN sys.storage_device_status s
  ON n.id = s.node_id
WHERE
  n.name = '<NODE_NAME>';
```

*Output*

```none
node_name |node_id                             |serial_number     |pci_address |device_status |device_model
----------+------------------------------------+------------------+------------+--------------+----------------------------------------
lts2      |f4845f46-dd0a-48bd-a2ed-614d5029e5df|CVMD6356009M800GGN|0000:3b:00.0|ACTIVE        |PCIe Data Center SSD INTEL SSDPE2ME800G4
lts2      |f4845f46-dd0a-48bd-a2ed-614d5029e5df|CVMD636400BZ800GGN|0000:3c:00.0|ACTIVE        |PCIe Data Center SSD INTEL SSDPE2ME800G4
lts2      |f4845f46-dd0a-48bd-a2ed-614d5029e5df|CVMD636400JA800GGN|0000:5e:00.0|ACTIVE        |PCIe Data Center SSD INTEL SSDPE2ME800G4
lts2      |f4845f46-dd0a-48bd-a2ed-614d5029e5df|CVMD6365003C800GGN|0000:5f:00.0|ACTIVE        |PCIe Data Center SSD INTEL SSDPE2ME800G4
lts2      |f4845f46-dd0a-48bd-a2ed-614d5029e5df|CVMD6365003P800GGN|0000:86:00.0|ACTIVE        |PCIe Data Center SSD INTEL SSDPE2ME800G4
lts2      |f4845f46-dd0a-48bd-a2ed-614d5029e5df|CVMD63650092800GGN|0000:87:00.0|ACTIVE        |PCIe Data Center SSD INTEL SSDPE2ME800G4
lts2      |f4845f46-dd0a-48bd-a2ed-614d5029e5df|CVMD6365009H800GGN|0000:88:00.0|ACTIVE        |PCIe Data Center SSD INTEL SSDPE2ME800G4
lts2      |f4845f46-dd0a-48bd-a2ed-614d5029e5df|CVMD6371005A800GGN|0000:89:00.0|FAILED        |PCIe Data Center SSD INTEL SSDPE2ME800G4
```

The possible storage device statuses are `ACTIVE` or `FAILED`. In this sample output, the device in PCI Address `0000:89:00.0` needs to be replaced.

If an NVMe drive is completely unavailable, it might not appear in the `sys.storage_device_status` table. For example, if a drive is not seated properly or if the node has rebooted and the drive is undetectable by Ocient. In this case, see the [Alternative Methods to Identify Failed Drives](./#alternative-methods-to-identify-failed-drives) to identify the serial number and PCI address of the drive.

In the event that a drive has failed or is experiencing issues, errors are logged in `/var/log/messages` and `/var/opt/ocient/log/rolehostd.log*` files. Inspection of these log files can assist with root cause analysis of the drive failure.

## Troubleshooting

1. To locate the failed drive in the chassis, use the query results from the [Check Drive Status via System Catalog Tables](./#check-drive-status-using-system-catalog-tables) query to identify the PCI Address of the failed drive.
2. Refer to the chassis diagram for your system to identify the slot of the failed drive based on the PCI address. Three common chassis and drive bay mappings are listed below:

&#x20;Supermicro®**&#x20;SYS-1029U-TN10RT Hot swap bays**

| NVMe 3c:00.0 | NVMe 5f:00.0 | NVMe 61:00.0 | NVMe 87:00.0 | NVMe 89:00.0 |
| ------------ | ------------ | ------------ | ------------ | ------------ |
| NVMe 3b:00.0 | NVMe 5e:00.0 | NVMe 60:00.0 | NVMe 86:00.0 | NVMe 88:00.0 |

## Recovery

### Prerequisites

- An NVMe drive of the same model and the same or larger capacity as the failed drive.

- The drive firmware is upgraded to the latest version using the [NVMe Drive Firmware Upgrade Process](<./NVMe Drive Firmware Upgrade Process.md>).

### Replacement Procedure

1\. Log in to the Foundation Node that needs a drive replacement as an administrator user and stop the `rolehostd` process on this node to ensure that the drive replacement does not impact the stability of the process:

```shell
sudo systemctl stop rolehostd
```

2\. Ensure that the `rolehostd` process has stopped. The result of this command should indicate that the process is `Stopped`.

```shell
sudo systemctl status rolehostd
```

3\. If using encrypted OPAL drives, back up the localKeyStore directory. This directory can be empty or absent on systems that do not have OPAL drives.

```shell
sudo cp -pR /var/opt/ocient/localKeyStore /root
```

4\. Physically remove the failed drive from the host and execute `nvme-driver-util.sh` to ensure that the failed drive is not showing up in the output.

```shell
/opt/ocient/scripts/nvme-driver-util.sh
```

*Output*

```none
[admin@go-lts9 ~]$ /opt/ocient/scripts/nvme-driver-util.sh
NVMe device status


use option 'bind-uio' to bind uio_pci_generic driver to Ocient Payload/unpartitioned drives
use option 'bind-nvme' to bind nvme driver to drives


BDF             Numa Node       Driver name             Device name
0000:3b:00.0    0               uio_pci_generic         -
0000:3c:00.0    0               uio_pci_generic         -
0000:5e:00.0    0               uio_pci_generic         -
0000:5f:00.0    0               uio_pci_generic         -
0000:86:00.0    1               uio_pci_generic         -
0000:87:00.0    1               uio_pci_generic         -
0000:88:00.0    1               uio_pci_generic         -
[admin@go-lts9 ~]$
```

5\. Insert the replacement drive in the chassis and ensure that it shows up in the output of the `nvme-driver-util.sh` script.

```shell
/opt/ocient/scripts/nvme-driver-util.sh
```

*O**utput*

```none
[admin@go-lts9 ~]$ /opt/ocient/scripts/nvme-driver-util.sh
NVMe device status


use option 'bind-uio' to bind uio_pci_generic driver to Ocient Payload/unpartitioned drives
use option 'bind-nvme' to bind nvme driver to drives


BDF             Numa Node       Driver name             Device name
0000:3b:00.0    0               uio_pci_generic         -
0000:3c:00.0    0               uio_pci_generic         -
0000:5e:00.0    0               uio_pci_generic         -
0000:5f:00.0    0               uio_pci_generic         -
0000:86:00.0    1               uio_pci_generic         -
0000:87:00.0    1               uio_pci_generic         -
0000:88:00.0    1               uio_pci_generic         -
0000:89:00.0    1               nvme                    -
[admin@go-lts9 ~]$
```

6\. Run this command by replacing `X` and `Y` to match the device name using the replacement drive.

```Text
sudo nvme format /dev/nvmeXnY format --ses=1
```

:::hint{type="danger"}
This command formats the specified drive and erases all data on the drive. If the OS drive is an NVMe drive, ensure that it is not formatted.
:::

7\. Execute the `nvme-driver-util.sh` script to bind drives to the UIO driver:

```shell
sudo /opt/ocient/scripts/nvme-driver-util.sh bind-uio
```

8\. Execute `nvme-driver-util.sh` script and confirm that the new drive is bound to `uio_pci_generic` driver.

```shell
/opt/ocient/scripts/nvme-driver-util.sh
```

*O**utput*

```none
[admin@go-lts9 ~]$ /opt/ocient/scripts/nvme-driver-util.sh
NVMe device status


use option 'bind-uio' to bind uio_pci_generic driver to Ocient Payload/unpartitioned drives
use option 'bind-nvme' to bind nvme driver to drives


BDF             Numa Node       Driver name             Device name
0000:3b:00.0    0               uio_pci_generic         -
0000:3c:00.0    0               uio_pci_generic         -
0000:5e:00.0    0               uio_pci_generic         -
0000:5f:00.0    0               uio_pci_generic         -
0000:86:00.0    1               uio_pci_generic         -
0000:87:00.0    1               uio_pci_generic         -
0000:88:00.0    1               uio_pci_generic         -
0000:89:00.0    1               uio_pci_generic         -
[admin@go-lts9 ~]$
```

9\. Start the `rolehostd` process.

```shell
sudo systemctl start rolehostd
```

10\. Check the output of the `sys.storage_device_status` catalog table by connecting to a SQL Node and running by following query after replacing the `<NODE_NAME>` with the name of your node. Ensure that all drives appear on the node with status of `ACTIVE`.

```sql
SELECT
  n.name AS node_name,
  s.node_id,
  s.id AS serial_number,
  s.pci_address,
  s.device_status,
  s.model
FROM
  sys.nodes n
JOIN sys.storage_device_status s
  ON n.id = s.node_id
WHERE
  n.name = '<NODE_NAME>';
```

*O**utput*

```none
node_name |node_id                             |serial_number     |pci_address |device_status |model
----------+------------------------------------+------------------+------------+--------------+----------------------------------------
lts2      |f4845f46-dd0a-48bd-a2ed-614d5029e5df|CVMD6356009M800GGN|0000:3b:00.0|ACTIVE        |PCIe Data Center SSD INTEL SSDPE2ME800G4
lts2      |f4845f46-dd0a-48bd-a2ed-614d5029e5df|CVMD636400BZ800GGN|0000:3c:00.0|ACTIVE        |PCIe Data Center SSD INTEL SSDPE2ME800G4
lts2      |f4845f46-dd0a-48bd-a2ed-614d5029e5df|CVMD636400JA800GGN|0000:5e:00.0|ACTIVE        |PCIe Data Center SSD INTEL SSDPE2ME800G4
lts2      |f4845f46-dd0a-48bd-a2ed-614d5029e5df|CVMD6365003C800GGN|0000:5f:00.0|ACTIVE        |PCIe Data Center SSD INTEL SSDPE2ME800G4
lts2      |f4845f46-dd0a-48bd-a2ed-614d5029e5df|CVMD6365003P800GGN|0000:86:00.0|ACTIVE        |PCIe Data Center SSD INTEL SSDPE2ME800G4
lts2      |f4845f46-dd0a-48bd-a2ed-614d5029e5df|CVMD63650092800GGN|0000:87:00.0|ACTIVE        |PCIe Data Center SSD INTEL SSDPE2ME800G4
lts2      |f4845f46-dd0a-48bd-a2ed-614d5029e5df|CVMD6365009H800GGN|0000:88:00.0|ACTIVE        |PCIe Data Center SSD INTEL SSDPE2ME800G4
lts2      |f4845f46-dd0a-48bd-a2ed-614d5029e5df|CVMD637100GA800GGN|0000:89:00.0|ACTIVE        |PCIe Data Center SSD INTEL SSDPE2ME800G4
```

As shown in the sample output, the device replaced at PCI Address `0000:89:00.0` is now active. This indicates that it has been successfully replaced.

### Next Steps: Rebuild Segments

If the failed drive had data on it prior to failure, a rebuild of data segments stored on the replaced drive is required after the drive is replaced. Refer to the [Guide to Rebuilding Segments](<././Guide to Rebuilding Segments.md>) for instructions on how to rebuild the missing segments.

## Alternative Methods to Identify Failed Drives

If the results of querying the `sys.storage_device_status` table do not indicate which drive has failed, alternative methods can be used to detect the failing drive. These methods require the command-line utility `jq`, but you can view the full results of the `curl` commands without it.

### Option 1: Cross Reference API Results to Identify the PCI Address

You can cross-reference the result of two API commands to determine the PCI address of the failed drive that is not appearing. A drive that does not appear in the `stats` API or catalog table can be present in the `sysconfig` API output.

The output of the following API shows NVMe storage device statuses. Any device status other than 10 indicates a potential issue. Replace `<IP_ADDRESS_OF_THE_NODE>` with the IP Address of your node and inspect the results.

```curl
curl -s http://<IP_ADDRESS_OF_THE_NODE>:9090/v1/stats | jq -r '.[] |select (.name == "localStorageService.device.status" )| "\(.device) \(.value)" '
```

*O**utput*

```none
$ curl -s http://1.2.3.4:9090/v1/stats | jq -r '.[] |select (.name == "localStorageService.device.status" )| "\(.device) \(.value)" '
BTLJ913605YR2P0BGN 10
BTLJ913608AY2P0BGN 10
BTLJ913608FR2P0BGN 10
BTLJ91360D6Q2P0BGN 10
BTLJ91360D6Y2P0BGN 10
BTLJ916306LS2P0BGN 10
BTLJ92020KP22P0BGN 10
BTLJ92030AJM2P0BGN 10
PHLJ9403002N2P0BGN 10
PHLJ941100RW2P0BGN 10
```

The possible device status values are listed in the table below:

| **Value** | **Device Status** |
| --------- | ----------------- |
| 0         | INVALID\_UNKNOWN  |
| 10        | ACTIVE            |
| 20        | UNINITIALIZED     |
| 61        | CORRUPT           |
| 80        | FAILED            |

### Check Device Presence

The output of the following API will show the configured NVMe drives on the Foundation Node. Depending on the failure mode, the output of this command can show a reduced number of drives. For example, if the Foundation Node chassis has 12 drive bays populated and only 11 appear in this result, it would indicate that one drive is not visible to Ocient. Run the following command by replacing `<IP_ADDRESS_OF_THE_NODE>` with the IP Address of your node:

```curl
curl -s http://<IP_ADDRESS_OF_THE_NODE>:9090/v1/sysconfig | jq -r '.storage_devices[] | select ( .use != "LOCAL_FILE_SYSTEM")| "\(.id) \(.pci_address)"'
```

*O**utput*

```none
$ curl -s http://1.2.3.4:9090/v1/sysconfig | jq -r '.storage_devices[] | select ( .use != "LOCAL_FILE_SYSTEM")| "\(.id) \(.pci_address)"'
BTLJ913608AY2P0BGN 0000:3b:00.0
BTLJ913605YR2P0BGN 0000:3c:00.0
BTLJ913608FR2P0BGN 0000:5e:00.0
PHLJ9403002N2P0BGN 0000:5f:00.0
BTLJ92020KP22P0BGN 0000:60:00.0
BTLJ92030AJM2P0BGN 0000:61:00.0
PHLJ941100RW2P0BGN 0000:86:00.0
BTLJ91360D6Y2P0BGN 0000:87:00.0
BTLJ916306LS2P0BGN 0000:88:00.0
BTLJ91360D6Q2P0BGN 0000:89:00.0
```

If the address of the failed drive cannot be determined in this way, proceed to Option 2 to identify the location of the drive.

### Option 2: Inspect Drives Using Operating System Commands

If drives are not appearing in the Ocient outputs, they might be unavailable to the Ocient service. In this case, directly examine the drives on the system with standard disk utilities and operating system commands to find the serial numbers of drives that meet the following criteria:

1. The drive is not the operating system boot drive.
2. The drive is not listed in the `sys.storage_device_status` table.

Any drives that meet this criteria are not in use by Ocient and might need to be replaced or repaired. Identify the Serial Number and PCI Addresses of these drives in order to complete the replacement procedure.

## Related Links

[System Catalog](https://docs.ocient.com/system-catalog)

***

**Linux®*&#x20;is the registered trademark of Linus Torvalds in the U.S. and other countries.*

