---
title: Guide to Rebuilding Segments
slug: pKLHOPHCAqwlsKKSi2_o4
description: Discover methods for rebuilding segments in Ocient, maintaining data integrity and system performance.
docTags: 
createdAt: Thu Jul 21 2022 17:05:13 GMT+0000 (Coordinated Universal Time)
updatedAt: Wed Apr 22 2026 20:49:38 GMT+0000 (Coordinated Universal Time)
---

Data in an Ocient® System is erasure coded to provide resilience to disk and Foundation Node failures. The resilience of a system depends on the parity width of the storage space, which represents the number of associated disks (or nodes) that can fail without interruption of service or data loss.&#x20;

For details about storage spaces and parity width, see [Configure Storage Spaces](<././Configure Storage Spaces.md>).

## Data Segment Statuses

You can check the status of your system segment groups by querying the `sys.segment_groups` system catalog table. For details, see [System Catalog](<./././SQL Reference/System Catalog Reference/System Catalog.md>).

This table describes the states for data segments.

| **Status** | **Description**                                                                                                                              | **Recovery Process**                                                                                                                               |
| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| INTACT     | The normal and operable status.                                                                                                              | No recovery needed.                                                                                                                                |
| DAMAGED    | The segment failed a checksum, meaning the segment data is corrupted and unusable.                                                           | If you have sufficient parity width, you can recover a damaged segment by invoking a rebuild segment task.                                         |
| MISSING    | The segment is on a node or disk that is currently offline. If the node or disk rejoins the cluster, it can transition to the INTACT status. | When a disk or Foundation Node is permanently removed, you can perform a rebuild task to recover the data. This requires sufficient parity width.  |
| REBUILDING | The segment is in recovery after damage or missing segment data.                                                                             | Recovery is already in progress.                                                                                                                   |

## Recovery Considerations

If a segment has the DAMAGED or MISSING status, queries can proceed by reconstructing the data on demand using the remaining erasure-coded data in the segment group. Having a non-INTACT status means that input or output (I/O) performance is significantly reduced.&#x20;

To restore full performance, you need to run a segment rebuild task. Segment rebuilding is not automatic. The system administrator must manually invoke it.

A segment rebuild fails, and data is lost completely if the number of segments with the DAMAGED status in a segment group exceeds the parity width of the storage space. To avoid data loss:

- Provision the parity width of the storage space at or above the number of expected concurrent node failures.
- Rebuild damaged or permanently missing segments as soon as possible.

## Checking for Abnormal Segments

You can find any segments that need a rebuild by querying for segment groups with an abnormal status.&#x20;

**Examples**

**Finding Faulty Segment Groups**

This example query finds any segment groups with the DAMAGED or MISSING status.&#x20;

```sql
SELECT * FROM sys.segment_groups WHERE status IN ('DAMAGED', 'MISSING');
```

*Output*

```none
|         "id"         |              "cluster_id"              | "segment_type" | "status" | "primary_owner" |              "loader_id"               |               "table_id"               |               "scope_id"               | "block_size" | "begin_time" | "end_time" | "coding_algorithm" | "coding_block_size" | "coding_threshold" | "coding_width" | "replication" | "parity_cycle" |     "created_time"      | "rolehostd_version" |               "commit_hash"                |    "timestamp"    | "build_user" | "depth" | "removal_type" |
|----------------------|----------------------------------------|----------------|----------|-----------------|----------------------------------------|----------------------------------------|----------------------------------------|--------------|--------------|------------|--------------------|---------------------|--------------------|----------------|---------------|----------------|-------------------------|---------------------|--------------------------------------------|-------------------|--------------|---------|----------------|
| "720576045199095878" | "074c32ec-4f92-4718-ada1-5eb55eafdcb3" | TKT_SEGMENT    | DAMAGED  |                 | "9a8f7ea7-613c-499f-ab2d-937ab0ce992e" | bd18d33b-aae9-4be2-a2f4-76ecdc34c2ba   | "8bc119d2-875a-47e4-b016-87fde83e77d6" |         4096 |            0 |          1 | XOR_PARITY         |                4096 |                  2 |              3 |             1 |              1 | 2025-02-26 20:23:34.274 | "25.0.0"            | "91ab76ae57491ade0122bc7b594d5c3c6e0bf40c" | "20250109.221519" |              |       0 | NOT_REMOVED    |
| "716072445571725367" | "074c32ec-4f92-4718-ada1-5eb55eafdcb3" | TKT_SEGMENT    | DAMAGED  |                 | "9a8f7ea7-613c-499f-ab2d-937ab0ce992e" | "9fe79009-ee75-4429-b29b-3a614a166751" | a8c805d0-2144-46a7-8374-52cb88d64244   |         4096 |            0 |          1 | XOR_PARITY         |                4096 |                  2 |              3 |             1 |              1 | 2025-02-26 20:23:19.447 | "25.0.0"            | "91ab76ae57491ade0122bc7b594d5c3c6e0bf40c" | "20250109.221519" |              |       0 | NOT_REMOVED    |
```

**Finding Clusters and Nodes with Faulty Segment Groups**

Inspect the count of damaged groups by cluster and node.

```sql
SELECT
  c.name AS cluster_name,
  n.name AS node_name,
  g.status AS segment_group_status,
  seg.status AS segment_status,
  seg.kind,
  COUNT(*) AS segment_count
FROM
  sys.segment_groups g
LEFT JOIN
  sys.clusters c
  ON c.id = g.cluster_id
LEFT JOIN
  sys.stored_segments seg
  ON seg.segment_group_id = g.id
LEFT JOIN
  sys.nodes n
  ON n.id = seg.node_id
WHERE
  g.status <> 'INTACT'
  AND (seg.status <> 'INTACT' OR seg.status IS NULL)
  GROUP BY 1,2,3,4,5
  ORDER BY 1,2,3,4,5;
```

*Output*

```none
|   "cluster_name"   | "node_name" | "segment_group_status" | "segment_status" | "kind"  | "segment_count" |
|--------------------|-------------|------------------------|------------------|---------|-----------------|
| foundation_cluster | foundation0 | DAMAGED                |                  | VIRTUAL |               2 |
```

## Starting a Segment Rebuild Task

A user with System Administrator privileges can start a segment rebuild using the `CREATE TASK TYPE REBUILD` SQL statement. You cannot cancel a segment rebuild task after it is started.&#x20;

Most commonly, a rebuild task repairs all damaged or missing segments across the system.

**Example**

Create a rebuild task.

```sql
CREATE TASK TYPE REBUILD;
```

The system can continue to perform queries while rebuilding segments, but the process can impact I/O performance.

### Advanced Rebuild Commands

Rebuild tasks can also execute on specific Foundation Nodes or clusters. For information on fine-tuning rebuild tasks, see [CREATE TASK](<./././SQL Reference/Data Definition Language _DDL_ Statement Reference/Distributed Tasks.md>).&#x20;

## Checking Rebuild Task Status

Monitor the status of current and past segment rebuild tasks from the `sys.subtasks` system catalog table. For details, see [System Catalog](<./././SQL Reference/System Catalog Reference/System Catalog.md>).

```sql
SELECT * FROM sys.subtasks WHERE task_type = 'rebuild';
```

This table describes the statuses for a rebuild task.

| **Status** | **Status Detail**      | **Description**                                                                                                 | **Next Steps**                                                                                                                                                                             |
| ---------- | ---------------------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| complete   | no\_work               | The segment groups were already available and healthy. No rebuilding was needed.                                | None                                                                                                                                                                                       |
| complete   | complete               | Rebuild completed successfully.                                                                                 | None                                                                                                                                                                                       |
| running    | rebuild\_in\_progress  | Rebuild is in process.                                                                                          | You can monitor the progress of the rebuild task by checking the JSON dictionary in the `details` column of the `sys.subtasks` system catalog table.                                       |
| failed     | rebuild\_not\_possible | The number of missing or damaged segments exceeds the parity width, and the data cannot be recovered currently. | If missing segments are available on an offline drive, you can attempt another rebuild task when that drive is made available to the system. <br />Otherwise, you cannot recover the data. |
| failed     | rebuild\_no\_space     | There is not enough space available to rebuild the segment.                                                     | You can complete the rebuild by truncating other data to free up space.                                                                                                                    |
| failed     | failed\_on\_node       | The cluster lost its connection to the node before the rebuild was completed.                                   | This is a transient error. You can retry the rebuild task.                                                                                                                                 |
| failed     | error                  | An unexpected internal error occurred.                                                                          | Review error message details using the `rolehostd` logs. For details, see [Log Monitoring](<././Monitoring/Log Monitoring.md>).                                                            |

## Related Links

[CREATE TASK](<./././SQL Reference/Data Definition Language _DDL_ Statement Reference/Distributed Tasks.md>)

[Erasure Coding](<./././Ocient Architecture/Compute-Adjacent Input and Output on Large Working Sets.md>)

