Skip to content
ClickHouse Docs
ClickHouse DocsClickHouse Docs
49.6kGet Started

How to restore a replica after storage failure

View as MarkdownEdit this pageReport an issueSet up docs MCP

Copy access configuration from the healthy replica

Copy the contents of the access folder which contains local users from the healthy replica:

/var/lib/clickhouse/access

Back up the metadata folder from the healthy replica

  1. Navigate to the ClickHouse data directory:
cd /var/lib/clickhouse
  1. Create a backup of the metadata folder (including symbolic links): The metadata directory contains DDLs for databases and tables. The database directory has symlinks to /var/lib/clickhouse/store/.. which contains all the table DDLs.
{ find metadata -type f; find metadata -type l; find metadata -type l | xargs readlink -f; } | tar -cPf backup.tar --files-from=-

Restore the metadata on the faulty replica

  1. Copy the generated backup.tar file to the faulty replica.
  2. Extract it to the ClickHouse data directory:
cd /var/lib/clickhouse/
tar -xvPf backup.tar

Create the force restore flag

To trigger automatic data synchronization from other replicas, create the following flag:

sudo -u clickhouse touch /var/lib/clickhouse/flags/force_restore_data

Restart the faulty replica

  1. Restart the ClickHouse server on the faulty node.
  2. Check the server logs, you should observe parts being downloaded from the healthy replicas:
2025.11.02 00:00:04.047097 [ 682 ] {} <Debug> analytics.events_local (...) (Fetcher): Downloading files 23
2025.11.02 00:00:04.055542 [ 682 ] {} <Debug> analytics.events_local (...) (Fetcher): Download of part 202511_0_0_0 onto disk disk2 finished.
2025.11.02 00:00:04.101888 [ 687 ] {} <Debug> warehouse.customers_local (...) (Fetcher): Downloading part 2025_0_0_1 onto disk default.
2025.11.02 00:00:04.102005 [ 687 ] {} <Debug> warehouse.customers_local (...) (Fetcher): Downloading files 11
2025.11.02 00:00:04.102210 [ 690 ] {} <Debug> warehouse.customers_local (...) (Fetcher): Downloading part 2022_0_0_1 onto disk disk1.
2025.11.02 00:00:04.102247 [ 688 ] {} <Debug> warehouse.customers_local (...) (Fetcher): Downloading part 2021_0_0_1 onto disk disk2.
2025.11.02 00:00:04.102331 [ 690 ] {} <Debug> warehouse.customers_local (...) (Fetcher): Downloading files 11
Navigation