In this video, we demonstrate how to safely replace a failed disk in a Proxmox 9 Ceph High Availability cluster. Follow step-by-step instructions to remove the faulty disk and integrate a new one without disrupting your VMs. Learn how Ceph automatically replicates data to maintain integrity across the cluster. This tutorial covers disk replacement procedures, cluster health checks, and monitoring real-time recovery. Perfect for IT professionals managing production clusters and home lab enthusiasts. Understand the best practices for maintaining Ceph HA and preventing data loss. Watch live recovery in action and see how to restore full cluster performance. Keep your Proxmox environment resilient and your virtual machines protected with this complete demo.

4. Simulate 1 Disk on the dead node instead

4.1. Symptoms

The simulation price for node pve03zfs is damaged 1 hard drive.

• Ceph reports OSD down or out
• The pool still has data, but a part is missing a copy

Ceph is reallocating data

4.2. Troubleshooting process

Step 1: Identify the OSD error

Use command or GUI to identify the OSD error

ceph status
ceph osd tree

Use command to view the serial Disk error
ls -l /dev/disk/by-id/
Disk fail: osd.5
lost sdc

Step 2: Check if the disk is really dead or just temporarily faulty

This step is only performed in the real environment, the Proxmox virtual environment will not detect it.
SSH into the node containing the OSD (e.g. pve03zfs):
Check if the disk is still present:
lsblk

Check SMART:
smartctl -a /dev/sdc
If:
• disk not present → really dead
• SMART reports failure → hardware error
• disk read-only → seriously damaged
If the disk is damaged → the disk must be replaced.

Step 3: Remove the faulty OSD from the cluster (mark out)

Mark OSD is down
ceph osd down osd.5

Mark OSD is out
ceph osd out osd.5

Remove OSD from CRUSH
ceph osd crush remove osd.5

Delete keyring & entry
ceph auth del osd.5
ceph osd rm osd.5

After this step, Ceph will automatically:
• rebalance PG
• recreate replicas on other OSDs
• cluster becomes stable again

Step 4: Replace new hard drive

Turn off the server with the faulty hard drive, proceed to replace the new Disk.
nano /etc/pve/qemu-server/102.conf
serial=DISK07

Step 5: Create a new OSD on that hard drive

Go to GUI  Ceph  OSD
Ceph automatically rebalances data from the remaining OSDs to the new OSD
According to the replication factor, ensure HA returns

Result: Faulty OSD is replaced → data is replicated → cluster returns to healthy state (take a time).
since this is a lab environment utilizing VM, the ceph process of resegmenting data onto actual disks
will be much faster depending on the actual device. The green area will be filled when the ceph data
resegmentation process is complete.