Rustfs node lost disk after pod restart

[jurim76]

k8s version - 1.33
rustfs helm version - 0.0.76
rustfs setup - 4 replicas, distributed mode, one data PVC per pod, PVC size 1Gi

> k get pod -l app.kubernetes.io/instance=rustfs
rustfs-0                                   1/1     Running   3 (114m ago)    114m
rustfs-1                                   1/1     Running   3 (114m ago)    114m
rustfs-2                                   1/1     Running   3 (114m ago)    114m
rustfs-3                                   1/1     Running   3 (114m ago)    114m

>  k get pvc -l app.kubernetes.io/instance=rustfs
NAME            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
data-rustfs-0   Bound    pvc-c0a1e7ad-3798-413a-9b4e-ae744da30909   1Gi        RWO            topolvm-xfs    <unset>                 116m
data-rustfs-1   Bound    pvc-f5ea3497-de3b-4299-b33a-2f7e4867f72a   1Gi        RWO            topolvm-xfs    <unset>                 115m
data-rustfs-2   Bound    pvc-153e8521-66cd-4430-8cf9-3665667a5d2c   1Gi        RWO            topolvm-xfs    <unset>                 115m
data-rustfs-3   Bound    pvc-1319df3b-d6d6-4125-b9c4-f274791f0d78   1Gi        RWO            topolvm-xfs    <unset>                 115m
logs-rustfs-0   Bound    pvc-47f58c8c-9fc8-4afd-8c3f-2e3300031513   1Gi        RWO            topolvm-xfs    <unset>                 116m
logs-rustfs-1   Bound    pvc-912b33d7-6809-4bdf-9f73-18a4be322682   1Gi        RWO            topolvm-xfs    <unset>                 115m
logs-rustfs-2   Bound    pvc-afa5404d-4fd3-4d6c-a7ad-5735176d3452   1Gi        RWO            topolvm-xfs    <unset>                 115m
logs-rustfs-3   Bound    pvc-8361487d-0820-4e41-a4f6-e1e8d1cf9b96   1Gi        RWO            topolvm-xfs    <unset>                 115m

UI status

---

Let's recreate rustfs-0 pod

k delete pod rustfs-0

k get pod  -l app.kubernetes.io/instance=rustfs
rustfs-0                                   1/1     Running     0               15m
rustfs-1                                   1/1     Running     3 (119m ago)    120m
rustfs-2                                   1/1     Running     3 (119m ago)    120m
rustfs-3                                   1/1     Running     3 (119m ago)    120m

UI status

rustfs-0 pod's log

[2025-12-27 15:43:06.018153 +00:00] ERROR [rustfs_ahm::heal::manager] [crates/ahm/src/heal/manager.rs:629] [rustfs-worker:ThreadId(8)] Heal task failed: 97af10f9-0ff1-42fe-abfb-cfbcdded49bc - Heal task execution failed: Failed to heal object .rustfs.sys/buckets/.usage.json: Other error: Io error: Failed to write data: erasure write quorum (offline-disks=4/4)

[2025-12-27 15:43:12.979283 +00:00] WARN [rustfs_ecstore::store_init] [crates/ecstore/src/store_init.rs:269] [rustfs-worker:ThreadId(10)] load_format_erasure err: "http://rustfs-0.rustfs-headless:9000/data" Io(Custom { kind: Other, error: "io error can not find disk" })

[2025-12-27 15:43:12.979609 +00:00] WARN [rustfs_ecstore::set_disk] [crates/ecstore/src/set_disk.rs:1828] [rustfs-worker:ThreadId(10)] renew_disk: connect_endpoint err Io(Custom { kind: Other, error: "io error can not find disk" })

I'm expecting that pod will be recreated successfully, the PVCs and other pods are fine.
What's the reason that rustfs-0 pod couldn't find the disk?

[jurim76]

I've deleted /data/.rustfs.sys completely inside the rustfs-0 pod and recreated the pod
After this action the node became healthy

But I'm not sure if this method is good enough to repair the node :)

[loverustfs]

It will automatically repair itself.If you are concerned, you can access all the data at once.

[jurim76]

The node wasn't repaired after one day, looks like auto-repair doesn't work for me

[loverustfs]

https://github.com/rustfs/rustfs/tree/weisd/scan

This branch merge will fix this bug.

If the updated PVC has no data, you can access the object, and it will be automatically fixed.

[jurim76]

@loverustfs any news about it?

Recently I've upgraded rustfs from 79 to 80 and got broken cluster.
By default sts uses podManagementPolicy: Parallel and if sts spec has been changed, all rustfs pods will be recreated in parallel.
After this rustfs cluster is unable to heal itself (ie cluster is broken) and all pods spam with Io error: Failed to write data: erasure write quorum (offline-disks=4/4) error.

[loverustfs]

Will it work normally if I revert to alpha.79?

[jurim76]

I reverted to alpha.79 and got the same result - the cluster is broken and data is not available.
Pods are fine, but UI is not available and logs full of errors Failed to write data: erasure write quorum (offline-disks=4/4)

rustfs-0                              1/1     Running   1 (7m46s ago)   9m58s
rustfs-1                              1/1     Running   3 (11m ago)     11m
rustfs-2                              1/1     Running   0               11m
rustfs-3                              1/1     Running   3 (13m ago)     13m

[jurim76]

@loverustfs

I've tried scaling down the rustfs sts to 0 replicas, then scaling up to 1 to run the first node, then scaling up to previous value 4
After this the cluster is still dead, the last entry in log on first node

[2026-01-21 20:13:38.810227 +00:00] WARN [rustfs_ahm::heal::task] [crates/ahm/src/heal/task.rs:370] [rustfs-worker:ThreadId(12)] Step 1: Checking object existence and metadata

[2026-01-21 20:13:38.810245 +00:00] WARN [rustfs_ahm::heal::task] [crates/ahm/src/heal/task.rs:370] [rustfs-worker:ThreadId(15)] Step 1: Checking object existence and metadata

[2026-01-21 20:13:48.811119 +00:00] WARN [rustfs_ahm::heal::task] [crates/ahm/src/heal/task.rs:370] [rustfs-worker:ThreadId(10)] Step 1: Checking object existence and metadata

[2026-01-21 20:13:48.811163 +00:00] WARN [rustfs_ahm::heal::task] [crates/ahm/src/heal/task.rs:370] [rustfs-worker:ThreadId(14)] Step 1: Checking object existence and metadata

On other nodes

RustFS server started successfully at [::]:9000, current time: 2026-01-21 20:13:47.064461220 UTC

With no any activity after (the pvc size is 4G, so the recovery time should be fast in theory)

I guess it's a serious blocker to use rustfs in k8s cluster.

Current rustfs version 1.0.0-alpha.80.

[jurim76]

The issue was fixed after upgrading to version 1.0.0-alpha.81

[loverustfs]

Ok, thank you for your feedback.

[cr1cr1]

I do not think is fixed, deleting just the pods in 1.0.0-alpha.98