One of my volume was stuck in attaching/detaching mode with the error as follows

[pvc-b5d01232-xyzx-4758-9008-zxxxxxxxz-e-aaaaaa] time="2026-02-01T05:18:07Z" level=info msg="Starting with replicas [\"tcp://10.42.2.185:10000\"]"
time="2026-02-01T05:18:07Z" level=info msg="Connecting to remote: 10.42.2.185:10000"
[pvc-b5d01232-xyzx-4758-9008-zxxxxxxxz-e-aaaaaa] time="2026-02-01T05:18:07Z" level=info msg="Opening: 10.42.2.185:10000"
[pvc-b5d01232-xyzx-4758-9008-zxxxxxxxz-e-aaaaaa] time="2026-02-01T05:18:07Z" level=warning msg="failed to create backend with address tcp://10.42.2.185:10000: failed to open replica 10.42.2.185:10000 from remote: rpc error: code = Unknown desc = EOF"
time="2026-02-01T05:18:07Z" level=info msg="Adding backend: tcp://10.42.2.185:10000"
[pvc-b5d01232-xyzx-4758-9008-zxxxxxxxz-e-aaaaaa] 2026/02/01 05:18:07 cannot create an available backend for the engine from the addresses [tcp://10.42.2.185:10000]
[longhorn-instance-manager] time="2026-02-01T05:18:07Z" level=info msg="Process Manager: process pvc-b5d01232-xyzx-4758-9008-zxxxxxxxz-e-aaaaaa error out, error msg: exit status 1"
[longhorn-instance-manager] time="2026-02-01T05:18:07Z" level=debug msg="Process update: pvc-b5d01232-xyzx-4758-9008-zxxxxxxxz-e-aaaaaa: state error: Error: exit status 1"

Since all other longhorn volumes on that node was healthy, concluded that this is not a firewall or resource issue. Then found out that the volume-head meta was missing. Recreating the volume-head metadata fixed the issue.

• cloudnative
• Devops
• kubernetes