Node Management

Rebooting a node

Key Considerations

1. Check if the node has any rook-ceph-osd-* pods. Verify the health of the corresponding Ceph cluster and bring down one node at a time.
2. Check for haproxy-ingress-* pods. If the node will be down for an extended period, disable its record in Constellix DNS.
3. Check if the node has the nautilus.io/linstor-server label. This node serves as a Linstor server. Some Linstor servers are redundant, while others are critical.
4. Check if the node has the nautilus.io/bgp-speaker label. There are two nodes used for MetalLB IPs—ensure one remains active.
5. Check if the node has the node-role.kubernetes.io/master label. Rebooting this node will make the cluster inaccessible unless it’s not an Admiralty virtual node.

Prerequisites

1. Install Ansible on your local computer.
2. Clone the repository of Ansible playbooks:

git clone https://gitlab.nrp-nautilus.io/prp/nautilus-ansible.git

3. Pull the latest updates from the playbook repository:

cd nautilus-ansible;
git pull

Reboot a Node Due to GPU Failure

Use the following command to reboot the node:

ansible-playbook reboot.yaml -i nautilus-ansible/nautilus-hosts.yaml -l <nodename>

Special Instructions to Reboot Ceph Nodes

To maintain redundancy in the Ceph cluster, only one node can be rebooted at a time.

Run this command to enter the rook-ceph-tools pod shell. Replace <namespace> with the appropriate Ceph cluster namespace (e.g., rook, rook-east, rook-pacific, rook-haosu, rook-suncave):

kubectl exec -it -n <namespace> $(kubectl get pods -n <namespace> --selector=app=rook-ceph-tools --output=jsonpath={.items..metadata.name}) -- bash

Once inside the pod shell, run:

watch ceph health detail

Wait until [WRN] OSD_DOWN: 1 osds down disappears from the ceph health detail output before rebooting the next node.

If OSD_DOWN does not clear once the node is back, look for its OSD pods stuck in Pending:

kubectl get pods -n <namespace> -o wide | grep rook-ceph-osd | grep -v Running

Since 2026-10-28, Gatekeeper gives Rook OSD and osd-prepare pods NoSchedule tolerations for nautilus.io/upgrading, nautilus.io/issue and cordon (node.kubernetes.io/unschedulable) when they are created (mutations/rook-osd-tolerations.yaml in prp/gatekeeper, applied by its CI), so those taints no longer keep a rebooted node’s OSDs down. Any other NoSchedule taint still does, for example remediation.medik8s.io/self-node-remediation, and kubectl describe pod on a stuck OSD pod then reports untolerated taint(s). The operator’s osd-prepare pod for that node stays Pending as well, so its OSD provisioning times out after 20 minutes and the CephCluster stays Progressing. Once whatever placed the taint is done with the node, remove it, wait until the OSD pods are Running and ceph osd stat shows every OSD up, then put it back; a NoSchedule taint does not evict pods that are already running. A pod that was already Pending before a toleration existed has to be deleted, so that its ReplicaSet creates one that has it.

kubectl taint node <node> <key>:NoSchedule-              # remove
kubectl taint node <node> <key>=<value>:NoSchedule       # put back once the OSDs are up

Ceph does not mark OSDs out on its own when a whole rack or larger CRUSH subtree is down (mon_osd_down_out_subtree_limit is rack), so nothing re-replicates while a site is down. A pool whose failure domain is host can place two copies or erasure-code shards of a placement group at one site that has two storage hosts; while that site is down, those placement groups are below min_size and stop serving I/O.

Recycling a node

1. If possible, do kubeadm reset on the node

2. Delete the node from kubernetes cluster

3. Delete the node from netbox

4. Close all gitlab issues related to the node

5. Check if there are any volumeattachments left for the node in kubernetes