Using the NeuReality Node Problem Detector
Check the NR1™ nodes in the cluster:
$ kubectl get node -l neureality.ai/arch=nr1
NAME STATUS ROLES AGE VERSION
aspen-nr1m-lab45 Ready <none> 40h v1.27.5-dirty
aspen-nr1m-lab46 Ready <none> 43h v1.27.5-dirty
See Node Conditions: notice AIServerDown, AIServerUnhealthy and NIFUnhealthy conditions.
$ kubectl describe node aspen-nr1m-lab45
...
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
KernelDeadlock False Wed, 05 Jun 2024 10:59:04 +0300 Wed, 05 Jun 2024 10:18:45 +0300 KernelHasNoDeadlock kernel has no deadlock
ReadonlyFilesystem False Wed, 05 Jun 2024 10:59:04 +0300 Wed, 05 Jun 2024 10:18:45 +0300 FilesystemIsNotReadOnly Filesystem is not read-only
AIServerDown False Wed, 05 Jun 2024 10:59:04 +0300 Wed, 05 Jun 2024 10:18:46 +0300 AIServerIsRunning AI Server is running and listening on port 5200
AIServerUnhealthy False Wed, 05 Jun 2024 10:59:04 +0300 Wed, 05 Jun 2024 10:18:46 +0300 AIServerHealthCheckSucceeding AI Server's healthcheck (<nodeIP>:5200/v1/health) is succeeding
NIFUnhealthy False Wed, 05 Jun 2024 10:59:04 +0300 Wed, 05 Jun 2024 10:18:46 +0300 NIFHealthCheckSucceeding NIF's healthcheck (<nodeIP>:5200/v1/health/nif) is succeeding
NetworkUnavailable False Wed, 05 Jun 2024 10:18:49 +0300 Wed, 05 Jun 2024 10:18:49 +0300 CalicoIsUp Calico is running on this node
MemoryPressure False Wed, 05 Jun 2024 11:00:22 +0300 Wed, 05 Jun 2024 10:15:24 +0300 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Wed, 05 Jun 2024 11:00:22 +0300 Wed, 05 Jun 2024 10:15:24 +0300 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Wed, 05 Jun 2024 11:00:22 +0300 Wed, 05 Jun 2024 10:15:24 +0300 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Wed, 05 Jun 2024 11:00:22 +0300 Wed, 05 Jun 2024 10:15:24 +0300 KubeletReady kubelet is posting ready status
...
If there is a problem with, for example, the AIServer, you will see AIServerUnhealthy status change to true.
$ kubectl describe node aspen-nr1m-lab45
...
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
...
AIServerDown True Wed, 05 Jun 2024 12:04:05 +0300 Wed, 05 Jun 2024 12:04:04 +0300 AIServerIsNotRunning unable to connect to 172.23.5.45:5200/v1/health
AIServerUnhealthy True Wed, 05 Jun 2024 12:04:05 +0300 Wed, 05 Jun 2024 12:03:07 +0300 AIServerHealthCheckFailing AI Server health check failed
...