VMware Cloud Director 维护节点之间的同步流式传输复制。如果备用节点变得无法访问,则必须确定原因并解决问题。
问题
VMware Cloud Director 设备管理 UI 显示集群运行状况为 DEGRADED,其中一个备用节点的状态为 unreachable。
/nodes API 返回以下信息:localClusterHealth 为 DEGRADED,节点的 status 为 unreachable,nodeHealth 为 UNHEALTHY。
例如,
/nodes API 可能会返回以下节点信息。
{
"localClusterFailover": "MANUAL",
"localClusterHealth": "DEGRADED",
"localClusterState": [
{
"connectionString": "host=primary_host_IP user=repmgr dbname=repmgr connect_timeout=2",
"failover": {
"details": "failover = manual",
"mode": "MANUAL",
"repmgrd": {
"details": "On node primary_node_ID (primary_host_name): repmgrd = not applicable",
"status": "NOT APPLICABLE"
}
},
"id": primary_node_ID,
"location": "default",
"name": "primary_host_name",
"nodeHealth": "HEALTHY",
"nodeRole": "PRIMARY",
"role": "primary",
"status": "* running",
"upstream": ""
},
{
"connectionString": "host=unreachable_standby_host_IP user=repmgr dbname=repmgr connect_timeout=2",
"failover": {
"details": "failover state unknown - unable to ssh to failed or unreachable node",
"mode": "UNKNOWN",
"repmgrd": {
"details": "On node unreachable_standby_node_ID (unreachable_standby_host_name): repmgrd = n/a",
"status": "UNKNOWN"
}
},
"id": unreachable_standby_node_ID,
"location": "default",
"name": "unreachable_standby_host_name",
"nodeHealth": "UNHEALTHY",
"nodeRole": "STANDBY",
"role": "standby",
"status": "? unreachable",
"upstream": "primary_host_name"
},
{
"connectionString": "host=running_standby_host_IP user=repmgr dbname=repmgr connect_timeout=2",
"failover": {
"details": "failover = manual",
"mode": "MANUAL",
"repmgrd": {
"details": "On node running_standby_node_ID (running_standby_host_IP): repmgrd = not applicable",
"status": "NOT APPLICABLE"
}
},
"id": running_standby_node_ID,
"location": "default",
"name": "running_standby_host_name",
"nodeHealth": "HEALTHY",
"nodeRole": "STANDBY",
"role": "standby",
"status": "running",
"upstream": "primary_host_name"
}
],
"warnings": [
"unable to connect to node \"unreachable_standby_host_name\" (ID: unreachable_standby_node_ID)",
"node \"unreachable_standby_host_name\" (ID: unreachable_standby_node_ID) is registered as an active standby but is unreachable"
]
}
原因
为确保数据完整性,PostgreSQL 数据库使用预写式日志记录 (WAL)。主节点不断地将 WAL 流式传输到活动备用节点,以便进行复制和恢复。备用节点在接收到 WAL 后对其进行处理。如果备用节点无法访问,则将停止接收 WAL,并且无法作为候选节点提升为新的主节点。
解决方案
- 验证无法访问的备用节点的虚拟机是否正在运行。
- 验证备用节点的网络连接是否正常工作。
- 验证是否存在可能阻止备用节点与其他节点通信的 SSH 问题。
- 验证备用节点上的 vpostgres 服务是否正在运行。