原本由两台Kubernetes组成的小集群,但是今天只开启了一台机器,也就是只有一个节点,造成了无法删除pod实例的原因。
先查看一下现在的容器的运行状态:
[root@k8s ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-controller-lv8md 1/1 Unknown 0 16h
nginx-controller-sb3fx 1/1 Unknown 2 16h
nginx2-1216651254-4b2dw 0/1 ImagePullBackOff 0 8m
nginx2-1216651254-dbtms 0/1 ImagePullBackOff 0 8m
nginx2-1216651254-fhb4r 0/1 ImagePullBackOff 0 8m
查看有哪些replicationcontroller [简写rc]
[root@k8s ~]# kubectl get rc
No resources found.
查看有哪些services
[root@k8s ~]# kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.254.0.1 <none> 443/TCP 2d
看到上面没有rc,也没有services,那尝试这样删除所有的pods:
[root@k8s ~]# kubectl delete pods --all
pod "nginx-controller-lv8md" deleted
pod "nginx-controller-sb3fx" deleted
pod "nginx2-1216651254-4b2dw" deleted
pod "nginx2-1216651254-dbtms" deleted
pod "nginx2-1216651254-fhb4r" deleted
但是还是无法删除,查看已经部署的容器;
[root@k8s ~]# kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
nginx2 3 3 3 0 16h
[root@k8s ~]# kubectl delete deployment nginx2
deployment "nginx2" deleted
为什么这三个Pod实例没有rc或者services呢,因为创建它的时候是使用run来实现的;
但是剩下的两个实例怎么删除呢?
[root@k8s ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-controller-lv8md 1/1 Unknown 0 20h
nginx-controller-sb3fx 1/1 Unknown 2 20h
查看一下实例的信息:
[root@k8s ~]# kubectl describe pod nginx-controller-lv8md
Name: nginx-controller-lv8md
Namespace: default
Node: k8s-node/10.0.10.11 #这里是重点,因为这两个容器是分配到了k8s-node节点上,而这个节点现在宕机了。
Start Time: Tue, 13 Jun 2017 02:01:45 +0800
Labels: app=nginx
Status: Terminating (expires Mon, 12 Jun 2017 21:46:21 +0800)
Termination Grace Period: 30s
Reason: NodeLost
Message: Node k8s-node which was running pod nginx-controller-lv8md is unresponsive
IP: 172.21.42.3
Controllers: ReplicationController/nginx-controller
Containers:
nginx:
Container ID: docker://03fa59f9efc06e43ed8c9acc7d4c7533983d5733223dbb2efa5f65928d965b5b
Image: reg.docker.lc/share/nginx:latest
Image ID: docker-pullable://reg.docker.lc/share/nginx@sha256:e5c82328a509aeb7c18c1d7fb36633dc638fcf433f651bdcda59c1cc04d3ee55
Port: 80/TCP
State: Running
Started: Tue, 13 Jun 2017 02:01:47 +0800
Ready: True
Restart Count: 0
Volume Mounts: <none>
Environment Variables: <none>
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
No volumes.
QoS Class: BestEffort
Tolerations: <none>
No events.
因为这两个容器的rc,services都已经删除了,但是还保持这个Unknown状态是由于目标主机无法响应并返回信息导致;既然目标主机都已经宕机了,那就直接移除节点;
[root@k8s ~]# kubectl delete node k8s-node
node "k8s-node" deleted
[root@k8s ~]# kubectl get node
NAME STATUS AGE
k8s Ready 2d
因为节点都不存在了,那就没有了容器的状态信息
[root@k8s ~]# kubectl get pods
No resources found.
上面这样虽然是删除了那些已经也错的实例,但却删除了节点。 那么如果节点现在恢复了,是否会自动加入到集群呢?
主节点上的日志很快就发现了k8s-node节点了;
Jun 13 15:06:13 k8s kube-controller-manager[34050]: E0613 15:06:13.917313 34050 actual_state_of_world.go:475] Failed to set statusUpdateNeeded to needed true because nodeName="k8s-node" does not exist
Jun 13 15:06:14 k8s kube-controller-manager[34050]: I0613 15:06:14.618864 34050 event.go:217] Event(api.ObjectReference{Kind:"Node", Namespace:"", Name:"k8s-node", UID:"c9864434-5006-11e7-ab16-000c29e9277a", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node k8s-node event: Registered Node k8s-node in NodeController
节点自动又回来了,因为在配置文件中已经配置好了,只要节点一上线就会加入到集群中。
[root@k8s ~]# kubectl get nodes
NAME STATUS AGE
k8s Ready 2d
k8s-node Ready 1m
创建一个Nginx实例:
[root@k8s ~]# kubectl create -f Nginx.yaml
replicationcontroller "nginx-controller" created
service "nginx-service" created
Kubernetes已经把此实例分配到刚启动的k8s-node节点上了;
[root@k8s ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-controller-zsx2q 1/1 Running 0 6s
[root@k8s ~]# kubectl describe pod nginx-controller-zsx2q |grep "Node"
Node: k8s-node/10.0.10.11
动态扩展,添加一个容器数量;
[root@k8s ~]# kubectl scale replicationcontroller --replicas=2 nginx-controller
replicationcontroller "nginx-controller" scaled
查看集群现在的状态信息;
[root@k8s ~]# kubectl describe svc nginx-service
Name: nginx-service
Namespace: default
Labels: <none>
Selector: app=nginx
Type: ClusterIP
IP: 10.254.132.82
External IPs: 10.0.10.10
Port: <unset> 8000/TCP
Endpoints: 172.21.42.2:80,172.21.93.2:80
Session Affinity: None
No events.
此时再删除k8s-node节点又会如何呢?
[root@k8s ~]# kubectl delete node k8s-node
node "k8s-node" deleted
已经删除了一个节点了:
[root@k8s ~]# kubectl get nodes
NAME STATUS AGE
k8s Ready 2d
Pod的数量没有减少还是两个:
[root@k8s ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-controller-43qpx 1/1 Running 0 14m
nginx-controller-zsx2q 1/1 Running 0 19m
查看集群中的分配信息时发现,两个容器的Ip段都是172.21.93的说明来自同一节点;
[root@k8s ~]# kubectl describe svc nginx-service
Name: nginx-service
Namespace: default
Labels: <none>
Selector: app=nginx
Type: ClusterIP
IP: 10.254.132.82
External IPs: 10.0.10.10
Port: <unset> 8000/TCP
Endpoints: 172.21.93.2:80,172.21.93.3:80
Session Affinity: None
No events.
重新加入k8s-node 节点,需要在k8s-node上重启一下服务:
systemctl restart kubelet