云原生知识星球
0%

K8S问题排查-外置存储插件nfs-provisioner报错selfLink was empty

发表于 更新于 分类于 阅读时长 ≈ 4 分钟

本次发现的问题是踩到了新版本Kubernetes的坑，查找资料发现zhangsi-lzq大佬已经分析的很清楚了，此处转载过来仅做学习记录，并新增永久解决方案。

作者：zhangsi-lzq
本文出处：https://www.cnblogs.com/zhangsi-lzq/p/14292628.html

问题背景

Kubernetes升级为1.20版本后，原有的后端nfs存储storageclass无法自动创建PV。查看PVC状态一直为pending状态。

分析过程

查看nfs-provisioner日志：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
kubectl logs nfs-client-provisioner-5f696dc8bb-qhmsn
E0118 03:01:07.352670       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=3737, ErrCode=NO_ERROR, debug=""
E0118 03:01:07.353951       1 reflector.go:322] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:668: Failed to watch *v1.StorageClass: Get https://10.96.0.1:443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=604432&timeoutSeconds=387&watch=true: dial tcp 10.96.0.1:443: connect: connection refused
W0118 03:01:07.366606       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:665: watch of *v1.PersistentVolume ended with: too old resource version: 11565 (605173)
W0118 03:01:07.367679       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:662: watch of *v1.PersistentVolumeClaim ended with: too old resource version: 11565 (605173)
I0118 03:08:28.340240       1 controller.go:987] provision "default/auth-platorm-redis-data-auth-platorm-redis-cluster-0" class "course-nfs-storage": started
E0118 03:08:28.343582       1 controller.go:1004] provision "default/auth-platorm-redis-data-auth-platorm-redis-cluster-0" class "course-nfs-storage": unexpected error getting claim reference: selfLink was empty, can't make reference
I0118 03:16:08.373590       1 controller.go:987] provision "default/auth-platorm-redis-data-auth-platorm-redis-cluster-0" class "course-nfs-storage": started
E0118 03:16:08.382178       1 controller.go:1004] provision "default/auth-platorm-redis-data-auth-platorm-redis-cluster-0" class "course-nfs-storage": unexpected error getting claim reference: selfLink was empty, can't make reference
I0118 03:30:41.647626       1 controller.go:987] provision "default/test-pvc" class "course-nfs-storage": started
E0118 03:30:41.658419       1 controller.go:1004] provision "default/test-pvc" class "course-nfs-storage": unexpected error getting claim reference: selfLink was empty, can't make reference
I0118 03:31:08.373713       1 controller.go:987] provision "default/auth-platorm-redis-data-auth-platorm-redis-cluster-0" class "course-nfs-storage": started
I0118 03:31:08.373969       1 controller.go:987] provision "default/test-pvc" class "course-nfs-storage": started
E0118 03:31:08.382279       1 controller.go:1004] provision "default/test-pvc" class "course-nfs-storage": unexpected error getting claim reference: selfLink was empty, can't make reference
E0118 03:31:08.382791       1 controller.go:1004] provision "default/auth-platorm-redis-data-auth-platorm-redis-cluster-0" class "course-nfs-storage": unexpected error getting claim reference: selfLink was empty, can't make reference

报错信息selfLink was empty，于是上网查询相关内容，在官方1.20的变更说明中[1]看到其中一条说明为：

1
Stop propagating SelfLink (deprecated in 1.16) in kube-apiserver

selfLink在1.16版本以后已经弃用，在1.20版本停用。而由于nfs-provisioner的实现是基于selfLink功能（同时也会影响其他用到selfLink这个功能的第三方软件），需要等nfs-provisioner的制作方重新提供新的解决方案。

解决方案

目前可用的临时方案是：修改/etc/kubernetes/manifests/kube-apiserver.yaml文件，找到如下内容后，在最后添加一项参数：

1
2
3
4
5
6
7
spec:
  containers:
  - command:
    - kube-apiserver    
    - --advertise-address=192.168.210.20    
    - --....... 
    - --feature-gates=RemoveSelfLink=false　　#添加此行

如果是高可用的k8s集群，则需要在所有master节点上进行此操作。

添加后需要删除apiserver的所有pod进行重启

1
2
3
kubectl delete pod kube-apiserver-master01 -n kube-system
kubectl delete pod kube-apiserver-master02 -n kube-system
kubectl delete pod kube-apiserver-master03 -n kube-system

三台apiserver被kubelet重启拉起后，再次查询PVC，可以看到PVC状态都为Bound，可以正常被PV绑定了。

（新增）永久解决方案：查资料发现，nfs-provisioner项目实际上已经放弃[2,3]，转而提供了下面3个provisioner，可以根据需要适配更新：

参考资料

  1. https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md
  2. https://github.com/kubernetes-retired/nfs-provisioner
  3. https://github.com/kubernetes-sigs/sig-storage-lib-external-provisioner