云原生知识星球
0%

总结分享-Calico-node镜像编译问题记录

发表于 更新于 分类于 阅读时长 ≈ 7 分钟

编译过程

参考官方资料[1]，执行编译命令：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[root@node01 projectcalico]# make -C node image
"Build dependency versions"
BIRD_VERSION          = v0.3.3-151-g767b5389
"Test dependency versions"
CNI_VER               = master
"Calico git version"
GIT_VERSION           =
make: Entering directory `/home/go/gopath/src/github.com/projectcalico/node'
mkdir -p .go-pkg-cache bin /home/go/gopath/pkg/mod && docker run --rm --net=host --init -e GOPRIVATE='github.com/tigera/*' -e GO111MODULE=on -v /home/go/gopath/pkg/mod:/go/pkg/mod:rw -e LOCAL_USER_ID=0 -e GOCACHE=/go-cache -e GOARCH=amd64 -e GOPATH=/go -e OS=linux -e GOOS=linux -e GOFLAGS= -v /home/go/gopath/src/github.com/projectcalico/node:/go/src/github.com/projectcalico/node:rw -v /home/go/gopath/src/github.com/projectcalico/node/.go-pkg-cache:/go-cache:rw -w /go/src/github.com/projectcalico/node calico/go-build:v0.40 sh -c ' go mod download'
...
Starting with UID : 0
useradd: UID 0 is not unique
su-exec: getpwnam(user): Success
make: *** [remote-deps] Error 1
make: Leaving directory `/home/go/gopath/src/github.com/projectcalico/node'

从日志看，构建在remote-deps阶段失败，错误是useradd: UID 0 is not unique。从日志中的docker启动容器的命令看，是有个LOCAL_USER_ID=0的参数，说明是想以root用户起容器，但这个过程执行了useradd命令添加用户（理论上是不应该执行到这里的）。

分析calico-node的entrypoint.sh，如果是以root用户启动，代码走到第10行就结束了，而判断是否为root用户的依据是RUN_AS_ROOT参数。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
#!/bin/bash

# Add local user
# Either use the LOCAL_USER_ID if passed in at runtime or
# fallback

USER_ID=${LOCAL_USER_ID:-9001}

if [ "${RUN_AS_ROOT}" = "true" ]; then
  exec "$@"
fi

echo "Starting with UID : $USER_ID" 1>&2
# Do not create mail box.
/bin/sed -i 's/^CREATE_MAIL_SPOOL=yes/CREATE_MAIL_SPOOL=no/' /etc/default/useradd
# Don't pass "-m" to useradd if the home directory already exists (which can occur if it was volume mounted in) otherwise it will fail.
if [[ ! -d "/home/user" ]]; then
    /usr/sbin/useradd -m -U -s /bin/bash -u $USER_ID user
else
    /usr/sbin/useradd -U -s /bin/bash -u $USER_ID user
fi

...

exec /sbin/su-exec user "$@"

从make的执行结果看，没有发现RUN_AS_ROOT变量，再查看calico-node的Makefile文件，也没有定义，猜测是缺少了RUN_AS_ROOT变量定义导致的：

1
[root@node01 projectcalico]# grep -r "RUN_AS_ROOT" ./node/

参考官网资料[2]，发现go-build的Makefile里有针对root用户的处理：

1
2
3
4
ifeq ("$(LOCAL_USER_ID)", "0")
# The build needs to run as root.
EXTRA_DOCKER_ARGS+=-e RUN_AS_ROOT='true'
endif

同步修改到calico-node的Makefile文件：

1
2
[root@node01 projectcalico]# grep -r "RUN_AS_ROOT" ./node/
./node/Makefile.common: EXTRA_DOCKER_ARGS+=-e RUN_AS_ROOT='true'

再次执行make命令：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
[root@node01 projectcalico]# make -C node image
"Build dependency versions"
BIRD_VERSION          = v0.3.3-151-g767b5389
"Test dependency versions"
CNI_VER               = master
"Calico git version"
GIT_VERSION           =
make: Entering directory `/home/go/gopath/src/github.com/projectcalico/node'
mkdir -p .go-pkg-cache bin /home/go/gopath/pkg/mod && docker run --rm --net=host --init -e GOPRIVATE='github.com/tigera/*' -e RUN_AS_ROOT='true' -e GO111MODULE=on -v /home/go/gopath/pkg/mod:/go/pkg/mod:rw -e GOCACHE=/go-cache -e GOARCH=amd64 -e GOPATH=/go -e OS=linux -e GOOS=linux -e GOFLAGS= -e LOCAL_USER_ID=0 -v /home/go/gopath/src/github.com/projectcalico/node:/go/src/github.com/projectcalico/node:rw -v /home/go/gopath/src/github.com/projectcalico/node/.go-pkg-cache:/go-cache:rw -w /go/src/github.com/projectcalico/node -e CGO_ENABLED=1 calico/go-build:v0.40 sh -c ' go build -v -o dist/bin//calico-node-amd64  -ldflags " -X github.com/projectcalico/node/pkg/startup.VERSION= -X github.com/projectcalico/node/buildinfo.GitVersion=<unknown> -X github.com/projectcalico/node/buildinfo.BuildDate=2023-05-09T06:06:42+0000 -X github.com/projectcalico/node/buildinfo.GitRevision=<unknown>" ./cmd/calico-node/main.go'
github.com/kelseyhightower/confd/pkg/backends
github.com/projectcalico/libcalico-go/lib/apis/v1/unversioned
github.com/projectcalico/libcalico-go/lib/backend/encap
...
Starting with UID : 9001
calico-node-amd64 -v

docker build --pull -t calico/node:latest-amd64 . --build-arg BIRD_IMAGE=calico/bird:v0.3.3-151-g767b5389-amd64 --build-arg QEMU_IMAGE=calico/go-build:v0.40 --build-arg GIT_VERSION= -f ./Dockerfile.amd64
Sending build context to Docker daemon   66.3MB
Step 1/40 : ARG ARCH=x86_64
Step 2/40 : ARG GIT_VERSION=unknown
Step 3/40 : ARG IPTABLES_VER=1.8.2-16
Step 4/40 : ARG RUNIT_VER=2.1.2
Step 5/40 : ARG BIRD_IMAGE=calico/bird:latest
Step 6/40 : FROM calico/bpftool:v5.3-amd64 as bpftool
...
Step 16/40 : RUN dnf install -y 'dnf-command(config-manager)' &&     dnf config-manager --set-enabled PowerTools &&     yum install -y rpm-build yum-utils make &&     yum install -y wget glibc-static gcc &&     yum -y update-minimal --security --sec-severity=Important --sec-severity=Critical
 ---> Running in eca2b4c5f0b4
CentOS Linux 8 - AppStream                       51  B/s |  38  B     00:00
Error: Failed to download metadata for repo 'appstream': Cannot prepare internal mirrorlist: No URLs in mirrorlist

从编译日志看，问题是yum安装依赖包出错了，原因是使用了默认源vault.centos.org，更改Dockerfile.amd64，替换成国内的阿里源[3]：

1
2
3
4
5
6
7
8
9
-ARG CENTOS_MIRROR_BASE_URL=http://vault.centos.org/8.1.1911
+ARG CENTOS_MIRROR_BASE_URL=https://mirrors.aliyun.com/centos-vault/8.1.1911

+RUN mv /etc/yum.repos.d /etc/yum.repo.d-bk && \
+   mkdir -p /etc/yum.repos.d && mv /centos.repo /etc/yum.repos.d && \
+   yum clean all && yum makecache && \
    dnf install -y 'dnf-command(config-manager)' && \
    # Enable PowerTools repo for '-devel' packages
-   dnf config-manager --set-enabled PowerTools && \

更改centos.repo文件，跳过gpgcheck校验，增加PowerTool源：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
[centos-8-base-os]
name = CentOS - BaseOS
baseurl = https://mirrors.aliyun.com/centos-vault/8.1.1911/BaseOS/x86_64/os
enabled = 1
gpgkey = https://mirrors.aliyun.com/keys/RPM-GPG-KEY-CentOS-Official
gpgcheck = 0

[centos-8-appstream]
name = CentOS - AppStream
baseurl = https://mirrors.aliyun.com/centos-vault/8.1.1911/AppStream/x86_64/os
enabled = 1
gpgkey = https://mirrors.aliyun.com/keys/RPM-GPG-KEY-CentOS-Official
gpgcheck = 0

[Centos8-PowerTool-local1]
name=Centos8-PowerTool-local1
baseurl=https://mirrors.aliyun.com/centos-vault/8.1.1911/PowerTools/x86_64/os
enabled=1
gpgcheck=0

继续编译：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
...
docker build --pull -t calico/node:latest-amd64 . --build-arg BIRD_IMAGE=calico/bird:v0.3.3-151-g767b5389-amd64 --build-arg QEMU_IMAGE=calico/go-build:v0.40 --build-arg GIT_VERSION= -f ./Dockerfile.amd64
Sending build context to Docker daemon   66.3MB
Step 1/41 : ARG ARCH=x86_64
Step 2/41 : ARG GIT_VERSION=unknown
Step 3/41 : ARG IPTABLES_VER=1.8.2-16
Step 4/41 : ARG RUNIT_VER=2.1.2
Step 5/41 : ARG BIRD_IMAGE=calico/bird:latest
Step 6/41 : FROM calico/bpftool:v5.3-amd64 as bpftool
...
Step 12/41 : ARG CENTOS_MIRROR_BASE_URL=https://mirrors.aliyun.com/centos-vault/8.1.1911
 ---> Using cache
 ---> a96f716928d7
...
Step 17/41 : RUN mv /etc/yum.repos.d /etc/yum.repo.d-bk &&     mkdir -p /etc/yum.repos.d && mv /centos.repo /etc/yum.repos.d &&     yum clean all && yum makecache &&     dnf install -y 'dnf-command(config-manager)' &&     yum install -y rpm-build yum-utils make &&     yum install -y wget glibc-static gcc &&     yum -y update-minimal --security --sec-severity=Important --sec-severity=Critical
 ---> Using cache
 ---> a9ffd418a7a4
...
Step 24/41 : FROM registry.access.redhat.com/ubi8/ubi-minimal:8.1-407
8.1-407: Pulling from ubi8/ubi-minimal
Digest: sha256:01b8fb7b3ad16a575651a4e007e8f4d95b68f727b3a41fc57996be9a790dc4fa
Status: Image is up to date for registry.access.redhat.com/ubi8/ubi-minimal:8.1-407
 ---> 6ce38bb5210c
...
Step 39/41 : COPY dist/bin/calico-node-amd64 /bin/calico-node
 ---> Using cache
 ---> 916fbf133fb0
Step 40/41 : COPY --from=bpftool /bpftool /bin
 ---> Using cache
 ---> f797db5c4eb4
Step 41/41 : CMD ["start_runit"]
 ---> Using cache
 ---> fe6496ded4a6
[Warning] One or more build-args [QEMU_IMAGE] were not consumed
Successfully built fe6496ded4a6
Successfully tagged calico/node:latest-amd64
touch .calico_node.created-amd64
make: Leaving directory `/home/go/gopath/src/github.com/projectcalico/node'

查看编译的镜像：

1
2
3
4
[root@node01 github.com]# docker images
REPOSITORY                                                  TAG                          IMAGE ID       CREATED         SIZE
calico/node                                                 latest-amd64                 77f4ca933207   7 hours ago     264MB
<none>                                                      <none>                       420e5252b060   7 hours ago     633MB

参考文献

  1. https://github.com/projectcalico/calico/blob/master/DEVELOPER_GUIDE.md
  2. https://github.com/projectcalico/go-build/blob/7a75e06f7e9b39df8697ca96f6d5f42369155902/Makefile.common
  3. https://mirrors.aliyun.com/centos-vault/8.1.1911/