Network issues about microk8s on my vps
Background⌗
My homelab is a cluster of 2 nodes.It works well.
And I have 3 vps(2 oracle free tier arm, 1 other) and want to set up a new microk8s cluster. But I encouter some network issues.
The ingress can’t forward traffic to other pods. It doesn’t happen on my homelab.
Solution⌗
I keep only one node in the cluster and check the network.
kubectl create deployment pingtest --image=busybox --replicas=2 -- sleep infinity
kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pingtest-6fd7cf8988-ms5m5 1/1 Running 0 19h 10.1.135.134 node0 <none> <none>
pingtest-6fd7cf8988-n78lh 1/1 Running 0 19h 10.1.135.135 node0 <none> <none>
kubectl exec -it pingtest-6fd7cf8988-ms5m5 -- ping 10.1.135.135
PING 10.1.135.135 (10.1.135.135): 56 data bytes
The ping is stuck. But the pod in the some node should be able to communicate with each other.
I think some request could be blocked by the firewall. So I try to find something in the firewall.
firewall-cmd --set-log-denied=all
firewall-cmd --reload
dmesg | grep -i reject.*10.1.135.135
Then I found something as follow:
[71344.740165] filter_FWD_public_REJECT: IN=cali19d479ac3a2 OUT=cali2a1cc96cf4c MAC=ee:ee:ee:ee:ee:ee:62:3e:02:eb:fd:9f:08:00 SRC=10.1.135.134 DST=10.1.135.135 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=2040 DF PROTO=ICMP TYPE=8 CODE=0 ID=19 SEQ=0 MARK=0x10000
We can see the packet is rejected by the firewall. The cali1***
is the interfaces which are created by calico(microk8s’s cni).
We should allow the traffic from the calico interfaces.
firewall-cmd --zone=trusted --change-interface=cali+ --permanent
firewall-cmd --reload
Then the ping is ok.
We can find the configuration file in
/etc/firewalld/zones/trusted.xml
.
For the pods in different nodes, I also catch the similar logs.
[72559.866341] filter_FWD_public_REJECT: IN=cali19d479ac3a2 OUT=vxlan.calico MAC=ee:ee:ee:ee:ee:ee:62:3e:02:eb:fd:9f:08:00 SRC=10.1.135.134 DST=10.1.35.194 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=37711 DF PROTO=ICMP TYPE=8 CODE=0 ID=25 SEQ=0 MARK=0x10000
The only difference is the interface name(vxlan.calico
).
I also allow the traffic from the vxlan.calico
interface.
firewall-cmd --zone=trusted --change-interface=vxlan.calico --permanent
firewall-cmd --reload
Then everything works well now.