Background⌗

My homelab is a cluster of 2 nodes.It works well.

And I have 3 vps(2 oracle free tier arm, 1 other) and want to set up a new microk8s cluster. But I encouter some network issues.

The ingress can’t forward traffic to other pods. It doesn’t happen on my homelab.

network issue

Solution⌗

I keep only one node in the cluster and check the network.

kubectl create deployment pingtest --image=busybox --replicas=2 -- sleep infinity

kubectl get pods -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP             NODE              NOMINATED NODE   READINESS GATES
pingtest-6fd7cf8988-ms5m5   1/1     Running   0          19h   10.1.135.134   node0             <none>           <none>
pingtest-6fd7cf8988-n78lh   1/1     Running   0          19h   10.1.135.135   node0             <none>           <none>

kubectl exec -it pingtest-6fd7cf8988-ms5m5 -- ping 10.1.135.135
PING 10.1.135.135 (10.1.135.135): 56 data bytes

The ping is stuck. But the pod in the some node should be able to communicate with each other.

I think some request could be blocked by the firewall. So I try to find something in the firewall.

firewall-cmd --set-log-denied=all
firewall-cmd --reload
dmesg | grep -i reject.*10.1.135.135

Then I found something as follow:

[71344.740165] filter_FWD_public_REJECT: IN=cali19d479ac3a2 OUT=cali2a1cc96cf4c MAC=ee:ee:ee:ee:ee:ee:62:3e:02:eb:fd:9f:08:00 SRC=10.1.135.134 DST=10.1.135.135 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=2040 DF PROTO=ICMP TYPE=8 CODE=0 ID=19 SEQ=0 MARK=0x10000

We can see the packet is rejected by the firewall. The cali1*** is the interfaces which are created by calico(microk8s’s cni).

We should allow the traffic from the calico interfaces.

firewall-cmd --zone=trusted --change-interface=cali+ --permanent
firewall-cmd --reload

Then the ping is ok.

We can find the configuration file in /etc/firewalld/zones/trusted.xml.

For the pods in different nodes, I also catch the similar logs.

[72559.866341] filter_FWD_public_REJECT: IN=cali19d479ac3a2 OUT=vxlan.calico MAC=ee:ee:ee:ee:ee:ee:62:3e:02:eb:fd:9f:08:00 SRC=10.1.135.134 DST=10.1.35.194 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=37711 DF PROTO=ICMP TYPE=8 CODE=0 ID=25 SEQ=0 MARK=0x10000

The only difference is the interface name(vxlan.calico).

I also allow the traffic from the vxlan.calico interface.

firewall-cmd --zone=trusted --change-interface=vxlan.calico --permanent
firewall-cmd --reload

Then everything works well now.