calico-node and calico-typha on debian crashed

[mathematicalsystems]

I installed projectcalico/tigera-operator version v3.29.3 but can't get calico-node and calico-typha pods to be in running state as follows:
errors:

• Warning Failed 5s (x5 over 68s) kubelet Error: failed to prepare subPath for volumeMount "tigera-ca-bundle" of container "calico-typha"
• Warning Failed 1s (x8 over 110s) kubelet Error: failed to prepare subPath for volumeMount "tigera-ca-bundle" of container "calico-node"

I've installed Kubernetes on debian 12:
export KUBERNETES_VERSION=v1.32
export CRIO_VERSION=v1.32

Calico Values:
cat <<-EOF>calico-values.yaml
installation:
kubernetesProvider: ""
cni:
type: Calico
calicoNetwork:
bgp: Enabled
ipPools:
- cidr: 10.128.0.0/16
encapsulation: VXLAN
EOF

install calico using helm:
helm repo add projectcalico https://docs.tigera.io/calico/charts
helm repo update
helm search repo projectcalico/tigera-operator --versions
helm install calico projectcalico/tigera-operator --version v3.29.3 --namespace tigera-operator --create-namespace -f calico-values.yaml

The problem
all pods up and running except these 2 pods keep getting errors as in describe pod shows:

$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
calico-system calico-node-m522s 0/1 CreateContainerConfigError 0 5m47s
calico-system calico-typha-5bb8d65d85-tlzms 0/1 CreateContainerConfigError 0 4m38s

$kubectl describe pod -n calico-system calico-typha-5bb8d65d85-tlzms
Events:
Type Reason Age From Message

---

Normal Scheduled 68s default-scheduler Successfully assigned calico-system/calico-typha-5bb8d65d85-tlzms to mathematicalsystems-com-server-001
Warning Failed 57s (x2 over 68s) kubelet Error: failed to mkdir /var/lib/kubelet/pods/f9598ad6-12d7-40d2-acde-8f3926dcb532/volume-subpaths/tigera-ca-bundle/calico-typha/1: mkdir /var/lib/kubelet/pods/f9598ad6-12d7-40d2-acde-8f3926dcb532/volume-subpaths/tigera-ca-bundle/calico-typha/1: file exists
Normal Pulled 5s (x7 over 68s) kubelet Container image "docker.io/calico/typha:v3.29.3" already present on machine
Warning Failed 5s (x5 over 68s) kubelet Error: failed to prepare subPath for volumeMount "tigera-ca-bundle" of container "calico-typha"

$ kubectl describe pod -n calico-system calico-node-m522s
Events:
Type Reason Age From Message

---

Normal Scheduled 2m4s default-scheduler Successfully assigned calico-system/calico-node-m522s to mathematicalsystems-com-server-001
Normal Pulled 2m4s kubelet Container image "docker.io/calico/pod2daemon-flexvol:v3.29.3" already present on machine
Normal Created 2m4s kubelet Created container: flexvol-driver
Normal Started 2m4s kubelet Started container flexvol-driver
Normal Pulled 2m3s kubelet Container image "docker.io/calico/cni:v3.29.3" already present on machine
Normal Created 2m3s kubelet Created container: install-cni
Normal Started 2m3s kubelet Started container install-cni
Warning Failed 36s (x3 over 2m2s) kubelet Error: failed to mkdir /var/lib/kubelet/pods/80d97545-fa44-418f-aa4a-6e3b5b73f15f/volume-subpaths/tigera-ca-bundle/calico-node/1: mkdir /var/lib/kubelet/pods/80d97545-fa44-418f-aa4a-6e3b5b73f15f/volume-subpaths/tigera-ca-bundle/calico-node/1: file exists
Normal Pulled 1s (x11 over 2m2s) kubelet Container image "docker.io/calico/node:v3.29.3" already present on machine
Warning Failed 1s (x8 over 110s) kubelet Error: failed to prepare subPath for volumeMount "tigera-ca-bundle" of container "calico-node"

[lucastigera]

Did you manage to solve the problem?
If not, could you share the kubelet logs and the calico-node logs? There might be something in them that can help us track down the issue.
Also, let us know which k8s distro and provider you're using.

[mathematicalsystems]

Yes I've upgraded Debian to latest, uninstalled Kubernetes, and reinitialized every thing starting up again, rejoining nodes back one by one to the cluster, and the problem just resolved, the main problem that there is no log message explains why, but for a high degree it was some software compatibility issues that has not appear clearly in the log.

[lucastigera]

Glad you were able to solve the problem, @mathematicalsystems !
Thanks for sharing the steps you took to overcome it.