2 posts tagged with "Pod"

Recently, I set out to create a multi-cluster kind environment where clusters could communicate over a flat network. The goal was simple: pods in one cluster should be able to directly talk to pods in another cluster without requiring tunneling or proxies.

At first glance, this seemed tricky — the pod IPs assigned inside kind clusters exist only within the containerized network of each cluster, and Docker isolates these networks by default. However, I eventually found a straightforward solution that uses native routing, and I wanted to share my journey and what worked.

The Challenge​

By design, kind runs Kubernetes clusters inside Docker containers. This means:

• Pod IPs are only visible inside the Docker network created for each kind cluster.
• There’s no built-in way for a pod in one cluster to directly communicate with a pod in another cluster using its pod IP.
• Bridging these isolated networks requires either complex overlay solutions or manual routing.

Imagine a situation where you have multiple different clusters, each running an Istio service mesh and which are federated together to talk to each other. Now also imagine that these clusters are networked together such that each pod IP is uniquely addressable and able to be communicated with from any other cluster.

Context​

I faced a situation like this, where I needed to group endpoints into logical hostnames that represented services backed by those endpoints. Because endpoints could live anywhere, in any cluster, I landed on using a ServiceEntry to register the hostname and WorkloadEntries to represent the endpoints service that hostname.

But a problem came on enabling PeerAuthentication:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: prod-istio-system # The namespace of our istio installation
spec:
  mtls:
    mode: STRICT

Curl stopped working!

$ curl global-echo
upstream connect error or disconnect/reset before headers. reset reason: connection termination

Through some debugging, I was able to get it to:

$ curl global-echo
upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, 
transport failure reason: TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end

But to find the true root cause and solution took some digging.

Let's get into it